Difference between revisions of "Group 8 Overview"
| (19 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
<!-- BANNER --> | <!-- BANNER --> | ||
| − | [[Image:Group8ProjectBanner.png| | + | [[Image:Group8ProjectBanner.png|1050px|right|width="100%"]] |
<!--MAIN HEADER --> | <!--MAIN HEADER --> | ||
{|style="background-color:#1B338F;" width="100%" cellspacing="0" cellpadding="0" valign="top" border="0" | | {|style="background-color:#1B338F;" width="100%" cellspacing="0" cellpadding="0" valign="top" border="0" | | ||
| Line 23: | Line 23: | ||
== Background == | == Background == | ||
| − | === Time | + | === Time Series Explorer === |
Time series data has always played an important role in understanding and evaluating past behaviours. The usage of time-series information has allowed companies and organizations to tune their operations instead of simply improving by trial and error. Time series data have also been used to evaluate whether an organization has grown throughout the years of operation. The analysis techniques for time-bound data help to describe and explore insights, while predicting the future outcomes such as revenues and profits. | Time series data has always played an important role in understanding and evaluating past behaviours. The usage of time-series information has allowed companies and organizations to tune their operations instead of simply improving by trial and error. Time series data have also been used to evaluate whether an organization has grown throughout the years of operation. The analysis techniques for time-bound data help to describe and explore insights, while predicting the future outcomes such as revenues and profits. | ||
| − | This category of data analysis is not only constrained to the business world. A country's economic data is also a rich pool of information that can be analysed for insights. One of the important example measures that can allow us to get a glimpse would be the Consumer Price Index (CPI). Using this easily available measure, we can explore living standards of a country's citizens by understanding the factors affecting their day-to-day lives. | + | This category of data analysis is not only constrained to the business world. A country's economic data is also a rich pool of information that can be analysed for insights. One of the important example measures that can allow us to get a glimpse would be the Consumer Price Index (CPI). Using this easily available measure, we can explore living standards of a country's citizens by understanding the factors affecting their day-to-day lives. |
| − | === Use Case === | + | === Use Case - Consumer Price Index === |
The Consumer Price Index (CPI), is a critical indicator to assess a country's consumer price inflation. To profile the weighted average price changes for households' cost of living, Singapore's CPI adopts a fixed basket of residents' commonly consumption goods and services. About 6,600 brands/varieties from 4,200 outlets are selected in the 2014-based CPI (the dataset for our study). At basket level, the composition of goods and services can be categorized into 10 major divisions, which are listed below: | The Consumer Price Index (CPI), is a critical indicator to assess a country's consumer price inflation. To profile the weighted average price changes for households' cost of living, Singapore's CPI adopts a fixed basket of residents' commonly consumption goods and services. About 6,600 brands/varieties from 4,200 outlets are selected in the 2014-based CPI (the dataset for our study). At basket level, the composition of goods and services can be categorized into 10 major divisions, which are listed below: | ||
| Line 36: | Line 36: | ||
Transport, Communication, Recreation & Culture, Education, Miscellaneous Goods & Services | Transport, Communication, Recreation & Culture, Education, Miscellaneous Goods & Services | ||
| − | [[Image:SG_2014_BASED_CPI.png|500px|right| | + | [[Image:SG_2014_BASED_CPI.png|500px|thumb|right|Figure 1: 2014 CPI Basket Items from Singstat<ref name="singstat">Department of Statistics Singapore. [http://www.singstat.gov.sg/educational-corner/faq-on-CPI], "FAQ on Consumer Price Index (CPI)", Retrieved on 30 November 2017</ref>]] |
To construct the CPI, two main types of data are required - the sample price data of the good or service, and the weighting data to represent the proportion of different categories' expenditure. The price data is gathered through a combination of data collection modes, while the frequency of price collection depends on the price behavior of the item. The weighting data is derived from the expenditure values collected in the Household Expenditure Survey, the latest one came from HES 2012/13, updated to 2014 values by taking into account price changes between 2012/13 and 2014. | To construct the CPI, two main types of data are required - the sample price data of the good or service, and the weighting data to represent the proportion of different categories' expenditure. The price data is gathered through a combination of data collection modes, while the frequency of price collection depends on the price behavior of the item. The weighting data is derived from the expenditure values collected in the Household Expenditure Survey, the latest one came from HES 2012/13, updated to 2014 values by taking into account price changes between 2012/13 and 2014. | ||
| − | As a Price index, CPI can also be affected by other types of costs in Singapore. Indicators like COE Bidding Price, Import and Export price index, Exchange Rates, are assumed to have interesting relationships with CPI which are worthy of further study. These data sources are also acceptable within the proposed system design within certain format restrictions. | + | As a Price index, CPI can also be affected by other types of costs in Singapore. Indicators like COE Bidding Price<ref name="COE">Yeap, R. [http://tralvex.com/pub/cars/coe.htm], "COE Prices", Retrieved on 30 November 2017</ref>, Import and Export price index<ref name="CEIC">CEICData.com [https://insights-ceicdata-com.libproxy.smu.edu.sg/Untitled-insight/views], "CEIC - A Euromoney Institutional Investor Company", Retrieved on 30 November 2017</ref>, Exchange Rates<ref name="MAS">Monetary Authority of Singapore. [https://secure.mas.gov.sg/msb/ExchangeRates.aspx], "Monetary Authority of Singapore", Retrieved on 30 November 2017</ref>, are assumed to have interesting relationships with CPI which are worthy of further study. These data sources are also acceptable within the proposed system design within certain format restrictions. |
| + | |||
| + | |||
| + | |||
| + | |||
| + | |||
| + | |||
| + | |||
== Motivation == | == Motivation == | ||
| Line 66: | Line 73: | ||
== Data Source == | == Data Source == | ||
| − | The Consumer Price Index (CPI) data is extracted from data.gov.sg in a monthly format which reveals the figures from January 1961 to August 2017, while the index reference period is 2014. The data has an overall index representing changes in the price level of the whole basket with all items considered, and can also be drilled down to sub-indices and sub-sub-indices for different categories and sub-categories of goods and services. For our system analysis, we plan to use filtered data from 1990 onwards. | + | The Consumer Price Index (CPI) data is extracted from ''data.gov.sg<ref name="datagov">Government of Singapore. [https://data.gov.sg/dataset/consumer-price-index-monthly?view_id=0063aa5a-c5de-4c74-94be-b9ec443878be&resource_id=67d08d6b-2efa-4825-8bdb-667d23b7285e], Last Updated on 30 November 2017, Retrieved on 30 November 2017</ref>'' in a monthly format which reveals the figures from January 1961 to August 2017, while the index reference period is 2014. The data has an overall index representing changes in the price level of the whole basket with all items considered, and can also be drilled down to sub-indices and sub-sub-indices for different categories and sub-categories of goods and services. For our system analysis, we plan to use filtered data from 1990 onwards. |
| − | == | + | == Methodology == |
| − | === Exploratory Analysis === | + | ==== Exploratory Analysis ==== |
We will explore the different trends of time-series data provided by the various economic data sets (Period cyclicity and seasonality). Different interactions of identified attributes might provide certain data insights that we can use for our analysis. | We will explore the different trends of time-series data provided by the various economic data sets (Period cyclicity and seasonality). Different interactions of identified attributes might provide certain data insights that we can use for our analysis. | ||
| − | === Explanatory Analysis === | + | ==== Explanatory Analysis ==== |
| − | Relationships between our data will be explained based on our understanding of possible real-world events or causes. | + | Relationships between our data will be explained based on our understanding of possible real-world events or causes. Using our CPI use-case as an example, the difference in CPI between the months of June and December can be explained as a result of the holiday seasons causing an increase of demand for clothing in December. |
| − | === Predictive Analysis === | + | ==== Predictive Analysis ==== |
| − | + | We can use analytics techniques such as Exponential Smoothing and ARIMA to predict future trends of our time-series data, due to the data's cyclical and seasonal nature. | |
== Application == | == Application == | ||
| Line 95: | Line 102: | ||
The system should also be able to forecast time-series data that have been filtered out from the Data Exploration feature. The forecasting techniques will utilize Exponential Smoothing and ARIMA techniques to perform predictions. An optimization algorithm will be used along with existing packages to find the best set of parameters and the top 3 models of each technique will be selected based on their AIC, BIC values. Once selected, the models can then be graphed on the page as a comparison. | The system should also be able to forecast time-series data that have been filtered out from the Data Exploration feature. The forecasting techniques will utilize Exponential Smoothing and ARIMA techniques to perform predictions. An optimization algorithm will be used along with existing packages to find the best set of parameters and the top 3 models of each technique will be selected based on their AIC, BIC values. Once selected, the models can then be graphed on the page as a comparison. | ||
| + | |||
| + | == Application Libraries & Packages == | ||
| + | {|class="wikitable" | ||
| + | |- | ||
| + | ! Package Name !! Descriptions | ||
| + | |- | ||
| + | | ''Shiny<ref name="shiny">RStudio.org. [https://shiny.rstudio.com/] "Interact. Analyze. Communicate.", Retrieved on 30 November 2017</ref>'' || Interactive web applications for data visualization | ||
| + | |- | ||
| + | | ''Tidyverse: tidyr, dplyr, ggplot2<ref name="tidyverse">tidyverse.org. [https://www.tidyverse.org] "R packages for data science", Retrieved on 30 November 2017</ref>'' || Tidying and manipulating data for visualizing in ggplot2 | ||
| + | |- | ||
| + | | ''Shinythemes<ref name="shinythemes">Chang, W, RStudio, and etc. [https://cran.r-project.org/web/packages/shinythemes/index.html] "shinythemes: Themes for Shiny", Retrieved on 30 November 2017</ref>'' || Provide consistent UI elements for aesthetics | ||
| + | |- | ||
| + | | ''forecast<ref name="forecast">Hyndman, R, and etc. [https://cran.r-project.org/web/packages/forecast/index.html] "forecast: Forecasting Functions for Time Series and Linear Models", Retrieved on 30 November 2017</ref>, broom, sweep<ref name="timetk">www.business-science.io. [http://www.business-science.io/r-packages.html] "Open Source Software For Business & Financial Analysis", Retrieved on 30 November 2017</ref>'' || Packages used to "tidy" data models for easy forecasting. Forecast package uses ''ts'' objects that is difficult to manipulate. sw_sweep from the sweep package uses broom-style tidiers to extract model infomation into 'tidy' data frames. sweep package also uses timekit at the back-end to maintain the original time series index throughout the whole process. | ||
| + | |- | ||
| + | | ''tibbletime<ref name="tibbletime">Vaughan, D, and etc. [https://cran.r-project.org/web/packages/tibbletime/index.html] "tibbletime: Time Aware Tibbles", Retrieved on 30 November 2017</ref>'' || Time-based data subsetting | ||
| + | |- | ||
| + | | ''lubridate<ref name="lubridate">Spinu, V, and etc. [https://cran.r-project.org/web/packages/lubridate/index.html] "lubridate: Make Dealing with Dates a Little Easier", Retrieved on 30 November 2017</ref>'' || Easy manipulation of datetime data | ||
| + | |- | ||
| + | | ''timetk<ref name="timetk">www.business-science.io. [http://www.business-science.io/r-packages.html] "Open Source Software For Business & Financial Analysis", Retrieved on 30 November 2017</ref>'' || Extracting/checking of datetime index from ts objects | ||
| + | |- | ||
| + | | ''stringr<ref name="stringr">Wickham, H and RStudio. [https://cran.r-project.org/web/packages/stringr/index.html] "stringr: Simple, Consistent Wrappers for Common String Operations", Retrieved on 30 November 2017</ref>'' || String manipulation | ||
| + | |- | ||
| + | | ''DT<ref name="DT">Xie, Y, and etc. [https://cran.r-project.org/web/packages/DT/index.html] "DT: A Wrapper of the Javascript Library 'DataTables'", Retrieved on 30 November 2017</ref>'' || Sortable data table UI element for model accuracy measures | ||
| + | |- | ||
| + | | ''cowplot<ref name="cowplot">Wilke, C, and etc. [https://cran.r-project.org/web/packages/cowplot/index.html] "cowplot: Streamlined Plot Theme and Plot Annotations for 'ggplot2'", Retrieved on 30 November 2017</ref>'' || Graph arrangement of ''ggplots'' in a single renderPlot function | ||
| + | |- | ||
| + | | ''shinycssloaders<ref name="shinycssloaders">Sali, A, and etc. [https://cran.r-project.org/web/packages/shinycssloaders/index.html] "shinycssloaders: Add CSS Loading Animations to 'shiny' Outputs", Retrieved on 30 November 2017</ref>'' || Loading animation for large data loading and model training | ||
| + | |- | ||
| + | |} | ||
== References == | == References == | ||
| − | + | <references/> | |
| − | |||
| − | |||
| − | |||
Latest revision as of 23:19, 30 November 2017
|
|
|
|
|
Contents
Background
Time Series Explorer
Time series data has always played an important role in understanding and evaluating past behaviours. The usage of time-series information has allowed companies and organizations to tune their operations instead of simply improving by trial and error. Time series data have also been used to evaluate whether an organization has grown throughout the years of operation. The analysis techniques for time-bound data help to describe and explore insights, while predicting the future outcomes such as revenues and profits.
This category of data analysis is not only constrained to the business world. A country's economic data is also a rich pool of information that can be analysed for insights. One of the important example measures that can allow us to get a glimpse would be the Consumer Price Index (CPI). Using this easily available measure, we can explore living standards of a country's citizens by understanding the factors affecting their day-to-day lives.
Use Case - Consumer Price Index
The Consumer Price Index (CPI), is a critical indicator to assess a country's consumer price inflation. To profile the weighted average price changes for households' cost of living, Singapore's CPI adopts a fixed basket of residents' commonly consumption goods and services. About 6,600 brands/varieties from 4,200 outlets are selected in the 2014-based CPI (the dataset for our study). At basket level, the composition of goods and services can be categorized into 10 major divisions, which are listed below:
Food, Clothing & Footwear, Housing & Utilities, Household Durables And Services, Health Care,
Transport, Communication, Recreation & Culture, Education, Miscellaneous Goods & Services
To construct the CPI, two main types of data are required - the sample price data of the good or service, and the weighting data to represent the proportion of different categories' expenditure. The price data is gathered through a combination of data collection modes, while the frequency of price collection depends on the price behavior of the item. The weighting data is derived from the expenditure values collected in the Household Expenditure Survey, the latest one came from HES 2012/13, updated to 2014 values by taking into account price changes between 2012/13 and 2014.
As a Price index, CPI can also be affected by other types of costs in Singapore. Indicators like COE Bidding Price[2], Import and Export price index[3], Exchange Rates[4], are assumed to have interesting relationships with CPI which are worthy of further study. These data sources are also acceptable within the proposed system design within certain format restrictions.
Motivation
During our personal data analysis research and experiences, we discovered a lack of freely available analysis tools that can help us optimize the parameter settings of time-series models. The result is a large amount of time and effort utilized to enter in different combinations of parameters and waiting for the models to be trained. The time series system that we have in mind would need to help us estimate the model's accuracy rates automatically while we perform our data analyses, so we can simply choose the best models for comparison.
Our team would also like to create exploratory and predictive models, which showcase these complex time-related trends of Singapore's CPI throughout the years (1990-2017) for different categories. We would like to use time-series visualization techniques such as tables and line charts representing Trend, Seasonality and Random to investigate any insights and to display potential forecasts to the audience.
We will apply system design principles to make the proposed system accept any form of generic time-series data. This is to allow flexibility of the system and expands its scope of usage.
There are several reasons why we found this project interesting:
- General exploration of time-series data. The project allows us to learn and re-learn time-series analysis techniques and concepts. This gives us the opportunity to let us put our theoretical knowledge into practical use.
- Closer to our daily standards of living, we wanted to understand what are the current categories that make up a country's CPI index and subsequently its impact on its citizens.
- We also wanted to explore whether different periods of time would indicate the different price index values in a cyclical manner. This would also allow us to understand the overall direction prices are taking, for every goods & services category in Singapore.
Even though Singapore is a relatively young country, it is able to provide rich data to help us explore these interesting observations. This is also in part due to the government initiative of 'SmartNation.sg'.
Objectives
1. Provide interactive platform to illustrate the trends and seasonalities within given time-series data (i.e. Singapore's CPI).
2. Discover data insights using visualization and interactivity that cannot be easily represented using raw data.
3. Make use of freely available Singapore economic data to arouse the interests of potential viewers and increase their curiosity on the current state of Singapore's consumer goods and services.
Data Source
The Consumer Price Index (CPI) data is extracted from data.gov.sg[5] in a monthly format which reveals the figures from January 1961 to August 2017, while the index reference period is 2014. The data has an overall index representing changes in the price level of the whole basket with all items considered, and can also be drilled down to sub-indices and sub-sub-indices for different categories and sub-categories of goods and services. For our system analysis, we plan to use filtered data from 1990 onwards.
Methodology
Exploratory Analysis
We will explore the different trends of time-series data provided by the various economic data sets (Period cyclicity and seasonality). Different interactions of identified attributes might provide certain data insights that we can use for our analysis.
Explanatory Analysis
Relationships between our data will be explained based on our understanding of possible real-world events or causes. Using our CPI use-case as an example, the difference in CPI between the months of June and December can be explained as a result of the holiday seasons causing an increase of demand for clothing in December.
Predictive Analysis
We can use analytics techniques such as Exponential Smoothing and ARIMA to predict future trends of our time-series data, due to the data's cyclical and seasonal nature.
Application
The proposed system would have three major functions:
Data Manipulation:
System users would be able to upload time-series data files within certain formats. The interface provided would allow some forms of data transformations such as the transposition of columns, the generation of an index column as a substitute for a missing datetime column, and the indication of missing time series periods. These functionalities are required because time series analysis needs data to be indexed by a form of datetime field. Metadata information of the uploaded dataset would be displayed for easy viewing and the uploaded dataset can also be previewed from the main panel.
Data Exploration:
The system should allow the functionality to decompose time-series information into its constituent parts: Observation, Seasonal, Trend, Random (Noise). From the separate parts, users can understand the different time-series patterns and derive insights. The system will also provide several filter functions such as denoting whether the data is additive or multiplicative trend, start and end dates of the data, and the frequency of the time-series periods.
Forecasting:
The system should also be able to forecast time-series data that have been filtered out from the Data Exploration feature. The forecasting techniques will utilize Exponential Smoothing and ARIMA techniques to perform predictions. An optimization algorithm will be used along with existing packages to find the best set of parameters and the top 3 models of each technique will be selected based on their AIC, BIC values. Once selected, the models can then be graphed on the page as a comparison.
Application Libraries & Packages
| Package Name | Descriptions |
|---|---|
| Shiny[6] | Interactive web applications for data visualization |
| Tidyverse: tidyr, dplyr, ggplot2[7] | Tidying and manipulating data for visualizing in ggplot2 |
| Shinythemes[8] | Provide consistent UI elements for aesthetics |
| forecast[9], broom, sweep[10] | Packages used to "tidy" data models for easy forecasting. Forecast package uses ts objects that is difficult to manipulate. sw_sweep from the sweep package uses broom-style tidiers to extract model infomation into 'tidy' data frames. sweep package also uses timekit at the back-end to maintain the original time series index throughout the whole process. |
| tibbletime[11] | Time-based data subsetting |
| lubridate[12] | Easy manipulation of datetime data |
| timetk[10] | Extracting/checking of datetime index from ts objects |
| stringr[13] | String manipulation |
| DT[14] | Sortable data table UI element for model accuracy measures |
| cowplot[15] | Graph arrangement of ggplots in a single renderPlot function |
| shinycssloaders[16] | Loading animation for large data loading and model training |
References
- ↑ Department of Statistics Singapore. [1], "FAQ on Consumer Price Index (CPI)", Retrieved on 30 November 2017
- ↑ Yeap, R. [2], "COE Prices", Retrieved on 30 November 2017
- ↑ CEICData.com [3], "CEIC - A Euromoney Institutional Investor Company", Retrieved on 30 November 2017
- ↑ Monetary Authority of Singapore. [4], "Monetary Authority of Singapore", Retrieved on 30 November 2017
- ↑ Government of Singapore. [5], Last Updated on 30 November 2017, Retrieved on 30 November 2017
- ↑ RStudio.org. [6] "Interact. Analyze. Communicate.", Retrieved on 30 November 2017
- ↑ tidyverse.org. [7] "R packages for data science", Retrieved on 30 November 2017
- ↑ Chang, W, RStudio, and etc. [8] "shinythemes: Themes for Shiny", Retrieved on 30 November 2017
- ↑ Hyndman, R, and etc. [9] "forecast: Forecasting Functions for Time Series and Linear Models", Retrieved on 30 November 2017
- ↑ 10.0 10.1 www.business-science.io. [10] "Open Source Software For Business & Financial Analysis", Retrieved on 30 November 2017
- ↑ Vaughan, D, and etc. [11] "tibbletime: Time Aware Tibbles", Retrieved on 30 November 2017
- ↑ Spinu, V, and etc. [12] "lubridate: Make Dealing with Dates a Little Easier", Retrieved on 30 November 2017
- ↑ Wickham, H and RStudio. [13] "stringr: Simple, Consistent Wrappers for Common String Operations", Retrieved on 30 November 2017
- ↑ Xie, Y, and etc. [14] "DT: A Wrapper of the Javascript Library 'DataTables'", Retrieved on 30 November 2017
- ↑ Wilke, C, and etc. [15] "cowplot: Streamlined Plot Theme and Plot Annotations for 'ggplot2'", Retrieved on 30 November 2017
- ↑ Sali, A, and etc. [16] "shinycssloaders: Add CSS Loading Animations to 'shiny' Outputs", Retrieved on 30 November 2017
