View source for ISSS608 2016-17 T3 Group8 Arules Final Projrct

<div style=background:#881332 border:#A3BFB1>
[[Image:Group eight Logo.png|200px]] 
<b><font size = 6; color="#FFFAF0">A Visual Application for Better Business Decision Making </font></b>

</div>
<!--MAIN HEADER -->
{|style="background-color:#ffefd8;" width="100%" cellspacing="0" cellpadding="0" valign="top" border="0"  |
| style="font-family:Century Gothic; font-size:100%; solid #000000; background:#000000; text-align:center;" width="25%" | 
;
[[ISSS608_2016-17_T3_Group8_Arules_Introduction|<b><font size="2"><font color="#FFFAF0">Introduction</font></font></b>]]

| style="font-family:Century Gothic; font-size:100%; solid #1B338F; background:#000000; text-align:center;" width="25%" | 
;
[[ISSS608_2016-17_T3_Group8_Arules_Project Proposal|<b><font size="2"><font color="#FFFAF0">Project Proposal</font></font></b>]]

| style="font-family:Century Gothic; font-size:100%; solid #1B338F; background:#000000; text-align:center;" width="25%" | 
;
[[ISSS608_2016-17_T3_Group8_Arules_Final Projrct|<b><font size="4"><font color="#FFFAF0">Final Report</font></font></b>]]

| style="font-family:Century Gothic; font-size:100%; solid #1B338F; background:#000000; text-align:center;" width="25%" | 
;
[[ISSS608_2016-17_T3_Group8_Arules_Application| <b><font size="2"><font color="#FFFAF0">Application</font></font></b>]]

|  &nbsp;
|}
<br/>

==Project Motivation==

<b><big>Association Rule Mining is Powerful</big></b><br>
Although association rule mining is usually applied in market basket analysis to mine the relationship between different products, it is actually a very powerful algorithm that can be applied for any dataset to discover the association, correlation and causation between variables. With an interesting target variable, we actually can find out the relevant association rules within the dataset, even if it is not a transaction data. [https://wiki.smu.edu.sg/1617t3isss608g1/ISSS608_2016-17_T3_Group8_Arules_Project_Proposal#Background_on_Association_Rules| Click here to read more on association rules discussed in our proposal]


However, not all datasets are ready to do this kind of data mining in R, for example continuous variables are not easy to handle, which is a barrier for users to step into the world of data analytics. We intend to develop an association rule mining application for non-statistician users to understand easily, to play with and get insights, and to bring them into data analytics using the fundamental yet powerful concept - probabilities.

<b><big>Room for Improvement of Current Packages</big></b><br>

Current R packages available for association rule mining are helpful for data analysts, but their limitations also brings difficulty to interpret the analysis results:

<b>1) Static Visualizations</b><br>
Visualizations provided in current ARM packages are mostly static, which are difficult for users to do see difference between different settings on the rules without editing a line of code. For example, it is hard to see how many rules were kicked-out if support threshold was increased by 0.005. <br>
<br>
<b>2) Lack Interactivity</b><br>
There is little interactivity in the current ARM packages. Users are unable to select or zoom in on the visualizations.<br>
<br>
<b>3) Manual Calibration</b><br>
Users are only able to manually calibrate the generated rules by changing thresholds for support, confidence, lift, antecedent or consequent.
<br>

==R Packages Used==
* For interactive application: R Shiny and Shiny Dashboard
Shiny is an R Studio package for developing interactive charts, data visualizations and applications to be hosted on the web using the R programming language. It enables developer to make an interactive application which allow user to understand a certain model or do some data explorations. In this case, we could visualize the underlying rules beyond given datasets which show a clear picture of how those items correlate with each other.  [https://cran.r-project.org/web/packages/shiny/shiny.pdf Package ‘shiny’][https://cran.r-project.org/web/packages/shinydashboard/shinydashboard.pdf Package ‘shinydashboard’]

*For mining association rules: arules family of packages

“Arules” is the very foundation on which we built this application. “Arules” enables users to apply association rule mining algorithms on transaction data or any other data that meets certain requirement. It is quite powerful at manipulating and transforming data, pruning redundant rules, as well as filtering association rules generated. Users can filter the rules by customizing thresholds for support, confidence, and lift, as well as the antecedent and consequent, and sort the rules by support, lift and confident.  [https://cran.r-project.org/web/packages/arules/arules.pdf Package ‘arules’][https://cran.rstudio.com/web/packages/arulesViz/arulesViz.pdf Package ‘arulesViz’]

*For interactive scatterplot: ggplot2 and plotly [https://cran.r-project.org/web/packages/plotly/plotly.pdf Package ‘plotly’] [https://cran.r-project.org/web/packages/ggplot2/ggplot2.pdf Package ‘ggplot2’]

*For network visualization: visNetwork and igraph [https://cran.r-project.org/web/packages/visNetwork/visNetwork.pdf Package ‘visNetwork’][https://cran.r-project.org/web/packages/igraph/igraph.pdf Package ‘igraph’]

*For data preparation: sqldf(for SQL operations in R),dplyr,stringr [https://cran.r-project.org/web/packages/sqldf/sqldf.pdf Package ‘sqldf’] [https://cran.r-project.org/web/packages/dplyr/dplyr.pdf Package ‘dplyr’] [https://cran.r-project.org/web/packages/stringr/stringr.pdf Package ‘stringr’]

==Choice of Visualizations and Critics==
This section discusses the choices of visualizations used in our application with respect to their usefulness. Critics on the default visualizations provided in the arulesviz packages will be discussed as well to the areas for improvement for our visualization designs.
<table>
<table border='1'>
<tr>
<th>Discussion</th>
<th>Visualization</th>
</tr>
<tr>
<td><b>Benefits of scatterplots</b>
*Good for multivariate comparison of support & confidence, colour by lift
*Good for stats explorer for general MBA – The fact that all the itemsets in the transactions are important to the user, it is good to have an overview of the stats first before the users decide on which rules they wish to further investigate or action upon.
<br>
The scatterplot on the left is generated by the arulesviz package, the limitations are: 
*Too clustered, overlapped dots
*Loss the information of associations
*Manual calibration of 3 interestingness statistics
</td>
<td>[[File:VRshiny_report_grp8_files_image009.png|500px|center]]</td>
</tr>
<tr>
<td> <b>Benefits of network diagrams</b>
*Clearly shows the casual relationship between the LHS items and RHS items (From & to)
*Differentiation of rules & itemsets as both rules and items are represented separately
*Interactions of rules,itemsets & 3 stats – allows us to visualize which rules are more important than others and which items are more popular/unpopular
<br>
The network graph on the left is generated by the arulesviz package, the limitations are: 
*Confusing
*Less room for user interaction
*Loss the information on 3 stats (only can see one, not three together)
*Manual configuration
</td>
<td>[[File:VRshiny_report_grp8_files_image011.png|500px|center]]</td>
</tr>
</table>

==Application Design in Details==

[https://wiki.smu.edu.sg/1617t3isss608g1/ISSS608_2016-17_T3_Group8_Arules_Application| Click here to view our application dashboards designs at a glance]
<br>

<b> 1. Load data </b>
<table>
<table border='1'>
<tr>
<td><b>1)Choose File to Upload</b>
[[File:VRshiny_report_grp8_files_image019.png|100px|center]]
</td>
<td>
Users can upload any dataset they want as long as they are in the following format:
<br>

1.Market Basket Analysis 
*Single Format: Col1 = Transaction ID, Col2 = Item Name(single) 
*Basket Format: Col1 = Transaction ID, cOL2 = Item Names(Multiple) 
<br>

2.Targeted ARM
Any dataset that contains a target variable that the user is interested in.
The users are able to choose if the dataset contains a header and the separator for the file.

</td>
</tr>
<tr>
<td> <b>2.Import Data</b>

[[File:VRshiny_report_grp8_files_image021.png|400px|center]]
</td>
<td>Once the data file is uploaded and the users are ready to do the association rule mining on this dataset, they can click the “Import data” button and then the data would be imported into our server and saved as a data frame, which would be used for all the following data transformation, analyses and visualizations.
<br>

This is done by using “eventReactive” function in R shiny. To save a data frame dependent on user’s uploaded file, we make a reactive data frame that would be stored only when some event happens, which, in this case, is clicking on the “Import data” button.
</td>
</tr>
<tr>
<td> <b>3.Variable Transformation </b>
*Check binary:
<br>binary_check=apply(HRdata,2,function(x) { all(na.omit(x) %in% 0:1) })	
<br>

*Transform binary:
<br>change_to_logical=as.data.frame(lapply(HRdata[binary_check],function(x) as.factor(as.logical(x))))
<br>

*Check numeric:
numeric_check=sapply(df1,is.numeric)
<br>

*Transform numeric:
<br>change_to_factor=as.data.frame(lapply(df1[numeric_check],function(x) factor(ntile(x, 3),levels = c(1,2,3),labels = c("low","mid","high"))))

</td>
<td>
a) Transforming binary column
Columns containing numbers of only 1 or 0 are considered as binary columns. (NA is allowed). Binary variables are recoded to “True” or “False”

b) From the remaining columns, transform numeric columns to 3 bins “low, mid, high” based on quantiles. 

*This step can be improved by allowing the user to choose the number of bins and the naming conventions for each bin.

c) Categorical columns: unchanged
Users were educated in the user guide to not using numeric numbers to represent categorical information, otherwise they will be transformed in step 2.

d) Combing the transformed columns and the original categorical columns back to form the new dataframe for association rule minng.
</td>
</tr>
</table>

<br>

<b> 2. Scatterplots Visualizations: </b>
<table>
<table border='1'>
<tr>
<th>Design Concepts</th>
<th>Visualization</th>
</tr>
<tr>
<td><b>2.1Overview Scatterplot with Box Selection</b>
Once rules are generated, they will be saved as data frame in the server. Each row represents one rule, which consists of the following variables: ruleID, Antecedent, Consequent, support, confidence, lift.
The first visualization that users would see is an overview scatterplot.
<br>

We plot all generated rules on a scatter plot where X axis is support and Y axis is confidence, colored by lift. Optimally, we would focus on those points at up-right corner of the plot and in darker red color, which is interpreted as high support, high confidence and high lift. This is a static scatterplot but we enable users to do box selection on it and they would see: 1) a zoomed-in interactive scatterplot on the right hand for them to further investigate those rules that they are more interested in; 2) a table of rule details for them to browse details of all selected rules; 3) a rule network to visualize all rules selected and the relationship between all related items.
To achieve this function, we add in one parameter in this plotOutput called “function”:
This “brush” function would read where the users click, drag, and release on the plot, show it on the plot as a box selection and store the box’s value range on X and Y axis.

<br>
The box’s value range thus serve as constraints on support and confidence to subset the rules for further visualization.
[[File:VRshiny_report_grp8_files_image027.png|300px|center]]
[[File:VRshiny_report_grp8_files_image029.png|300px|center]]

</td>
<td>[[File:VRshiny_report_grp8_files_image023.png|500px|center]]
<br>
[[File:VRshiny_report_grp8_files_image025.png|400px|center]]</td>
</tr>
<tr>
<td> <b>2.2 Zoomed-in Scatterplot</b>
Our zoomed-in scatterplot is an interactive one, built using “ggplot2” wrapped in “plotly”.
<br>
Within the X, Y value range defined by box selection, users can still further zoom-in or zoom-out on the zoomed-in scatterplot. More importantly, once the user hovers on any point, a tooltip will pop out showing the support, confidence and lift of this rule, and its lhs and rhs content, which gives the user a clearer picture what are there inside the rules they selected.

</td>
<td>[[File:VRshiny_report_grp8_files_image031.png|600px|center]]</td>
</tr>
<tr>
<td> <b>2.3 Rule Details</b>
We also provide a data table below the zoomed-in scatterplot, which gives rule details for users to have a full-screen view of selected rules. This is done by using “brushedPoints” function which subset data based on brushed points.
[[File:VRshiny_report_grp8_files_image034.png|300px|center]]
</td>
<td>[[File:VRshiny_report_grp8_files_image033.png|600px|center]]</td>
</tr>
</table>

<br>

<b>3.Network Visualizations</b>

The network visualizations were built using package “vizNetwork” and “igraph”. The trick to create a network visualization from the association rules is to first visualize the rules using the default network visualization provided by the “arulesviz” package (graph method = “graph”). The arulesviz uses “igraph” package to visualize the rules into network diagram. Next, using the “get.data.frame” function from igraph package will automatically create two data frames, one for the nodes and one for the edges. VizNetwork packages could then be used to create the network visualizations. Viznetwork provides additional customization functionalities to the netowork diagrams like highlight the nearest nodes or allow the users to select certain nodes to zoom in. 

<br>
It shall be highlighted that the vertices dataframe created by the “get.data.frame” function has a little bug. The quality measures are not correctly matched to the rules, with the last rule always showing having blank quality measures. This affects the visualization if we wish to reflect the interestingness of the rules in the network diagram, eg size the rules nodes by “lift”. The bug was corrected by adjusting the dataframes back so that the quality measures are in line with the individual rules. We also created additional columns to colour the rules and the items respectively to better differentiate them in the network diagrams.

<table>
<table border='1'>
<tr>
<td>
[[File:VRshiny_report_grp8_files_image036.png|400px|center]]
</td>
<td>
[[File:VRshiny_report_grp8_files_image037.png|500px|center]]
</td>
</tr>
</table>

<br>
The table below explains how to interpret the network visualizations.

<table>
<table border='1'>
<tr>
<th>Descriptions</th>
<th>Visualization</th>
</tr>
<tr>
<td>
*Grey nodes: individual rules
<br>

*Size of the grey nodes: indicating the interestingness of the rule, the bigger the better. By default, it is set to “lift” which can be changed to “support” or “confidence”
<br>

*Red nodes: indicating individual items, they are linked to the rules by edges with arrows

</td>
<td>[[File:VRshiny_report_grp8_files_image039.png|400px|center]]

</tr>
<tr>
<td> 
Clicking on any rule, the items linking to that rule will be highlighted and the rest will be faded. 
<br>
The arrows show the direction of influences. 

<br> 
*LHS (antecedent) items will have arrows pointing to the rule nodes, indicating they are the factors of influences.

*RHS(consequent) items will have the arrows pointing from the rule nodes, indicating they are the outcome.

</td>
<td>[[File:VRshiny_report_grp8_files_image040.png|500px|center]]</td>
</tr>
</table>

<br>

<b>4.User interactivity of the application</b>
<table>
<table border='1'>
<td>
1.Model calibration (available for both generic MBA dashboard and targeted ARM dashboard)
*Select the “support” threshold with a slide bar
*Select the “confidence” threshold with a slide bar
*Select the number of items to show at the LHS, or as antecedent with a slide bar. This defines the length of the rule

<br>



Below calibrations are only available on the targeted ARM dashboard.

*Filter the RHS or the consequent to display the interested target only, by entering free text. This can be further improved by populating the list of items in the user defined consequent column.

*Choose to size the rule nodes by one quality measure, defaulted as lift with an option list

*Choose the number of rules to display with a slide bar</td>
<td>[[File:VRshiny_report_grp8_files_image046.png|200px|center]]</td>
</table>

==Use Cases==
<b>1.Market Basket Analysis</b>
*The data used was the default "Groceries" data contained in the arules package.
*Threshold: support>0.002, confidence>0.05, length of rule: 3, size of rule nodes by: lift

<table>
<table border='1'>
<tr>
<td>
1.Select the rules with highest confidence & lowest support 
*Items likely to be purchased together but not popularly purchased
[[File:VRshiny_report_grp8_files_image047.png|500px|center]]
</td>
<td>
The rule with highest lift is:
<b>{tropical fruits,whole milk,grapes} =>{other vegetables}</b>
[[File:VRshiny_report_grp8_files_image048.png|400px|center]]
</td>
</tr>
<tr>
<td>
2.Select the rules with highest support at varying confidence
*Most popular combinations of items but not always in the same combinations
[[File:VRshiny_report_grp8_files_image049.png|500px|center]]
</td>
<td>
The rule with highest lift is:
<b>{yoghurt,other vegetables,whole milk} =>{tropical fruit}</b>

[[File:VRshiny_report_grp8_files_image050.png|400px|center]]
</td>
</tr>
<tr>
<td>
3.Select the rules with highest lift
*Most useful rule but not a very popular choice (low support)
[[File:VRshiny_report_grp8_files_image051.png|500px|center]]
</td>
<td>
The rule with highest lift is:
<b>{yoghurt,root vegetables,bottled water} =>{tropical fruit}</b>

[[File:VRshiny_report_grp8_files_image052.png|300px|center]]
</td>
</tr>
</table>

<br>
<b>2.Targeted ARM Using HR Data – Why Did the Employees Leave?</b>
*The data used was the HR Analytics data downloaded from Kaggle on why employees left the company. [https://www.kaggle.com/ludobenistant/hr-analytics Human Resources Analytics]
*Threshold: support>0.001, confidence>0.5, length of rule: 3, RHS: left=”TRUE” (only showing who has left)

<table>
<table border='1'>
<td>
1.The rules with highest lift 
Which factors combining together will lead to employee leaving?
<b>{average monthly hours=low, satisfaction level =low, number of projects =low}</b>

[[File:VRshiny_report_grp8_files_image054.png|350px|center]]
</td>
<td>
2.The most popular reason why people left
The item with linked to most rules -> <b>{satisfaction level=low}</b>
[[File:gr8_image055.png|350px|center]]

</td>
<td>
<br>
3.Zoom to a specific item(reason)
On the contrary to common beliefs, <b>{salary=low}</b> is not one of the most popular reason on why emolyees quit, it is only associated with one rule, whose lift is only in the 13th out of all the 15 rules generated (lift 3.1)
[[File:VRshiny_report_grp8_files_image056.png|350px|center]]
</td>
</table>

==References==
*[http://datastorm-open.github.io/visNetwork/  visNetwork, an R package for interactive network visualization]
*[http://www.kdnuggets.com/2016/04/association-rules-apriori-algorithm-tutorial.html  Association Rules and the Apriori Algorithm: A Tutorial,Annalyn Ng, Ministry of Defence of Singapore]
*[http://www2.rdatamining.com/uploads/5/7/1/3/57136767/rdatamining-slides-association-rules.pdf  Association Rule Mining with R,Yanchang Zhao]
*[https://rpubs.com/sbushmanov/180410  Market basket analysis,S. Bushmanov]
*[https://www.interworks.com/blog/modonnell/2016/01/15/market-basket-analysis-using-r-and-shiny  Market Basket Analysis Using R and Shiny, Maureen O'Donnell]
*[https://shiny.rstudio.com/articles/layout-guide.html  Shiny Application layout guide,JJ Allaire]
*[https://gallery.shinyapps.io/105-plot-interaction-zoom/  Zoomable plots in Shiny, RStudio, Inc.]
*[https://cran.r-project.org/web/packages/visNetwork/vignettes/Introduction-to-visNetwork.html Introduction to visNetwork,B. Thieurmel - DataStorm]
*[https://gist.github.com/timelyportfolio/762aa11bb5def57dc27f  Interactive arules with arulesViz and visNetwork,timelyportfolio]
*[http://kateto.net/network-visualization  Network visualization with R Workshop,Katya Ognyanova]
*[https://shiny.rstudio.com/articles/selecting-rows-of-data.html Selecting rows of data R Shiny]
*[http://mhahsler.github.io/arules/reference/write.html Write Transactions or Associations to a File]
*[https://shiny.rstudio.com/reference/shiny/latest/observeEvent.html Event handler R Shiny]
*[https://shiny.rstudio.com/articles/action-buttons.html Using Action Buttons R Shiny]
*[https://stackoverflow.com/questions/7531868/how-to-rename-a-single-column-in-a-data-frame Create an object for storing reactive values R Shiny]

==Acknowledgements==
Thanks Prof Kam for the guidance on our project! Special thanks to the kind souls on Stackoverflow for clearing the pieces of obstacles blocking our way. 

*[https://stackoverflow.com/questions/4126326/how-to-quickly-form-groups-quartiles-deciles-etc-by-ordering-columns-in-a How to quickly form groups (quartiles, deciles, etc) by ordering column(s) in a data frame]
*[https://stackoverflow.com/questions/12763890/exclude-blank-and-na-in-r Exclude Blank and NA in R]
*[https://stackoverflow.com/questions/7669767/how-to-bin-ordered-data-by-percentile-for-each-id-in-r-dataframe-r How to bin ordered data by percentile for each id in R dataframe]
*[https://stackoverflow.com/questions/17654913/check-if-variable-has-the-value Check if variable has the value '']
*[https://stackoverflow.com/questions/2175809/alternative-to-is-null-in-r alternative to “!is.null()” in R]
*[https://stackoverflow.com/questions/36424644/how-to-update-the-file-related-to-a-fileinput-variable-in-r-shiny-without-user How to update the file related to a `fileInput` variable in R Shiny without user interaction?]
*[https://stackoverflow.com/questions/31790908/how-to-detect-binary-categorical-columns-coded-as-discrete-numeric-in-r How to detect binary & categorical columns coded as discrete numeric in r]
*[https://stackoverflow.com/questions/11728419/using-cut-and-quartile-to-generate-breaks-in-r-function Using CUT and Quartile to generate breaks in R function]
*[https://stackoverflow.com/questions/41409890/interactive-selection-in-ggplotly-with-geom-hex-scatterplot Interactive selection in ggplotly with geom_hex() scatterplot]
*[https://stackoverflow.com/questions/24964884/convert-arules-transaction-data-to-an-item-matrix-in-r-programming Convert arules Transaction Data to an item matrix in R programming]
*[https://stackoverflow.com/questions/31658486/using-the-lapply-and-sapply-functions-in-r-instead-of-a-nested-loop Using the lapply and sapply functions in R instead of a nested loop]
*[https://stackoverflow.com/questions/7669767/how-to-bin-ordered-data-by-percentile-for-each-id-in-r-dataframe-r How to bin ordered data by percentile for each id in R dataframe]
*[https://stackoverflow.com/questions/3452205/subsetting-a-data-frame-based-on-contents-of-another-data-frame Subsetting a data frame based on contents of another data frame]
*[https://stackoverflow.com/questions/38850629/r-subset-a-column-in-data-frame-based-on-another-data-frame-list R subset a column in data frame based on another data frame/list]
*[https://stackoverflow.com/questions/5234117/how-to-drop-columns-by-name-in-a-data-frame How to drop columns by name in a data frame]
*[https://stackoverflow.com/questions/41667116/obtain-observations-in-geom-hex-in-a-single-plot-shiny Obtain observations in geom_hex in a single plot (Shiny)]
*[https://stackoverflow.com/questions/17284615/plotting-a-hex-bin-in-r-and-ggplot2-using-a-continuous-z-fill-variable Plotting a hex bin in R and ggplot2 using a continuous Z fill variable]
*[https://stackoverflow.com/questions/14495111/setting-hex-bins-in-ggplot2-to-same-size/14586941#14586941 Setting hex bins in ggplot2 to same size]