Difference between revisions of "ISSS608 2017-18 T3 Assign Lu Yanzhang Data Preparation Methodology"
(Created page with "<div style=background:#2B3856 border:#A3BFB1> left|250px <font size = 6; font face: "Arial";font color = #FFFFF0>VAST Challenge 2018 MC3: <br>Who hurts...") |
|||
| (21 intermediate revisions by the same user not shown) | |||
| Line 6: | Line 6: | ||
<!--MAIN HEADER --> | <!--MAIN HEADER --> | ||
{|style="background-color:#1B338F;" width="100%" cellspacing="0" cellpadding="0" valign="top" border="0" | | {|style="background-color:#1B338F;" width="100%" cellspacing="0" cellpadding="0" valign="top" border="0" | | ||
| − | | style="font-family:Century Gothic; font-size:100%; solid #000000; background:#2B3856; text-align:center;" width=" | + | | style="font-family:Century Gothic; font-size:100%; solid #000000; background:#2B3856; text-align:center;" width="25%" | |
; | ; | ||
[[ISSS608_2017-18_T3_Assign_Lu_Yanzhang| <font color="#FFFFFF">'''INTRODUCTION'''</font>]] | [[ISSS608_2017-18_T3_Assign_Lu_Yanzhang| <font color="#FFFFFF">'''INTRODUCTION'''</font>]] | ||
| − | | style="font-family:Century Gothic; font-size:100%; solid #1B338F; background:#2B3856; text-align:center;" width=" | + | | style="font-family:Century Gothic; font-size:100%; solid #1B338F; background:#2B3856; text-align:center;" width="25%" | |
; | ; | ||
[[ISSS608_2017-18_T3_Assign_Lu_Yanzhang_Data Preparation_Methodology| <font color="#FFFFFF">'''DATA PREPARATION & METHODOLOGY'''</font>]] | [[ISSS608_2017-18_T3_Assign_Lu_Yanzhang_Data Preparation_Methodology| <font color="#FFFFFF">'''DATA PREPARATION & METHODOLOGY'''</font>]] | ||
| − | | style="font-family:Century Gothic; font-size:100%; solid #1B338F; background:#2B3856; text-align:center;" width=" | + | | style="font-family:Century Gothic; font-size:100%; solid #1B338F; background:#2B3856; text-align:center;" width="25%"| |
; | ; | ||
[[ISSS608_2017-18_T3_Assign_Lu_Yanzhang_Insights| <font color="#FFFFFF">'''OBSERVATION AND INSIGHTS'''</font>]] | [[ISSS608_2017-18_T3_Assign_Lu_Yanzhang_Insights| <font color="#FFFFFF">'''OBSERVATION AND INSIGHTS'''</font>]] | ||
| − | | style="font-family:Century Gothic; font-size:100%; solid #1B338F; background:# | + | | style="font-family:Century Gothic; font-size:100%; solid #1B338F; background:#2B3856; text-align:center;" width="25%" | |
| − | + | ; | |
| − | [[ | + | [[Assignment_Dropbox_G1| <font color="#FFFFFF">Back to Dropbox</font>]] |
| | | | ||
|} | |} | ||
<br/> | <br/> | ||
| + | |||
| + | ==Tools== | ||
| + | The following tools have been used in this assignment | ||
| + | |||
| + | 1. '''Python''' - Timestamp transformation and new data source generation. | ||
| + | |||
| + | The following packages are used in this assignment: | ||
| + | pandas, numpy, glob, datetime. | ||
| + | |||
| + | 2. '''JMP Pro''' - Data preparation | ||
| + | |||
| + | 3. '''Tableau''' - Visualization | ||
| + | |||
| + | 4. '''Gephi''' - Social network modeling and visualization | ||
| + | |||
| + | ==Timestamp Transformation in Python== | ||
| + | The raw timestamp format is the second record from '2015-05-11 14:00:00'. | ||
| + | |||
| + | For further use of timestamp data, the format needs to be transformed to '''YYYY/MM//DD''' rather than raw second format. | ||
| + | |||
| + | ==Join operation among diverse tables in JMP== | ||
| + | |||
| + | Join the tables where the source or target is “suspicious” and select out the suspicious transactions for the further visualizations in Tableau and social network analytics in Gephi | ||
| + | |||
| + | ==Social network modeling in Gephi== | ||
| + | Import the suspicious data file into Gephi and model the data with two methodologies: | ||
| + | |||
| + | 1. Eigenvalue centrality for vertex importance calculation. | ||
| + | |||
| + | 2. Modularity for clustering calculation. | ||
| + | |||
| + | ==Visualization in Tableau== | ||
| + | |||
| + | 1. Visualize the communication table by day and by month to interpret the growth from 2015 to 2017. | ||
| + | |||
| + | 2. Visualize the suspicious staffs' activities. | ||
Latest revision as of 14:52, 10 July 2018
VAST Challenge 2018 MC3:
Who hurts the brid?
|
|
|
|
|
Contents
Tools
The following tools have been used in this assignment
1. Python - Timestamp transformation and new data source generation.
The following packages are used in this assignment: pandas, numpy, glob, datetime.
2. JMP Pro - Data preparation
3. Tableau - Visualization
4. Gephi - Social network modeling and visualization
Timestamp Transformation in Python
The raw timestamp format is the second record from '2015-05-11 14:00:00'.
For further use of timestamp data, the format needs to be transformed to YYYY/MM//DD rather than raw second format.
Join operation among diverse tables in JMP
Join the tables where the source or target is “suspicious” and select out the suspicious transactions for the further visualizations in Tableau and social network analytics in Gephi
Social network modeling in Gephi
Import the suspicious data file into Gephi and model the data with two methodologies:
1. Eigenvalue centrality for vertex importance calculation.
2. Modularity for clustering calculation.
Visualization in Tableau
1. Visualize the communication table by day and by month to interpret the growth from 2015 to 2017.
2. Visualize the suspicious staffs' activities.
