Difference between revisions of "IS428 AY2019-20T1 Assign Christine Data Analysis Transformation"
Line 32: | Line 32: | ||
Before using the raw dataset downloaded, this section will elaborate on the dataset analysis and transformation process in order to prepare the data for import and analysis can be conducted. | Before using the raw dataset downloaded, this section will elaborate on the dataset analysis and transformation process in order to prepare the data for import and analysis can be conducted. | ||
<br/> | <br/> | ||
− | ==<div><font color=#100c08 face="Century Gothic"><b>Combine Static Readings and | + | |
− | + | ==<div><font color=#100c08 face="Century Gothic"><b>COMBINE STATIC READINGS AND STATIC LOCATIONS</b></font></div>== | |
+ | File used: '''StaticSensorReadings.csv''' and '''StaticSensorLocations.csv''' | ||
+ | <div id="Fb.1">[[File:christine.2016_figure_b.1_staticReadingLocations.png|300px|frameless|center]] | ||
+ | <center>''Figure b.1 - Combine Static Readings and Locations''</center></div> | ||
+ | Both files will be joined through the Applied Join Clauses by using Sensor-id in both files ([[#Fb.2|Figure b.2]]). Through this Join clause, tidier data (Tall & Skinny) can be generated. | ||
+ | <div id="Fb.2">[[File:christine.2016_figure_b.2_staticIDCleaning.png|300px|frameless|center]] | ||
+ | <center>''Figure b.2 - Clause in Join and Calculated Field''</center></div> | ||
+ | Because of on the next few steps, this static sensor readings will be combined with mobile sensor readings; hence, to avoid confusion on the Sensor-id, I decided to add identifier (e.g. Static) to the initial Sensor-id. Initially, Sensor-id was in Integer (numeric) format; in order to add String to Integer, Sensor-id need to be changed to String type. | ||
+ | On next step needs to create the Calculated field, by adding a calculated field named “SensorID_Static” which consists of Sensor-id along with the word '''‘Static’'''. Final look of table will be as [[#Fb.3|Figure b.3]]. | ||
+ | <div id="Fb.3">[[File:christine.2016_figure_b.3_afterProcess.png|500px|frameless|center]] | ||
+ | <center>''Figure b.3 - Final look of table''</center></div> | ||
<br/> | <br/> | ||
==<div><font color=#100c08 face="Century Gothic"><b>DATA CLEANING ON MOBILE SENSOR READINGS</b></font></div>== | ==<div><font color=#100c08 face="Century Gothic"><b>DATA CLEANING ON MOBILE SENSOR READINGS</b></font></div>== | ||
+ | |||
<br/> | <br/> | ||
Line 44: | Line 55: | ||
==<div><font color=#100c08 face="Century Gothic"><b>FINAL WORKFLOW AND DATASET</b></font></div>== | ==<div><font color=#100c08 face="Century Gothic"><b>FINAL WORKFLOW AND DATASET</b></font></div>== | ||
<br/> | <br/> | ||
+ | [[#Fb.1|Figure b.1]] |
Revision as of 17:09, 10 October 2019
The very first step of analysis is located at the data cleaning and transformation so that it can bring value to the analysis conducted. Raw dataset in zip file is downloaded from VAST Challenge – Mini Challenge 2 are as follow:
- StHimarkNeighborhoodShapefile Folder (consist of shape file that allow user to use the map file in geometry format along with the ID and name of neighbourhood)
- StaticSensorReadings.csv (contains multiple Static Sensor ID respectively with the sensor readings over a period of time)
- StaticSensorLocations.csv (contains multiple Static Sensor ID respectively with the its location)
- MobileSensorReadings.csv (contains multiple Mobile Sensor ID respectively with the sensor readings and its location over a period of time)
Before using the raw dataset downloaded, this section will elaborate on the dataset analysis and transformation process in order to prepare the data for import and analysis can be conducted.
Contents
COMBINE STATIC READINGS AND STATIC LOCATIONS
File used: StaticSensorReadings.csv and StaticSensorLocations.csv
Both files will be joined through the Applied Join Clauses by using Sensor-id in both files (Figure b.2). Through this Join clause, tidier data (Tall & Skinny) can be generated.
Because of on the next few steps, this static sensor readings will be combined with mobile sensor readings; hence, to avoid confusion on the Sensor-id, I decided to add identifier (e.g. Static) to the initial Sensor-id. Initially, Sensor-id was in Integer (numeric) format; in order to add String to Integer, Sensor-id need to be changed to String type. On next step needs to create the Calculated field, by adding a calculated field named “SensorID_Static” which consists of Sensor-id along with the word ‘Static’. Final look of table will be as Figure b.3.
DATA CLEANING ON MOBILE SENSOR READINGS
COMBINE STATIC SENSORS AND MOBILE SENSORS' READINGS