Difference between revisions of "IS428 AY2019-20T1 Assign Chye Soon Hang Data Cleaning"
Jump to navigation
Jump to search
VAST Challenge 2019 MC 2: Managing Radiation Together As A Nation
Shchye.2017 (talk | contribs) |
Shchye.2017 (talk | contribs) |
||
| Line 41: | Line 41: | ||
<li>Join StaticSensorLocations.csv and StaticSensorReadings.csv via the Join function</li> | <li>Join StaticSensorLocations.csv and StaticSensorReadings.csv via the Join function</li> | ||
<ul> | <ul> | ||
| − | <li>Set Join type to be Inner Join and Join Clause to be matching Sensor-id | + | <li>Set Join type to be Inner Join and Join Clause to be matching Sensor-id<br> |
| − | [[File:JoinStatic.png| | + | [[File:JoinStatic.png|300px]] |
</li> | </li> | ||
<li>This will help to give each static sensor record its respective locations (Long, Lat)</li> | <li>This will help to give each static sensor record its respective locations (Long, Lat)</li> | ||
| − | <li>Due to conflicting sensor ids with mobile sensors, I have change all the static sensor id to start from 60 to 68.</li> | + | <li>Due to conflicting sensor ids with mobile sensors, I have change all the static sensor id to start from 60 to 68.<br> |
| + | [[File:StaticSensorID.png|200px]]</li> | ||
</ul> | </ul> | ||
| + | <li>Merge Static Sensors Records together with Mobile Sensors Records via the Union Function<br> | ||
| + | [[File:CSVMerge.png|270px]] | ||
| + | </li> | ||
| + | <li>Export combined dataset via Output function | ||
| + | <ul> | ||
| + | <li>Set the export version to .hyper so that the accessing speed will be much faster compared to .csv. | ||
| + | <br> | ||
| + | [[File:PrepOutput.png|270px]] | ||
| + | </li> | ||
| + | </ul> | ||
| + | </li> | ||
| + | </ol> | ||
| + | |||
| + | <b>Final Workflow in Tableau Prep</b><br> | ||
| + | [[File:Workflow.png|600px]] | ||
| + | |||
| + | <h2>Importing Combined Dataset and StHimark.shp into Tableau Desktop</h2> | ||
| + | <ol> | ||
| + | <li>Join the CombinedDataset with StHimark.shp | ||
| + | <ul> | ||
| + | <li>Via Left Join Function as we want to keep the unmatched records in CombinedDataset (Static Records)</li> | ||
| + | <li>Join Clause will be MAKEPOINT([Lat], [Long]) from CombinedDataset intersecting with Geometry from StHimark.shp | ||
| + | <br> | ||
| + | [[File:MakePoint.png|270px]] | ||
| + | <br> | ||
| + | [[File:LeftJoin.png|320px]] | ||
| + | </li> | ||
| + | </ul> | ||
| + | </li> | ||
| + | <li>Set connection to be Extract for better loading speed</li> | ||
</ol> | </ol> | ||
Latest revision as of 12:49, 13 October 2019
Data Cleaning
This dashboard is supported by the following 4 datasets, given in CSV and SHP formats:
- StaticSensorLocations.csv – contains locations of the static sensors
- StaticSensorReadings.csv – contains readings from 9 static sensors that are situated either around hospitals if not somewhere near to Always Safe Nuclear Power Plant
- MobileSensorReadings.csv - contains readings from 50 mobile sensors that are attached to cars
- StHimark.shp – contains geometry and sensor id for individual neighborhood
Data Transformation on Tableau Prep
- Import all CSV files on to Tableau Prep
- Join StaticSensorLocations.csv and StaticSensorReadings.csv via the Join function
- Set Join type to be Inner Join and Join Clause to be matching Sensor-id
- This will help to give each static sensor record its respective locations (Long, Lat)
- Due to conflicting sensor ids with mobile sensors, I have change all the static sensor id to start from 60 to 68.
- Merge Static Sensors Records together with Mobile Sensors Records via the Union Function
- Export combined dataset via Output function
- Set the export version to .hyper so that the accessing speed will be much faster compared to .csv.
- Set the export version to .hyper so that the accessing speed will be much faster compared to .csv.
Final Workflow in Tableau Prep
Importing Combined Dataset and StHimark.shp into Tableau Desktop
- Join the CombinedDataset with StHimark.shp
- Via Left Join Function as we want to keep the unmatched records in CombinedDataset (Static Records)
- Join Clause will be MAKEPOINT([Lat], [Long]) from CombinedDataset intersecting with Geometry from StHimark.shp
- Set connection to be Extract for better loading speed
