Difference between revisions of "IS428 AY2019-20T1 Assign Victor Lin Data"

From Visual Analytics for Business Intelligence
Jump to navigation Jump to search
(Created page with " {|style="background-color:#25324c;" width="100%" cellspacing="0" cellpadding="0" valign="top" border="0" | | style="font-family:Century Gothic; font-size:100%; solid #25324c...")
 
 
Line 1: Line 1:
 +
<div style=background:#25324c border:#A3BFB1>
 +
[[File:NuclearPlant.jpg|300px]]
 +
<b><font size = 5; color="#FFFFFF">VAST Challenge MC2: Citizen Science to the Rescue </font></b>
 +
</div>
  
 
{|style="background-color:#25324c;" width="100%" cellspacing="0" cellpadding="0" valign="top" border="0"  |
 
{|style="background-color:#25324c;" width="100%" cellspacing="0" cellpadding="0" valign="top" border="0"  |
Line 18: Line 22:
 
;
 
;
 
|}
 
|}
 +
 +
==Data Provided==
 +
The data provided contains readings (from 12 am on April 6, 2020 to 11:59 pm on April 10, 2020), the locations of the sensors, and the map of the neighbourhood:
 +
{| class="wikitable"
 +
|-
 +
! Dataset !! Data Attributes
 +
|-
 +
| StaticSensorLocations.csv ||
 +
* Sensor-id: unique identifier of sensor
 +
* Lat
 +
* Long
 +
|-
 +
| StaticSensorReadings.csv ||
 +
* Timestamp: 5 second intervals
 +
* Sensor-id
 +
* Value (Measurement of radiation)
 +
* Units (Counts per minute)
 +
 +
|-
 +
| MobileSensorReadings.csv||
 +
* Timestamp: 5 second intervals
 +
* Sensor-id
 +
* Long
 +
* Lat
 +
* Value (Measurement of radiation)
 +
* Units (Counts per minute)
 +
* User-id
 +
|-
 +
| StHimarkNeighbourhoodShapefile||
 +
* Shapefile for the city of St. Himark
 +
|}
 +
 +
<br>
 +
 +
==Data Preparation==
 +
=== Combining the Static Sensor Readings with the Static Sensor Locations===
 +
[[File:StaticDataFusion.png|700px|frameless|center]]
 +
Within the static sensor readings data, there was no information on where the sensors are. Using the Union Tool, each sensor is now assigned a specific latitude and longitude value based on sensor-id. The resulting output is the saved down as a tableau extract (.tde file).
 +
 +
<br>
 +
===Combining Static Sensor Data with Mobile Sensor Data===
 +
In order to be able to analyse the data in its entirety, the two datasets have to be fused before being put through '''Tableau'''. As such, the data is prepped and combined in this step:
 +
{| class="wikitable"
 +
|-
 +
| [[File:Fusion1.png|500px|frameless|center]]
 +
|| First, a new calculated field is created to avoid confusion of the sensor IDs, since both the static and mobile sensors use numeric IDs. The "S" prefix is added to the sensor ID for static sensors. The columns containing the old sensor IDs are then removed from the dataset to prep the dataset for merging.
 +
|-
 +
| [[File:Fusion2.png|500px|frameless|center]]
 +
|| Next, the same is done for mobile sensor data. A new calculated field is created and a "M" prefix is added to the sensor IDs to differentiate them from their static counterparts. The old column containing the numeric ID is then removed.
 +
|-
 +
| [[File:Fusion3.png|500px|frameless|center]]
 +
|| Finally, the two datasets are combined into one large dataset using the union tool. The "Units" column is also removed from the dataset as it does not value-add to the analysis of the data. The resulting output is the saved down as a tableau extract (.tde file).
 +
|}
 +
 +
<br>
 +
===Aggregation of Data Values===
 +
With over 4 million rows of data in the combined dataset, the running time for Tableau will take very long to process a query.
 +
[[File:Aggregation.png|700px|frameless|center]]
 +
In order to reduce the total number of rows, I aggregated the data by taking the average of the readings within a 1-minute interval for each sensor ID. The assumption made here was that the mobile sensors will remain within the same location during the 1-minute intervals. 

Latest revision as of 22:10, 13 October 2019

NuclearPlant.jpg VAST Challenge MC2: Citizen Science to the Rescue

Overview

Data

Interactive Visualisation

Observations & Findings

References

Data Provided

The data provided contains readings (from 12 am on April 6, 2020 to 11:59 pm on April 10, 2020), the locations of the sensors, and the map of the neighbourhood:

Dataset Data Attributes
StaticSensorLocations.csv
  • Sensor-id: unique identifier of sensor
  • Lat
  • Long
StaticSensorReadings.csv
  • Timestamp: 5 second intervals
  • Sensor-id
  • Value (Measurement of radiation)
  • Units (Counts per minute)
MobileSensorReadings.csv
  • Timestamp: 5 second intervals
  • Sensor-id
  • Long
  • Lat
  • Value (Measurement of radiation)
  • Units (Counts per minute)
  • User-id
StHimarkNeighbourhoodShapefile
  • Shapefile for the city of St. Himark


Data Preparation

Combining the Static Sensor Readings with the Static Sensor Locations

StaticDataFusion.png

Within the static sensor readings data, there was no information on where the sensors are. Using the Union Tool, each sensor is now assigned a specific latitude and longitude value based on sensor-id. The resulting output is the saved down as a tableau extract (.tde file).


Combining Static Sensor Data with Mobile Sensor Data

In order to be able to analyse the data in its entirety, the two datasets have to be fused before being put through Tableau. As such, the data is prepped and combined in this step:

Fusion1.png
 First, a new calculated field is created to avoid confusion of the sensor IDs, since both the static and mobile sensors use numeric IDs. The "S" prefix is added to the sensor ID for static sensors. The columns containing the old sensor IDs are then removed from the dataset to prep the dataset for merging.
Fusion2.png
 Next, the same is done for mobile sensor data. A new calculated field is created and a "M" prefix is added to the sensor IDs to differentiate them from their static counterparts. The old column containing the numeric ID is then removed.
Fusion3.png
 Finally, the two datasets are combined into one large dataset using the union tool. The "Units" column is also removed from the dataset as it does not value-add to the analysis of the data. The resulting output is the saved down as a tableau extract (.tde file).


Aggregation of Data Values

With over 4 million rows of data in the combined dataset, the running time for Tableau will take very long to process a query.

Aggregation.png

In order to reduce the total number of rows, I aggregated the data by taking the average of the readings within a 1-minute interval for each sensor ID. The assumption made here was that the mobile sensors will remain within the same location during the 1-minute intervals. 

Retrieved from ‘https://wiki.smu.edu.sg/1920t1is428g1/index.php?title=IS428_AY2019-20T1_Assign_Victor_Lin_Data&oldid=6588’