IS428 AY2019-20T1 Assign Sean Chai Shong Hee Data Preprosessing

Problem And Motivation

Data Preprocessing

Interactive Visualisation

Interesting Anomalies & Observations

References

DATA PREPROCESSING

Data was provided in the form of 3 csv files:

• MobileSensorReadings.csv: Consists of measures of radiation values in count per minute. Also contained the longitude and latitude of where the values were collected, and the respective time they were recorded
• StaticSensorReadings.csv: Consists of measure of radiation values in count per minute. Also contained the respective time at which the values were recorded.
• StaticSensorLocations.csv: Consists of the longitude and latitude of each static sensor

A polygon Shapefile was also provided, which provides the spatial representation of St. Himark

Before any meaningful observations can be made with Tableau, the csv files have to preprocessed to allow Tableau to carry out a spatial join between point and polygon geometric data. As such, the longitude and latitude values needs to be converted to Point geometry values using geoPandas. The geoPandas Dataframe is later saved as a Shapefile.

1. Importing python libraries

We first start off by importing libraries that are necessary in our creation of our new Shapefile. These libraries include Pandas, Shapely and GeoPandas.

2. Reading csv files to Pandas Dataframe

Next, we read our csv files to a Pandas Dataframe.

3. Merging StaticSensorReadings and StaticSensorLocations

We proceed by merging our static_reading dataframe with our static_loc dataframe. Here, we do a full outer join on "Sensor-id" to ensure all values are accounted for, even if they are null.

4. Creating our GeoPandas Dataframe

We first convert our Longitude and Latitude into geometry point values. Here, we append the geometry point values into a new column called "geometry" and create our new geo Dataframe

We can see that we now have a new column called "geometry", which contains geometry point values for each of the rows

5. Exporting our Dataframe to a Shapefile

Finally, we export our geo Dataframe to a Shapefile. This Shapefile can be later used for spatial join with the provided "StHimark.shp"

6. Creating Shapefile for MobileSensorReadings

Following the steps above, we do the same for "MobileSensorReadings.csv" and generate our mobile sensors Shapefile.