Difference between revisions of "IS428 AY2019-20T1 Assign Sean Chai Shong Hee Data Preprosessing"
Line 44: | Line 44: | ||
== 3. Merging StaticSensorReadings and StaticSensorLocations == | == 3. Merging StaticSensorReadings and StaticSensorLocations == | ||
− | [[File:Merging dataframes.png|thumb]] | + | [[File:Merging dataframes.png|thumb|none]] |
<p>We proceed by merging our static_reading dataframe with our static_loc dataframe. Here, we do a full outer join on "Sensor-id" to ensure all values are accounted for, even if they are null.</p> | <p>We proceed by merging our static_reading dataframe with our static_loc dataframe. Here, we do a full outer join on "Sensor-id" to ensure all values are accounted for, even if they are null.</p> | ||
Revision as of 18:19, 11 October 2019
|
|
|
|
|
Contents
Data Preprocessing
Data was provided in the form of 3 csv files:
- MobileSensorReadings.csv: Consists of measures of radiation values in count per minute. Also contained the longitude and latitude of where the values were collected, and the respective time they were recorded
- StaticSensorReadings.csv: Consists of measure of radiation values in count per minute. Also contained the respective time at which the values were recorded.
- StaticSensorLocations.csv: Consists of the longitude and latitude of each static sensor
A Shapefile was also provided, which contained Polygons of the region of St. Himark
Before any meaningful observations could be made with Tableau, the csv files had to be preprocessed to allow Tableau to carry out a spatial join between the point and polygon data. As such, the longitude and latitude values were converted to Point geometry values using geoPandas. The geoPandas Dataframe was later saved as a Shapefile.
1. Importing python libraries
We first start off by importing libraries that are necessary in our creation of our new Shapefile. These libraries include Pandas, Shapely and GeoPandas.
2. Reading csv files to Pandas Dataframe
Next, we read our csv files to a Pandas Dataframe.
3. Merging StaticSensorReadings and StaticSensorLocations
We proceed by merging our static_reading dataframe with our static_loc dataframe. Here, we do a full outer join on "Sensor-id" to ensure all values are accounted for, even if they are null.
4. Creating our GeoPandas Dataframe
We first convert our Longitude and Latitude into geometry point values. Here, we append the geometry point values into a new column called "geometry" and create our new geo Dataframe
We can see that we now have a new column called "geometry", which contains geometry point values for each of the rows
== 5.