Difference between revisions of "IS428 AY2019-20T1 Assign Sean Chai Shong Hee Data Preprosessing"

From Visual Analytics for Business Intelligence
Jump to navigation Jump to search
Line 22: Line 22:
 
|}
 
|}
  
−
== Data Preprocessing ==
+
== Data Preprocessing with Pandas ==
−
Data was provided in the form of 3 csv files:
+
<p>Data was provided in the form of 3 csv files:</p>
 +
<br>
 
*MobileSensorReadings.csv: Consists of measures of radiation values in count per minute. Also contained the longitude and latitude of where the values were collected, and the respective time they were recorded
 
*MobileSensorReadings.csv: Consists of measures of radiation values in count per minute. Also contained the longitude and latitude of where the values were collected, and the respective time they were recorded
 
*StaticSensorReadings.csv: Consists of measure of radiation values in count per minute. Also contained the respective time at which the values were recorded.  
 
*StaticSensorReadings.csv: Consists of measure of radiation values in count per minute. Also contained the respective time at which the values were recorded.  
Line 30: Line 31:
 
<p>A polygon Shapefile was also provided, which provides the spatial representation of St. Himark</p>
 
<p>A polygon Shapefile was also provided, which provides the spatial representation of St. Himark</p>
  
−
<p>Before any meaningful observations can be made with Tableau, the csv files have to preprocessed to allow Tableau to carry out a spatial join between point and polygon geometric data. As such, the longitude and latitude values needs to be converted to Point geometry values using geoPandas. The geoPandas Dataframe is later saved as a Shapefile.</p>
+
<p>Before any meaningful observations can be made with Tableau, the csv files have to preprocessed to allow Tableau to carry out a spatial join between point and polygon geometric data. As such, the longitude and latitude values needs to be converted to Point geometry values using GeoPandas. The GeoDataframe is later exported into a Shapefile.</p>
  
 
=== Importing python libraries ===
 
=== Importing python libraries ===
 
[[File:Importing relevant libraries.png|thumb|none]]
 
[[File:Importing relevant libraries.png|thumb|none]]
−
<p>We first start off by importing libraries that are necessary in our creation of our new Shapefile. These libraries include Pandas, Shapely and GeoPandas.</p>
+
<p>I first start off by importing libraries that are necessary in the creation of the new Shapefiles. These libraries include Pandas, Shapely and GeoPandas.</p>
  
 
=== Reading csv files to Pandas Dataframe ===
 
=== Reading csv files to Pandas Dataframe ===
 
[[File:Reading StaticSensorReadings.csv and StaticSensorLocations.csv.png|thumb|none]]
 
[[File:Reading StaticSensorReadings.csv and StaticSensorLocations.csv.png|thumb|none]]
−
<p>Next, we read our csv files to a Pandas Dataframe.</p>
+
<p>Next, I read the csv files into a Pandas Dataframe.</p>
  
 
=== Merging StaticSensorReadings and StaticSensorLocations ===
 
=== Merging StaticSensorReadings and StaticSensorLocations ===
 
[[File:Merging dataframes.png|thumb|none]]
 
[[File:Merging dataframes.png|thumb|none]]
−
<p>We proceed by merging our static_reading dataframe with our static_loc dataframe. Here, we do a full outer join on "Sensor-id" to ensure all values are accounted for, even if they are null.</p>
+
<p>I proceed by merging the static_reading dataframe with the static_loc dataframe. Here, a full outer join on "Sensor-id" is done to ensure all values are accounted for, even if they are null.</p>
  
−
=== Creating our GeoPandas Dataframe ===  
+
=== Creating the GeoDataframe ===  
 
[[File:GeoPandas Dataframe.png|thumb|none]]
 
[[File:GeoPandas Dataframe.png|thumb|none]]
−
<p>We first convert our Longitude and Latitude into geometry point values. Here, we append the geometry point values into a new column called "geometry" and create our new geo Dataframe</p>
+
<p>Longitude and Latitude values are first converted into geometry point values. Here, I append the geometry point values into a new column called "geometry" and create my new GeoDataframe</p>
  
 
[[File:Sample of dataframe.png|thumb|none]]
 
[[File:Sample of dataframe.png|thumb|none]]
−
<p>We can see that we now have a new column called "geometry", which contains geometry point values for each of the rows</p>
+
<p>We can see that I now have a new column called "geometry", which contains geometry point values for each of the rows</p>
  
 
=== Exporting our Dataframe to a Shapefile ===  
 
=== Exporting our Dataframe to a Shapefile ===  
 
[[File:Exporting Shapefile.png|thumb|none]]
 
[[File:Exporting Shapefile.png|thumb|none]]
−
<p>Finally, we export our geo Dataframe to a Shapefile. This Shapefile can be later used for spatial join with the provided "StHimark.shp"</p>
+
<p>Finally, I export my GeoDataframe to a Shapefile. This Shapefile can be later used for spatial join with the provided "StHimark.shp"</p>
  
 
=== Creating Shapefile for MobileSensorReadings ===
 
=== Creating Shapefile for MobileSensorReadings ===
 
[[File:Creating MobileSensorReadings Shapefile.png|thumb|none]]
 
[[File:Creating MobileSensorReadings Shapefile.png|thumb|none]]
−
<p>Following the steps above, we do the same for "MobileSensorReadings.csv" and generate our mobile sensors Shapefile.</p>
+
<p>Following the steps above, I did the same for "MobileSensorReadings.csv" and generated my mobile sensors Shapefile.</p>
 +
 
 +
 
 +
== Tableau ==
 +
<p> With my new Shapefile created, I will now begin to work on visualisations in Tableau.</p>
 +
 
 +
=== Spatial Join ===

Revision as of 23:34, 11 October 2019

Problem And Motivation

 

Data Preprocessing

 

Interactive Visualisation

 

Interesting Anomalies & Observations

 

References

 

Data Preprocessing with Pandas

Data was provided in the form of 3 csv files:


  • MobileSensorReadings.csv: Consists of measures of radiation values in count per minute. Also contained the longitude and latitude of where the values were collected, and the respective time they were recorded
  • StaticSensorReadings.csv: Consists of measure of radiation values in count per minute. Also contained the respective time at which the values were recorded.
  • StaticSensorLocations.csv: Consists of the longitude and latitude of each static sensor

A polygon Shapefile was also provided, which provides the spatial representation of St. Himark

Before any meaningful observations can be made with Tableau, the csv files have to preprocessed to allow Tableau to carry out a spatial join between point and polygon geometric data. As such, the longitude and latitude values needs to be converted to Point geometry values using GeoPandas. The GeoDataframe is later exported into a Shapefile.

Importing python libraries

Importing relevant libraries.png

I first start off by importing libraries that are necessary in the creation of the new Shapefiles. These libraries include Pandas, Shapely and GeoPandas.

Reading csv files to Pandas Dataframe

Reading StaticSensorReadings.csv and StaticSensorLocations.csv.png

Next, I read the csv files into a Pandas Dataframe.

Merging StaticSensorReadings and StaticSensorLocations

Merging dataframes.png

I proceed by merging the static_reading dataframe with the static_loc dataframe. Here, a full outer join on "Sensor-id" is done to ensure all values are accounted for, even if they are null.

Creating the GeoDataframe

GeoPandas Dataframe.png

Longitude and Latitude values are first converted into geometry point values. Here, I append the geometry point values into a new column called "geometry" and create my new GeoDataframe

Sample of dataframe.png

We can see that I now have a new column called "geometry", which contains geometry point values for each of the rows

Exporting our Dataframe to a Shapefile

Exporting Shapefile.png

Finally, I export my GeoDataframe to a Shapefile. This Shapefile can be later used for spatial join with the provided "StHimark.shp"

Creating Shapefile for MobileSensorReadings

Creating MobileSensorReadings Shapefile.png

Following the steps above, I did the same for "MobileSensorReadings.csv" and generated my mobile sensors Shapefile.


Tableau

With my new Shapefile created, I will now begin to work on visualisations in Tableau.

Spatial Join

Retrieved from ‘https://wiki.smu.edu.sg/1920t1is428g1/index.php?title=IS428_AY2019-20T1_Assign_Sean_Chai_Shong_Hee_Data_Preprosessing&oldid=3012’