Difference between revisions of "IS428 AY2019-20T1 Assign Pham Anh Kim Chau"
Line 128: | Line 128: | ||
[[File:Calendar heatmap by neighborhood.png|500px|centre]] | [[File:Calendar heatmap by neighborhood.png|500px|centre]] | ||
− | For the unidentified neighborhood, when we compare the measurement with the map of St. | + | For the unidentified neighborhood, when we compare the measurement with the map of St. Himark, the "NA" location are at highways/ bridges out of St.Himark. That is why there is information about the neighborhood. On the other hand, when we check the location of Wilson Forest and Scenic Vista, the distance is actually quite close so there might be the chance that the radiation is spread from one area to the other |
[[File:Radiation spot.png|500px|centre]] | [[File:Radiation spot.png|500px|centre]] | ||
Revision as of 15:42, 10 October 2019
Contents
Problem Statement
One of St. Himark’s largest employers is the Always Safe nuclear power plant. The pride of the city, it produces power for St. Himark’s needs and exports the excess to the mainland providing a steady revenue stream. However, the plant was not compliant with international standards when it was constructed and is now aging. As part of its outreach to the broader community, Always Safe agreed to provide funding for a set of carefully calibrated professional radiation monitors at fixed locations throughout the city. Additionally, a group of citizen scientists led by the members of the Himark Science Society started an education initiative to build and deploy lower cost homemade sensors, which people can attach to their cars. The sensors upload data to the web by connecting through the user’s cell phone. The goal of the project was to engage the community and demonstrate that the nuclear plant’s operations were not significantly changing the region’s natural background levels of radiation.
When an earthquake strikes St. Himark, the nuclear power plant suffers damage resulting in a leak of radioactive contamination. Further, a coolant leak sprayed employees’ cars and contaminated them at varying levels. Now, the city’s government and emergency management officials are trying to understand if there is a risk to the public while also responding to other emerging crises related to the earthquake as well as satisfying the public’s concern over radiation
Data Provided
- St. Hillmark shapefile
- StaticSensorReadings.csv
- 4 columns: Timestamp, Sensor-id, Value, Units
- Import to Python as "value" dataframe
- StaticSensorLocations.csv
- 3 columns: Sensor-id, Lat, Long
- Import to Python as "location" dataframe
- MobileSensorReadings.csv
- 7 columns: Timestamp, Sensor-id, Long, Lat, Value, Units, User-id
- Import to Python as "mobile" dataframe
Data Analysis and Preparation
Aggregate the data
- Problem: The 2 datasets is recorded every 5 seconds which lead to the enormous amount of data points. It is impossible to merge the 2 datasets (mobile & static) and store the data back to Excel
- Solution: Aggregate the data from every 5 seconds to every 5 minutes with an underlying assumption - The mobile sensors would not be able to move so far within 5 minutes. The radiation measure would be the average value of all data points within 5 minutes time to prevent the distortion of data (The radiation would be higher if there is no missing data in the period but it would not mean that there is a spike in radiation at the point in time)
- Tool and Packages: Python (Numpy, Pandas, Datetime)
- General
Step | Code |
---|---|
Create a new list of data & time ranges within the start time and end time of the data set |
Real start date & time |
Create a list that contains the IDs of all static sensors |
Create a list that contains all unique values in column "Sensor-id" |
Create an list that contains the IDs of all mobile sensors |
Create a list that contains all unique values in column "Sensor-id" |
- Static Data:
1/ Create a new list called "static_data"
2/ Loop through each index in the date list and each item in the static id list, check the requirements and carry out the steps depend on the scenario
Scenario | Step | Code |
---|---|---|
Scenario 1: When date-time is the start date-time (There will be 1 value for each sensor) | Assign value to the variables | time = str(dd[d]) ss_id = i |
Create a new list called "row" that contains the newly created variables | row = [time,ss_id,num] | |
Append "row" to the "static_data" list | static_data.append(row) | |
Scenario 2: When date-time is the end date-time (There will be multiple values for each sensor) | Assign value to the variables | time = str(d3) ss_id = i |
Create a new list called "row" that contains the newly created variables | row = [time,ss_id,num] | |
Append "row" to the "static_data" list | static_data.append(row) | |
Scenario 3: All the other date-time in between (There will be multiple values for each sensor) | Assign value to the variables | time = str(dd[d]) ss_id = i |
Create a new list called "row" that contains the newly created variables | row = [time,ss_id,num] | |
Append "row" to the "static_data" list | static_data.append(row) |
3/ Create a new dataframe with the data from static_data: static = pd.DataFrame(data=static_data, columns=['Timestamp','Sensor ID','Value'])
4/ Vertically merge the new dataframe with the static sensors' location by matching the Sensor ID: static=pd.merge(static,location,how='left',left_on='Sensor ID',right_on='Sensor-id')
5/ Delete the duplicated "Sensor-id" column: static.drop('Sensor-id',axis =1, inplace=True)
6/ Add in two new columns, User-id and Type of sensor:
static['User-id'] = 'Static'
static['Type of sensor'] = 'Static sensor'
- Mobile Data
Merge mobile data and static data
Add neighborhood information to the data set
Dashboard Design
Interactive Visualization
Color Legends
Observation & Insight
Question 1
Visualize radiation measurements over time from both static and mobile sensors to identify areas where radiation over background is detected. Characterize changes over time.
From the calendar heatmap below, we can clearly see that there are 5 neighborhoods contaminated with radiation at unhealthy level: Old Town, Safe Town, Scenic Vista, Wilson Forest and unidentified neighborhood (NA). Among these neighborhoods, Wilson Forest is the worst of all with extremely high level of radiation by the end of 9th till the beginning of 10th April
For the unidentified neighborhood, when we compare the measurement with the map of St. Himark, the "NA" location are at highways/ bridges out of St.Himark. That is why there is information about the neighborhood. On the other hand, when we check the location of Wilson Forest and Scenic Vista, the distance is actually quite close so there might be the chance that the radiation is spread from one area to the other
Question 2
Use visual analytics to represent and analyze uncertainty in the measurement of radiation across the city
- Compare uncertainty of the static sensors to the mobile sensors. What anomalies can you see? Are there sensors that are too uncertain to trust?
In the picture below, there are the calendar heatmaps based on the values captured by static sensors and mobile sensors separately. For Calendar heatmap for mobile sensors, the filter is set to show only the neighborhoods that have static sensors. Despite displaying the radiation level at the regions at the same time frame with the same color legends setting, there are big differences between these 2 visualizations. The mobile sensors could pick up a few concerning radiation signal while according to static sensors, the radiation level is safe all the time.
Comparing the locations of static sensors against the radiation hotspots detected by mobile sensors, there are 2 points where the static sensors are quite near to the radiation zones but they did not pick up anything the whole time. From this point, we can concluded that one type of sensors might provide incorrect value
- Which regions of the city have greater uncertainty of radiation measurement? Use visual analytics to explain your rationale.
According to this visualization below, Old Town and Safe Town are the 2 neighborhood with the greater uncertainty of radiation measurement
- What effects do you see in the sensor readings after the earthquake and other major events? What effect do these events have on uncertainty?
Question 3
Given the uncertainty you observed in question 2, are the radiation measurements reliable enough to locate areas of concern?
- Highlight potential locations of contamination, including the locations of contaminated cars. Should St. Himark officials be worried about contaminated cars moving around the city?
From the visualization on the left, we can identify that there are
- Estimate how many cars may have been contaminated when coolant leaked from the Always Safe plant. Use visual analysis of radiation measurements to determine if any have left the area.
- Indicated where you would deploy more sensors to improve radiation monitoring in the city. Would you recommend more static sensors or more mobile sensors or both? Use your visualization of radiation measurement uncertainty to justify your recommendation.
Question 4
Summarize the state of radiation measurements at the end of the available period. Use your novel visualizations and analysis approaches to suggest a course of action for the city. Use visual analytics to compare the static sensor network to the mobile sensor network. What are the strengths and weaknesses of each approach? How do they support each other?