|
|
| (13 intermediate revisions by 2 users not shown) |
| Line 5: |
Line 5: |
| | {|style="background-color:#f79b4c; color:#7fc6cb; padding: 10 0 10 0;" width="100%" cellspacing="0" cellpadding="0" valign="top" border="0" | | | {|style="background-color:#f79b4c; color:#7fc6cb; padding: 10 0 10 0;" width="100%" cellspacing="0" cellpadding="0" valign="top" border="0" | |
| | | style="padding:0.3em; font-size:100%; background-color:#7fc6cb; text-align:center; color:#7fc6cb" width="10%" | | | | style="padding:0.3em; font-size:100%; background-color:#7fc6cb; text-align:center; color:#7fc6cb" width="10%" | |
| − | [[IS480 Team wiki: 2018T2 Metanoia | | + | [[IS428_AY2019-20T1_Assign_Tommy Johnson | |
| − | <font color="#000000" size=2><b>HOME</b></font>]] | + | <font color="#000000" size=2><b>Problem & Motivation</b></font>]] |
| | | | |
| | | style="background:none;" width="1%" | | | | style="background:none;" width="1%" | |
| | | style="padding:0.3em; font-size:100%; background-color:#f79b4c; text-align:center; color:#7fc6cb" width="10%" | | | | style="padding:0.3em; font-size:100%; background-color:#f79b4c; text-align:center; color:#7fc6cb" width="10%" | |
| − | [[Image:MetanoiaAboutUs.png|22px | | + | [[IS428_AY2019-20T1_Assign_Tommy Johnson_DataAnalysisTransformation | |
| − | link=IS480 Team wiki: 2018T2 Metanoia About Us]] [[IS480 Team wiki: 2018T2 Metanoia About Us |
| + | <font color="#000000" size=2><b>Data Analysis & Transformation</b></font>]] |
| − | <font color="#000000" size=2><b>ABOUT US</b></font>]] | |
| | | | |
| | | style="background:none;" width="1%" | | | | style="background:none;" width="1%" | |
| − | | style="padding:0.3em; font-size:100%; background-color:#f79b4c; border-bottom:0px solid #3D9DD7; text-align:center; color:#7fc6cb" width="12%" | | + | | style="padding:0.3em; font-size:100%; background-color:#f79b4c; text-align:center; color:#7fc6cb" width="10%" | |
| − | [[Image:MetanoiaProOverview.png|22px | | + | [[IS428_AY2019-20T1_Assign_Tommy Johnson_Visualization | |
| − | link=IS480 Team wiki: 2018T2 Metanoia Project Overview]] [[IS480 Team wiki: 2018T2 Metanoia Project Overview |
| + | <font color="#000000" size=2><b>Interactive Visualization</b></font>]] |
| − | <font color="#000000" size=2><b>PROJECT OVERVIEW</b></font>]] | |
| | | | |
| | | style="background:none;" width="1%" | | | | style="background:none;" width="1%" | |
| − | | style="padding:0.3em; font-size:100%; background-color:#f79b4c; border-bottom:0px solid #3D9DD7; text-align:center; color:#7fc6cb" width="12%" | | + | | style="padding:0.3em; font-size:100%; background-color:#f79b4c; text-align:center; color:#7fc6cb" width="10%" | |
| − | [[Image:MetanoiaPM.png |22px | | + | [[IS428_AY2019-20T1_Assign_Tommy Johnson_Observation | |
| − | link=IS480 Team wiki: 2018T2 Metanoia Project Management]] [[IS480 Team wiki: 2018T2 Metanoia Project Management |
| + | <font color="#000000" size=2><b>Observation & Anomalies</b></font>]] |
| − | <font color="#000000" size=2><b>PROJECT MANAGEMENT</b></font>]] | |
| | | | |
| − | | style="background:none;" width="1%" |
| |
| − | | style="padding:0.3em; font-size:100%; background-color:#f79b4c; border-bottom:0px solid #008e97; text-align:center; color:#7fc6cb" width="10%" |
| |
| − | [[Image: MetanoiaDocumentation.png|22px |
| |
| − | link=IS480 Team wiki: 2018T2 Metanoia Documentation]] [[IS480 Team wiki: 2018T2 Metanoia Documentation|
| |
| − | <font color="#000000" size=2><b>DOCUMENTATION</b></font>]]
| |
| | |} | | |} |
| − | == Problem & Motivation ==
| + | |
| | One of St. Himark’s largest employers is the Always Safe nuclear power plant. The pride of the city, it produces power for St. Himark’s needs and exports the excess to the mainland providing a steady revenue stream. However, the plant was not compliant with international standards when it was constructed and is now aging. As part of its outreach to the broader community, Always Safe agreed to provide funding for a set of carefully calibrated professional radiation monitors at fixed locations throughout the city. Additionally, a group of citizen scientists led by the members of the Himark Science Society started an education initiative to build and deploy lower cost homemade sensors, which people can attach to their cars. The sensors upload data to the web by connecting through the user’s cell phone. The goal of the project was to engage the community and demonstrate that the nuclear plant’s operations were not significantly changing the region’s natural background levels of radiation. | | One of St. Himark’s largest employers is the Always Safe nuclear power plant. The pride of the city, it produces power for St. Himark’s needs and exports the excess to the mainland providing a steady revenue stream. However, the plant was not compliant with international standards when it was constructed and is now aging. As part of its outreach to the broader community, Always Safe agreed to provide funding for a set of carefully calibrated professional radiation monitors at fixed locations throughout the city. Additionally, a group of citizen scientists led by the members of the Himark Science Society started an education initiative to build and deploy lower cost homemade sensors, which people can attach to their cars. The sensors upload data to the web by connecting through the user’s cell phone. The goal of the project was to engage the community and demonstrate that the nuclear plant’s operations were not significantly changing the region’s natural background levels of radiation. |
| | | | |
| Line 41: |
Line 33: |
| | * Identify observations and anomalies to the existing data available | | * Identify observations and anomalies to the existing data available |
| | * Identify contaminated areas and cars so that evacuation can be done efficiently and quickly | | * Identify contaminated areas and cars so that evacuation can be done efficiently and quickly |
| − |
| |
| − | == Dataset Analysis & Transformation Process ==
| |
| − | Before moving on to the analysis, it is essential to clean and transform the raw data so that I can bring value to the analysis. In the zipped file, I am given 3 raw data sets namely:
| |
| − | * MobileSensorReadings.csv - contains the sensor readings of different mobile sensor Ids over a period of time and its locations
| |
| − | * StaticSensorReadings.csv - contains the sensor readings of different static sensor Ids over a period of time
| |
| − | * StaticSensorLocations.csv - contains the different static sensor Ids with its locations
| |
| − |
| |
| − | I will be using Tableau Prep to clean and transform the data. It is a new feature provided by the Tableau. The following section will explain step by step on how I prepare the data sets.
| |
| − |
| |
| − | === 1. Combine the Static sensor readings and locations ===
| |
| − | [[File:JoinStaticsensor.png|500px|centre]]
| |
| − | The first step is to combine the two csv files of static into one. This is to create a tidier data (Tall and skinny structure). I use Join to combine the columns from two different files into one.
| |
| − |
| |
| − | === 2. Create a calculated field for Static and Mobile sensor Ids ===
| |
| − | [[File:CleanSensorId.PNG|500px|centre]]
| |
| − |
| |
| − | The next step is to concatenate "- Static" or "- Mobile" at the back of the sensor Ids with the [Sensor-id] + "- Mobile" or [Sensor-id] + "- Static" calculated fields. This is because I realize that the static sensor Ids have the same Id number although they are referring to different records. Hence, this is done to avoid confusion at the later part. The final output will be as follow.
| |
| − |
| |
| − | Static Sensor
| |
| − | [[File:Staticdata.PNG|centre|500px]]
| |
| − |
| |
| − |
| |
| − | Mobile Sensor
| |
| − | [[File:Mobiledata.PNG|centre|500px]]
| |
| − |
| |
| − | === 3. Combine the static and mobile sensor data into one ===
| |
| − | [[File:Combinedata.PNG|500px|centre]]
| |
| − |
| |
| − | The next step is to combine the static and mobile sensor data into 1 file. This will be the working file that I am going to use in Tableau. I use Union because I am just going to append more rows. After that, I will still need to clean the data to remove any duplicate columns so that all columns are arranged.
| |
| − |
| |
| − | === 4. Aggregated the Timestamp by 1 minute interval ===
| |
| − | Using the following formula, I aggregate the timestamp by 1 minute interval
| |
| − | </br>
| |
| − | <b> DATETIME(INT([Timestamp]) + (INT(FLOAT([Timestamp]) % 1 * (60*24)) / (60*24))) </b> </br>
| |
| − | DATETIME(INT([Timestamp]) : Retain the datetime of the timestamp </br>
| |
| − | (INT(FLOAT([Timestamp]) % 1 * (60*24)) / (60*24))) : A calculation to create a 1-minute interval. ([60 minute] / [desired minute interval] * [24 hour]) </br>
| |
| − | After that, i deleted the original Timestamp from the raw data.
| |
| − | [[File:1-minuteinterval.PNG|250px|centre]]
| |
| − |
| |
| − | === 5. Aggregated the Value based on the Timestamp ===
| |
| − | After aggregated the timestamp, I sum all the measuring readings for that particular sensor ID within that 1 minute interval. For example, 12 counts of readings level for Sensor ID 1 across different seconds. I summed the 12 counts and created 1 row within that minute. This will reduce the number of rows in the data set. Hence, create a faster running time.
| |
| − | [[File:AggregatedValue.PNG|500px|centre]]
| |
| − |
| |
| − | The final Workflow will look like this:
| |
| − |
| |
| − | [[File:Finalworkflow.PNG|500px|centre]]
| |
| − | [[File:Finalworkflow(2).PNG|500px|centre]]
| |
| − |
| |
| − | == Interactive Visualization ==
| |
| − | The interactive visualization can be accessed here:
| |
| − | === Home Page===
| |
| − |
| |
| − | === Radiation Level by Map Chart ===
| |
| − | I am using map chart to visualize the radiation measurements for both static and mobile sensors. With this chart, analysts will be able to see which areas are prone to high or low radiation levels from each sensor types. Adding on, they are able to visualize the track that one mobile sensor took throughout the day and how these affect the readings.
| |
| − | ==== Static sensor radiation level ====
| |
| − | [[File:MapchartRadiationLevel.PNG|500px|centre]]
| |
| − |
| |
| − | ==== Mobile sensor radiation level====
| |
| − | [[File:Mobilemapchartradiationlevel.PNG|500px|centre]]
| |
| − |
| |
| − | To enhance the visualization of the data, implementing interactive elements would help users in analyzing the data intuitively. The following elements are used in this graph.
| |
| − | {| class="wikitable"
| |
| − | |-
| |
| − | ! style="font-weight: bold;background: #7fc6cb;width: 20%;" | Interactive Features
| |
| − | ! style="font-weight: bold;background: #7fc6cb;width: 40%" | Rationale
| |
| − | ! style="font-weight: bold;background: #7fc6cb;" | Brief Implementation Steps
| |
| − | |-
| |
| − | | <center>'''Highlight Mobile sensor ID ''' <br/>[[File:HighlightsensorID.PNG|200px|center]]</center>
| |
| − | || <center>To provide a better insight for the analyst to understand how one particular mobile sensor ID move and how it can affect the readings</center>
| |
| − | || Click on the arrow button on the Sensor ID filter and choose "Show Highlighter
| |
| − |
| |
| − | |-
| |
| − | | <center>'''Filter dates with the use of checkboxes ''' <br>[[File:Datefilter.PNG|200px|center]]</center>
| |
| − | || <center>To provide flexibility for analysts to choose the dates that they are interested to analyse. They can choose only one or multiple dates.</center>
| |
| − | ||
| |
| − | # Drag the Timestamp to the Filter
| |
| − | # Change the format of the timestamp to custom date Month/Date/Year with Date Part option <br/>[[File:CustomDate.PNG|200px|center]]
| |
| − | |-
| |
| − | | <center>'''Animate the radiation level throughout the day''' <br>[[File:TimestampAnimation(1).PNG|200px|center]]</center>
| |
| − | || <center>To allow for greater analysis and aesthetics of the data. Analysts will be able to view the changes of radiation level and identify the highest / lowest level at a particular time clearly.</center>
| |
| − | ||
| |
| − | # Put the Timestamp in Pages section of the Tableau
| |
| − | # Change the format of the timestamp to Hour with a custom option of Date Value
| |
| − | |}
| |
| − |
| |
| − | === Readings Level by sensor type ===
| |
| − | For further breakdown, I visualize how the readings changes over time according to the sensor type.This visualization allows the analyst to look at the pattern of the readings at one glance.
| |
| − | ==== Static reading level ====
| |
| − | [[File:Staticreadings.PNG|500px|centre]]
| |
| − |
| |
| − | ==== Mobile reading level ====
| |
| − | The snapshot below is not limited. Analysts are able to scroll down to view more reading levels from different sensor Id
| |
| − | [[File:Mobilereading.PNG|500px|centre]]
| |
| − | {| class="wikitable"
| |
| − | |-
| |
| − | ! style="font-weight: bold;background: #7fc6cb;width: 20%;" | Major features included
| |
| − | ! style="font-weight: bold;background: #7fc6cb;width: 40%" | Rationale
| |
| − | ! style="font-weight: bold;background: #7fc6cb;" | Brief Implementation Steps
| |
| − | |-
| |
| − | | <center>'''Differentiate sensor ID by the colour''' <br/>[[File:SensorIDColour.PNG|200px|center]]</center>
| |
| − | || <center>To provide easy readability and improve on aesthetics.</center>
| |
| − | || Drag the sensor ID to the color Marks
| |
| − | |-
| |
| − | | <center>'''Differentiate the timestamp by days''' <br>[[File:Timestampdifferentiate.PNG|200px|center]]</center>
| |
| − | || <center>To provide easy readability and improve on aesthetics</center>
| |
| − | || Drag the Timestamp in the Columns and custom the date to Month/Date/Year with Date Part option <br/>[[File:CustomDate.PNG|200px|center]]
| |
| − | |-
| |
| − | | <center>'''Log Transformation of the value''' <br>[[File:LogValue.PNG|200px|center]]</center>
| |
| − | || <center>Log transformation will make the distribution to be less skewed. This will make the pattern to be more interpretable and inferential statistics are met </center>
| |
| − | || Create a calculated field of log value. Then, put it at the Rows
| |
| − | |-
| |
| − | |}
| |
| − |
| |
| − | == Interesting Observation and Anomalies==
| |
| − | This section aims to answer the questions posed at the mini challenges
| |
| − | === Q1. Visualize radiation measurements over time from both static and mobile sensors===
| |
| − | To answer this question, you may look at the <b> Overall Radiation Level </b> and <b> Readings Level by sensor type </b>
| |
| − | * I found out that the highest readings level for static sensor comes from <b> sensor 15-Static at 1,077 cpm on 8 April 2020 at 10:20:10 AM </b>
| |
| − | [[File:Higheststatic.PNG|500px|centre]]
| |
| − | <br/>
| |
| − | * The lowest reading level for static sensor comes from <b> sensor 14 - Static at 0 cpm on 10 April 2020 at 6:06:25 AM </b>
| |
| − | [[File:Loweststatic.PNG|500px|centre]]
| |
| − | <br/>
| |
| − | * The highest readings level for mobile sensor comes from sensor <b> 2-Mobile at 1,673 cpm on 6 April 2020 at 9:17:50 PM </b>
| |
| − | [[File:Highestmobile.PNG|500px|centre]]
| |
| − | <br/>
| |
| − | * The lowest reading level for mobile sensor comes from sensor <b> 19-Mobile at 0 cpm across both 6 April 2020 to 7 April 2020 </b>
| |
| − | [[File:Lowestmobile.PNG|500px|centre]]
| |
Visualization Analysis on Always Safe nuclear power plant
