Difference between revisions of "ISSS608 2017-18 T3 Assignment SHI CHEN Data Exploratory"
(5 intermediate revisions by the same user not shown) | |||
Line 7: | Line 7: | ||
| style="font-family:Century Gothic; font-size:100%; solid #000000; background:#ACDDE5; text-align:center;" width="25%" | | | style="font-family:Century Gothic; font-size:100%; solid #000000; background:#ACDDE5; text-align:center;" width="25%" | | ||
; | ; | ||
− | [[ISSS608_2017-18_T3_Assignment_SHI CHEN|<b><font size="4"><font color="# | + | [[ISSS608_2017-18_T3_Assignment_SHI CHEN|<b><font size="4"><font color="#42485D">Overview</font></font></b>]] |
| style="font-family:Century Gothic; font-size:100%; solid #1B338F; background:#ACDDE5; text-align:center;" width="25%" | | | style="font-family:Century Gothic; font-size:100%; solid #1B338F; background:#ACDDE5; text-align:center;" width="25%" | | ||
; | ; | ||
− | [[ISSS608_2017-18_T3_Assignment_SHI CHEN_Data_Exploratory|<b><font size="4"><font color="# | + | [[ISSS608_2017-18_T3_Assignment_SHI CHEN_Data_Exploratory|<b><font size="4"><font color="#FFFFFF">Methodology & Data Exploratory</font></font></b>]] |
| style="font-family:Century Gothic; font-size:100%; solid #1B338F; background:#ACDDE5; text-align:center;" width="25%" | | | style="font-family:Century Gothic; font-size:100%; solid #1B338F; background:#ACDDE5; text-align:center;" width="25%" | | ||
Line 19: | Line 19: | ||
| style="font-family:Century Gothic; font-size:100%; solid #1B338F; background:#ACDDE5; text-align:center;" width="25%" | | | style="font-family:Century Gothic; font-size:100%; solid #1B338F; background:#ACDDE5; text-align:center;" width="25%" | | ||
; | ; | ||
− | [[ | + | [[Assignments|<b><font size="4"><font color="#42485D">Back to main</font></font></b>]] |
− | + | | style="font-family:Century Gothic; font-size:100%; solid #1B338F; background:#ACDDE5; text-align:center;" width="25%" | | |
+ | ; | ||
| | | | ||
|} | |} | ||
<br/> | <br/> | ||
+ | |||
+ | If you want to take a look at my work on Tableau when you go through this page. | ||
+ | [https://public.tableau.com/views/EnvironmentalAnalysis/EnvironmentalAnalysis?:embed=y&:display_count=yes&publish=yes Click here] | ||
+ | |||
==<font color = "#326569">Dataset Description</font>== | ==<font color = "#326569">Dataset Description</font>== | ||
Line 50: | Line 55: | ||
|- | |- | ||
| Measure | | Measure | ||
− | || Measurement indicators, including | + | || Measurement indicators, including 104 chemicals and water temperature monitoring as well as monitoring of a biological quantity |
|- | |- | ||
Line 97: | Line 102: | ||
"> | "> | ||
=== Hydrological overview === | === Hydrological overview === | ||
− | ==== <font color ="# | + | ==== <font color ="#1B338F>*'''How many observation location and streams are there in this area?''' </font> ==== |
In total, Hydrology department has gradually set 10 stations from 1998-2016. More specifically, Achara, Decha, Tansanee were added as observation station since 2009. | In total, Hydrology department has gradually set 10 stations from 1998-2016. More specifically, Achara, Decha, Tansanee were added as observation station since 2009. | ||
Line 105: | Line 110: | ||
In order to facilitate the analysis of the correlation of different points in a river, these observation points were grouped. From right to left are path 1, 2, 3 and 4. In the below graph, the locations with same color are in the same stream. | In order to facilitate the analysis of the correlation of different points in a river, these observation points were grouped. From right to left are path 1, 2, 3 and 4. In the below graph, the locations with same color are in the same stream. | ||
− | [[File:2-5.jpg| | + | [[File:2-5.jpg|350px]] |
− | ==== <font color ="# | + | ==== <font color ="#1B338F>* '''What is the flow of the river?''' </font> ==== |
As all the streams are originated from the north part, it can be speculated that the terrain in the north is higher than other area. The river flows southward as a whole. | As all the streams are originated from the north part, it can be speculated that the terrain in the north is higher than other area. The river flows southward as a whole. | ||
− | ==== <font color ="# | + | ==== <font color ="#1B338F>* '''What is the monitoring time and frequency of the Hydrological Department?''' </font> ==== |
'''Monitoring Time:''' | '''Monitoring Time:''' | ||
Line 119: | Line 124: | ||
<font color ="#0F334D> Conjecture1: </font> This may indicate that at earlier years, these three years appeared environment problems. | <font color ="#0F334D> Conjecture1: </font> This may indicate that at earlier years, these three years appeared environment problems. | ||
+ | '''Frequency of Detection:''' | ||
+ | |||
+ | [[File:2-6.jpg|400px]] | ||
+ | |||
+ | {| class="wikitable" style="background-color:#FFFFFF;" width="100%" | ||
+ | |- | ||
+ | ! Group !! Findings | ||
+ | |- | ||
+ | | Boonsri and Chai | ||
+ | || Belonging to upstream and middle stream of path1. | ||
+ | |||
+ | |- | ||
+ | | Kannika and Sakda | ||
+ | || Belonging to downstream of river and had closer distance. | ||
+ | |||
+ | |- | ||
+ | | Kohsoom and Busarakhan and Somchair | ||
+ | || Belonging to upper area of the map, which are also originated from same mountain. | ||
+ | |||
+ | |- | ||
+ | | Achara, Decha, Tansanee | ||
+ | || They are all added from 2009.) | ||
+ | |} | ||
+ | |||
+ | From the time point of view, the department has increased the measurement frequency from 2004 to 2007 at almost all observation locations, and location Boonsri and Chai are particularly prominent. | ||
+ | |||
+ | <font color ="#0F334D> Conjecture2: </font> There might be a serious environmental problem happened before 2004, especially in Boonsri and Chai. So the hydrological department has already estimated that environmental problems had been discovered in these two locations. | ||
+ | |||
+ | === Observations overview === | ||
+ | |||
+ | ==== <font color ="#1B338F>*'''How many kinds of chemicals are there?''' </font> ==== | ||
+ | |||
+ | Based on the count of distinct kinds of chemicals, it is observed that there are 104 kinds of chemicals, as well as two other indicators, water temperature and Macrozoobenthos (creature). | ||
+ | |||
+ | ==== <font color ="#1B338F>*'''How the measured substance type changes over time? 2003 might be an important year to be noticed.''' </font> ==== | ||
+ | |||
+ | Area chart has been used to detect average chemicals value changes over time. | ||
+ | |||
+ | [[File:2-7.jpg|400px]] | ||
+ | |||
+ | It can be found that in 2003 there were explosive growth of certain chemicals. Later on in anomaly analysis part, this would be explained more. | ||
+ | This phenomenon can also verify that the previous speculation is basically reasonable. Due to the large-scale growth of certain substances, environmental problems have gradually become prominent. | ||
+ | |||
+ | ==== <font color ="#1B338F>*'''What is the difference in the type of measurement at each location?''' </font> ==== | ||
+ | |||
+ | This can be interactively explored through dashboard. | ||
+ | |||
+ | ==== <font color ="#1B338F>*'''Other observations''' </font> ==== | ||
+ | |||
+ | Based on unit of measures, it can be noticed that the Hydrological Department | ||
+ | |||
+ | === Anomaly Analysis:=== | ||
+ | |||
+ | ==== <font color ="#1B338F>*'''Were there any anomaly chemicals throughout different locations?''' </font> ==== | ||
+ | |||
+ | [[File:2-8.jpg|380px]] | ||
+ | |||
+ | In the above graph, each dot represents every chemical. Green to Red are used to show the value of z-score. The deeper the green, the more stable it is. The deeper the red, the greater the fluctuation. You can not only see how many chemicals that readings detected in that particular year, but also faster find out the volatile chemicals in that period of time. | ||
+ | |||
+ | ==== <font color ="#1B338F>*'''For deeper mining, how did these chemicals fluctuate in the past years?''' </font> ==== | ||
+ | |||
+ | When clicking the dot that you want to explore, the below graph would be shown the changing pattern of this chemicals. | ||
+ | |||
+ | [[File:2-9.jpg|380px]] | ||
+ | [[File:2-10.jpg |380px]] | ||
+ | ==== <font color ="#1B338F>*'''How did it distributed in different locations?''' </font> ==== | ||
+ | |||
+ | [[File:2-11.jpg|380px]] | ||
+ | |||
+ | Right hand side map would allow you to see when and where it distributed in past years. | ||
==<font color = "#326569">Reference</font>== | ==<font color = "#326569">Reference</font>== | ||
Line 127: | Line 202: | ||
| Hydrological measurements - ''published by Important India'' || https://ocw.tudelft.nl/courses/hydrological-measurements/ | | Hydrological measurements - ''published by Important India'' || https://ocw.tudelft.nl/courses/hydrological-measurements/ | ||
|- | |- | ||
− | + | ||
− | |||
|} | |} | ||
</div> | </div> |
Latest revision as of 18:14, 8 July 2018
|
|
|
|
|
If you want to take a look at my work on Tableau when you go through this page. Click here
Contents
Dataset Description
Here is the brief description of data given by Vast Challenge. In total, there are two files, one is "Boonsong Lekagul waterways readings", which is showing the record of each chemicals throughout the timeline and observation location.
Variable | Description |
---|---|
Id | ID of each record |
Value | Measured value for the chemical or property in this record |
Location | Name of the location sample was taken from. They are Achara, Boonsri, Busarakhan, Chai, Decha, Kannika, Kohsoom, Sakda, Somchair, Tansanee. |
Sample Date | Sampling time (The record shows it was from Jan 1998 to Dec 2016) |
Measure | Measurement indicators, including 104 chemicals and water temperature monitoring as well as monitoring of a biological quantity |
Unit | Unit of measured chemicals. There are mg/l, µg/l and C. (Macrozoobenthos is not listed with any type of units. ) |
Methodology
Workflow
I. Data preparation:
Combine three tables together to get full picture of changes in all observation years, observation location and chemicals.
II. Geographical preparation:
There is a topographic map in this area. But there’s no other information like latitude and longitude. It should only be used to generate relative coordinates for each observation location.
III. Data exploration:
The geography, the number of species, the number and type of measurement, the number of detections, and the timeline are combined together to explore, to discover their associations and identify the potential factors affecting birds.
IV. Statistical application:
To find out the abnormal chemical elements faster, in addition to looking at the change values of each substance in turn, I also introduced the concept of Z-score to assist the judgment.
The larger the value of z score, the greater the volatility that represents it.
V. Final deliverable: This challenge analysis was done in tableau. Link:
Tool
- Tableau : Used to do EDA and build dashboard
- JMP : To look through distribution of data
Data exploratory
Hydrological overview
*How many observation location and streams are there in this area?
In total, Hydrology department has gradually set 10 stations from 1998-2016. More specifically, Achara, Decha, Tansanee were added as observation station since 2009.
According to the map river distribution, the 10 stations are distributed in the upper, middle and downstream of 4 different streams. In order to facilitate the analysis of the correlation of different points in a river, these observation points were grouped. From right to left are path 1, 2, 3 and 4. In the below graph, the locations with same color are in the same stream.
* What is the flow of the river?
As all the streams are originated from the north part, it can be speculated that the terrain in the north is higher than other area. The river flows southward as a whole.
* What is the monitoring time and frequency of the Hydrological Department?
Monitoring Time:
Current data indicates that the monitoring period for the hydrological sector began in 1998 and three new locations were added in 2008.
Conjecture1: This may indicate that at earlier years, these three years appeared environment problems.
Frequency of Detection:
Group | Findings |
---|---|
Boonsri and Chai | Belonging to upstream and middle stream of path1. |
Kannika and Sakda | Belonging to downstream of river and had closer distance. |
Kohsoom and Busarakhan and Somchair | Belonging to upper area of the map, which are also originated from same mountain. |
Achara, Decha, Tansanee | They are all added from 2009.) |
From the time point of view, the department has increased the measurement frequency from 2004 to 2007 at almost all observation locations, and location Boonsri and Chai are particularly prominent.
Conjecture2: There might be a serious environmental problem happened before 2004, especially in Boonsri and Chai. So the hydrological department has already estimated that environmental problems had been discovered in these two locations.
Observations overview
*How many kinds of chemicals are there?
Based on the count of distinct kinds of chemicals, it is observed that there are 104 kinds of chemicals, as well as two other indicators, water temperature and Macrozoobenthos (creature).
*How the measured substance type changes over time? 2003 might be an important year to be noticed.
Area chart has been used to detect average chemicals value changes over time.
It can be found that in 2003 there were explosive growth of certain chemicals. Later on in anomaly analysis part, this would be explained more. This phenomenon can also verify that the previous speculation is basically reasonable. Due to the large-scale growth of certain substances, environmental problems have gradually become prominent.
*What is the difference in the type of measurement at each location?
This can be interactively explored through dashboard.
*Other observations
Based on unit of measures, it can be noticed that the Hydrological Department
Anomaly Analysis:
*Were there any anomaly chemicals throughout different locations?
In the above graph, each dot represents every chemical. Green to Red are used to show the value of z-score. The deeper the green, the more stable it is. The deeper the red, the greater the fluctuation. You can not only see how many chemicals that readings detected in that particular year, but also faster find out the volatile chemicals in that period of time.
*For deeper mining, how did these chemicals fluctuate in the past years?
When clicking the dot that you want to explore, the below graph would be shown the changing pattern of this chemicals.
*How did it distributed in different locations?
Right hand side map would allow you to see when and where it distributed in past years.
Reference
Hydrological measurements - published by Important India | https://ocw.tudelft.nl/courses/hydrological-measurements/ |