View source for ISSS608 2018-19 T1 Assign Kateryna Mazurenko Methodology

<div style=background:#dadaeb border: #000080>
[[File:Photo_verybig_155447.jpg|300px]]
<b><font size = 5; color="#2B3856"> <big>'''AIR QUALITY IN BULGARIA'''</big> </font></b>
</div>

<!--MAIN HEADER -->
{|style="background-color:#dadaeb;" width="100%" cellspacing="0" cellpadding="0" valign="top" border="0"  |
| style="font-size:100%; solid #FFFFFF; background:#2B3856; text-align:center;" width="20%" | 
;
[[ISSS608_2018-19_T1_Assign_Kateryna_Mazurenko| <font color="#C0C0C0">Introduction</font>]]

| style="font-size:100%; solid #C0C0C0; background:#2B3856; text-align:center;" width="20%" | 
;
[[ISSS608_2018-19_T1_Assign_Kateryna_Mazurenko_Methodology| <font color="#FFFFFF">Methodology</font>]]

| style="font-size:100%; solid #C0C0C0; background:#2B3856; text-align:center;" width="20%" | 
;
[[ISSS608_2018-19_T1_Assign_Kateryna_Mazurenko_Insights| <font color="#C0C0C0">Insights</font>]]

| style="font-size:100%; solid #1B338F; background:#2B3856; text-align:center;" width="20%" | 
;
[[ISSS608_2018-19_T1_Assign_Kateryna_Mazurenko_Conclusion| <font color="#C0C0C0">Conclusion</font>]]

| style="font-size:100%; solid #1B338F; background:#2B3856; text-align:center;" width="20%" | 
;
[[ISSS608_2018-19_T1_Assign_Kateryna_Mazurenko_Dashboard| <font color="#C0C0C0">Dashboard</font>]]
|  &nbsp;
|}
<br/>

=== Data Preparation ===
'''Air Tube (air quality and meteo data measured by citizen)'''


1. Reverse geocode using geohash package in R - for both files (2017 and 2018 data) and saving it as csv back.

[[File:CitizenPrep00.png|frameless|500px|Citizen data - geohash recode]]

2. Data comes as two files for 2017 and 2018 year, so at first, is was concatenated using JMP software (applying function directly as data has the same structure).

3. Concatenated file: missing pattern run to check for empty rows, 4 rows were excluded and data with no missing fields was saved.

4. Run distribution to check for trustworthy values.<br>

[[File:CitizenPrep02.png|frameless|300px|Missing pattern run]] <br>
[[File:CitizenQty03.png|frameless|300px|Citizen measurements distribution]]
[[File:CitizenQty04.png|frameless|300px|Citizen measurements distribution meteo]]
<br>


5. Distribution shows a lot of negative values for temperature, pressure and humidity as well as unreasonable(impossible) maximal values. I checked for possible weather conditions in Bulgaria and applied formulas to replace these values with empty space.

[[File:CitizenQty05.png|frameless|300px|P1 values]]
[[File:CitizenQty06.png|frameless|150px|Citizen data preparation formula]]
[[File:CitizenQty07.png|frameless|150px|Citizen Data preparation formula]]

I didn’t recode big values in P1/P2 measures as there is no confidence if it’s valid or not. I also regard there two measures separately in future analysis.

6. Format date and time using lubridate in R

[[File:CitizenQty08.png|500px|frameless|Citizen data time format]]



'''EEA Data (official air quality measures)'''

1. Perform concatenation using JMP - combining all data into one file

2. Check missing pattern, check distribution how many data comes as measured hourly/daily/various

3. As it was not in uniform format, I grouped it as daily average in R. I will use average daily concentration in future analysis.

[[File:EEADataCleaning01.png|frameless|500px|EEA time masure distribution]]
[[File:EEADataCleaning02.png|frameless|500px|EEA date format in R]]



'''Meteo-data''' 

1. Run missing patterns and run distributions in JMP - all the data looks trustworthy except for missing values coded as -9999 (as was said in metadata file)

2. Used formulas in JMP to replace unreasonable values with empty space - example shown.

3. Change date format using lubridate in R so it will be possible to join with other files

[[File:MeteoQty01.png|frameless|300px|Meteo data distributions]]
[[File:MeteoQty02.png|frameless|150px|Meteo data formula example]]
[[File:MeteoQty03.png|frameless|500px|Meteo data date format in R]]


'''Topo data''' was taken as is



=== Dashboard Design ===



1. Connect to data source and join all the prepared tables. I used outer join for all the cases to keep all the data in place, official stations and meteodata was joined to citizen file using date as unique id, topology was joined using latitude and longitude fields.

[[File:Screen Shot 2018-11-18 at 2.10.21 PM.png|300px|frameless|Data join in Tableau]]


2. Check topology data to see if there is any use of elevation data to further analysis. 
To do that: 
* Bring lat and long fields to columns and rows (map will be shown)
* Uncheck aggregate values to see all the points on map
* Drag elevation to details 
* All the point are on the same distance, so select shape-square in marks to cover the area at most, adjust size
* Drag elevation to color and adjust opacity

[[File:Screen Shot 2018-11-18 at 2.12.05 PM.png|200px|frameless|Topology creation]]
[[File:Screen Shot 2018-11-18 at 2.12.25 PM.png|200px|frameless|Topology creation - 2nd]]
[[File:Screen Shot 2018-11-18 at 2.12.33 PM.png|200px|frameless|Topology creation - 3rd]]

As we can see, topology data only shows that there are mountains at the south west of the city - the same information is provided by the background map itself, so I will not use topology layer further.

[[File:Screen Shot 2018-11-18 at 2.12.41 PM.png|500px|frameless|Topology map created]]


3. Overall pattern of official stations measurements
I want to see high-level data provided by all stations to compare their measurements as well as see the pattern through all the years. 
To do that:
* Bring date: year and month into columns
* Check the format to be generic year/month
* Drag PM10 concentration into rows, check it as average
* Bring stations into filters and to colour 
* As we know from EEA standards, harmful level of PM10 in the air is 50mg/m3, so add this as reference (analytics - constant line)

[[File:Screen Shot 2018-11-18 at 2.26.11 PM.png|200px|frameless|Official stations building 01]]
[[File:Screen Shot 2018-11-18 at 2.31.12 PM.png|300px|frameless|Official station building 02]]
[[File:Screen Shot 2018-11-18 at 2.32.41 PM.png|300px|frameless]]
[[File:Screen Shot 2018-11-18 at 2.34.09 PM.png|300px|frameless|Official station building 04]]


4. Build daily/calendar view for a station in selected year
I want to see how the concentration of PM10 changes day to day within a year period so understand patterns in details. 
To do that:
* Date field (year and month) bring to rows, checking as generic format
* Day - to columns
* Station to columns
* PM10 concentration (as average) - to rows, selecting bars for a view
* Station and Year - include to filters to select one year only and one station only 
* Format tooltip to make it interactive
* Format title to make it interactive 

[[File:Screen Shot 2018-11-18 at 2.39.49 PM.png|300px|frameless]]
[[File:Screen Shot 2018-11-18 at 2.43.23 PM.png|150px|frameless|Build calendar view 02]]
[[File:Screen Shot 2018-11-18 at 2.43.49 PM.png|200px|frameless]]
[[File:Screen Shot 2018-11-18 at 2.44.25 PM.png|200px|frameless|Build calendar view 04]]

Overall view gives us possibility to check concentration for selected date (year - in filter, and day - on hover as tooltip), it also gives an understanding of trend.

[[File:Screen Shot 2018-11-18 at 2.44.36 PM.png|500px|frameless|Calendar view]]


5. Combine overall pattern of official stations measurements and daily/calendar view for a station into one dashboard. Add dashboard action to highlight selected date - so that I can see the exact data for selected date and how it compares to stations pattern and yearly pattern.
Add text to illustrate where the reference is taken from.

[[File:Screen Shot 2018-11-18 at 2.57.44 PM.png|500px|frameless|Dashboard stations]]
[[File:Screen Shot 2018-11-18 at 2.57.57 PM.png|500px|frameless|actions]]


6. Meteo data in comparison to air pollution level
I want to see is there any relation between change in weather conditions and PM10 particles concentration.
To do that:
* Use exact date at columns 
* Drag average concentration and temperature to rows and make them dual-axis  
* Add Station selection to filters, add date range to filters  
* Format axis to make user-friendly name, add units of measure 
* As reference add 50mg/m3 level to show harmful level of concentration 
* As reference to temperature add 12 degrees as the temperature when heating season starts in Bulgaria  
* Annotate marks 
* Change colours so that temperature will be red, as well as assigning the colours for the stations so they will be the same in all meteo-measurements graphs 
* Format title and tooltips to make it interactive 

[[File:Screen Shot 2018-11-18 at 2.48.43 PM.png|200px|frameless|temp 01]]
[[File:Screen Shot 2018-11-18 at 2.49.15 PM.png|200px|frameless|temp02]]<br>

[[File:Screen Shot 2018-11-18 at 2.54.28 PM.png|500px|frameless|temperature trend]]

All the steps: repeat for humidity, wind speed and precipitation. Ensure that colours for the stations are the same. 


7. I will put temperature graph into separate dashboard as it shows visible trend: under lower temperatures concentration of PM10 is higher. 

[[File:Screen Shot 2018-11-18 at 3.01.40 PM.png|500px|frameless|temp dashboard]]


8. All other meteo-data will be combined into one dashboard. Add action to see the point for selected date through all the measures taken. Ensure that filters apply to all the sheets on dashboard. 

[[File:Screen Shot 2018-11-18 at 3.03.02 PM.png|500px|frameless|temp dashboard]]
[[File:Screen Shot 2018-11-18 at 3.03.22 PM.png|500px|frameless|meteo dashboard action]]


9. Comparison of official data and citizen measurement. 
I want to see how they relate one to another. I also want to see the difference between P1, P2 and combined data. 
To do that:
* Create calculated field for P1+P2 concentration
* Add date to columns, concentrations - to rows
* Make them dual-axis, sync axis to have the same scale, edit axis to make it user-friendly, add units of measure
* Add time range to filters
* All the meteo-data available: show in tooltip 
* Add average showing value as reference line for both measurement, add median to have spread understanding
* Annotate highest point to put attention 
* The same steps repeat for P1 and P2 separately as we can not conclude which of these field is the right comparison to PM10.

[[File:Screen Shot 2018-11-18 at 3.06.27 PM.png|150px|frameless|comparison 01]]
[[File:Screen Shot 2018-11-18 at 3.07.13 PM.png|300px|frameless|comparison 02]]
[[File:Screen Shot 2018-11-18 at 3.09.24 PM.png|300px|frameless|comparison 03]] <br>
[[File:Screen Shot 2018-11-18 at 3.11.12 PM.png|500px|frameless]]


10. Add the sheet showing P1, P2 and combined data into the dashboard, ensure filtration applies to all the sheets there. Add action to highlight the same date selected for all options.

[[File:Screen Shot 2018-11-18 at 3.13.40 PM.png|500px|frameless]]


11. Map of citizen measurements.
I want to see where on map the citizen measurements were taken place, as well as what was the concentrations and how it changed through the time.
To do that:
* Bring longitude to columns and latitude to rows, add geohash to details to see all the points on map 
* Drag second longitude to columns to have to maps  for comparison 
* Bring date to pages to play video about situation changing in time 
* Bring date also to filter to select specific date snapshot 
* Drag P1 to first longitude tab, put P1 concentration both to colour and size 
* For the colour: select diverging palette with reversed blue to red and center = 50 (harmful level concentration)  
* Repeat the same for P2 on second map  
* Edit tooltips and title 
* Duplicate sheet, removing all maps details but adding data about weather, to show it as separate panel, not in tooltip

[[File:Screen Shot 2018-11-18 at 3.18.04 PM.png|300px|frameless|citizenmap building 01]]
[[File:Screen Shot 2018-11-18 at 3.18.45 PM.png|150px|frameless|citizenmap building 02]]
[[File:Screen Shot 2018-11-18 at 3.20.48 PM.png|300px|frameless|citizenmap building 03]]
[[File:Screen Shot 2018-11-18 at 3.23.26 PM.png|300px|frameless|citizenmap building 04]]
 
Add citizen data map and weather info-panel to dashboard, add highlight action. As a result, by click to point on map I can see the same point on second map, I also see the weather conditions. 

[[File:Screen Shot 2018-11-18 at 3.22.12 PM.png|500px|frameless]]<br>
[[File:Screen Shot 2018-11-18 at 3.24.46 PM.png|500px|frameless]]

12. For all the dashboards created: add buttons with tooltips to navigate through the infographics. Each repeated button correspond to the same dashboard - for example “meteo” always will bring to meteo stations dashboard, similar to webpage scenario.

[[File:Screen Shot 2018-11-18 at 3.27.43 PM.png|150px|frameless]]