Assign NGO SIEW HUI Q1

ISSS608 Visual Analytics and Applications - Individual Assignment Report

Background

Data Preparation

Question 1

Question 2

Question 3

Conclusion

Feedback

Question 1

Characterize the sensors' performance and operation. Are they all working properly at all times? Can you detect any unexpected behaviors of the sensors through analyzing the readings they capture? Limit your response to no more than 9 images and 1000 words.

Response

As the sensors were supposed to be taking readings at every hour of the day, it would be interesting to find out if there were any missing records. Hence, a dashboard was developed with interactive filters (i.e. select sensor, month and chemical) to enable the viewing of captured readings on an hourly basis. The visualisation was in the form of a trellis chart to facilitate a single view of the selected month, with each day of the month represented by each panel.

Note that this chart was designed to provide a quick overview of the data gaps (if any), and it was not meant for analysing the sensor readings (refer to Question 2 for analysis of the readings). Hence, the scale of the y-axis for the sensor readings had been adjusted accordingly, i.e. the plotted lines were 'flatten' in order to spot the missing records easily.

Sample View of Dashboard: Hourly Time Series of Sensor Readings

Observation 1: There were indeed many missing records spread across all 3 months of data (April, August and December), and also across all 9 sensors and 4 chemicals. For example, from the above image, it was clear that quite a few readings were missing from Sensor 9 for chemical Methylosmolene on 11 December 2016.

After establishing that there were missing records from the Sensor Data, the next step would be to analyse the number of records captured per day (break-down by sensors and chemicals). Hence, a dashboard was developed to view the deviations from the expected number of records, on the basis that there should be 24 readings expected per day for the selected chemical and sensor. To obtain the deviation from the expected number of records, a new calculated field was created (i.e. 'count of records' minus 24). Note that interactive filters were added to the dashboard so as to enable the selection of a single / multiple chemical(s).

Note: If the number of records was as expected (i.e. 24), the value of deviation would be zero, and hence it would not show up on the chart (i.e. ideal scenario). The zero-axis was used as a reference line to view the negative deviations (i.e. missing records) and the positive deviations (i.e. extra records).

Sample View of Dashboard: Extra / Missing Sensor Readings by Chemical (Daily View)

Observation 2: Aside from the missing records, it became clear from the dashboard that there were extra records present in the data for chemical AGOC-3A, as evident by the number of records captured per day being greater than 24 for each sensor (i.e. positive deviations, above the zero-axis line). From the above image, it was noted that the extra records were spread across the sensors and days, but there were 2 days (i.e. 13 August and 11 December) with particularly high number of extra records (i.e. 10 and 9 extra records respectively).

Note: This also highlighted the undesirable situation that there were more than one single record for a particular sensor, chemical and hour of the day. Hence, considerations should be made during downstream analysis (e.g. avoid double-counting of sensor readings which would skew the analysis).

Sample View of Dashboard: Extra / Missing Sensor Readings by Chemical (Daily View)

Observation 3: When both chemicals, AGOC-3A and Methylosmolene, were selected on the dashboard, it was observed that the number of missing records for Methylosmolene matched the number of extra records for AGOC-3A, and hence a mirror image was observed in the chart. This might be due to a system bug which had wrongly recorded the readings for Methylosmolene against AGOC-3A instead.

Note: This would impact the downstream analysis of sensor readings for both chemicals, AGOC-3A and Methylosmolene. Hence, it is recommended that further checks should be conducted by the operations team supporting sensors' operations to investigate if this discrepancy in sensor readings is due to any system bug or malfunction of sensor.

Sample View of Dashboard: Extra / Missing Sensor Readings by Chemical (Daily View)

Observation 4: When both chemicals, Appluimonia and Chlorodinine, were selected on the dashboard, it was observed that the missing records for Appluimonia and Chlorodinine had mostly occurred on the same days. To elaborate further, there were missing records for both chemicals across all 9 sensors on the following 7 days (i.e. across the 3 months' of data provided): 02 April, 06 April, 02 August, 04 August, 07 August, 02 December and 07 December.

Further analysis on the dashboard (i.e. selecting all 4 chemicals) revealed that on these 7 days, all 4 chemicals had missing records consistently. This might be due to a system bug or it could be some system maintenance process (e.g. rebooting of the sensors' application) which led to all 9 sensors having missing records for all 4 chemicals during those days.

Based on the above observation, it would be interesting to find out if the missing records had occurred during the same timing. Hence, a different dashboard with a similar view as above was developed based on the number of records at the hourly basis. In this case, however, there should be 30 or 31 readings expected per month for the selected chemical and sensor at the hourly basis (e.g. there should be 30 records at 00:00hr and every subsequent hour for each chemical during the month of April). To obtain the deviation from the expected number of records, a new calculated field was created (i.e. 'count of records' minus 30/31 depending on which month). Similarly, interactive filters were added to the dashboard so as to enable the selection of a single / multiple chemical(s).

Sample View of Dashboard: Extra / Missing Sensor Readings by Chemical (Hourly View)

Observation 5: From the dashboard, it was clear that there were many missing records which had occurred during midnight (00:00hr) consistently across all sensors (for all 4 chemicals). This would be in line with the above hypothesis, i.e. the missing records could be likely due to some system maintenance process (e.g. rebooting of the sensors' application) which had led to the sensors not being able to capture any readings during the specific hour designated for system shut-down / reboot.

---

To access the interactive version of the above dashboards, please go to the following URL on Tableau Public:

[Placeholder]