ISSS608 2017-18 T1 Assign KYONG HWAN KIM Data Preparation

Vastropolis Contagion Report

Background

Data Preparation

Visualization

Solution

Feedback

Microblog

1. Data cleaning

Description	Illustration
1. Split of Columns Created_at column is splitted to Date and Time columns. Date column is used in other analytics. Also, Location column is splitted to Latitude and Longitude columns. These data is used to plot in Vastropolis map.
2. Outliers There are 21 items with invalid time format. They are removed from analysis. Also, there are 6 items with Longitude outside of given map range. They are removed as well so that all data are within parameters. Total 27 rows are removed and 1,023,050 rows are used for analysis with file name "Microblog_Final.csv".

2. Key Words

Some of key words are given. However, there may be additional key words to enhance accuracy of analysis.

Key word "flu" and "cold" are chosen as they are diagnosis words whereas other words are symptoms. Above graph shows the text distribution by "Date" that contains diagnosis words. Text traffic shoots up from May 18th and remain high until 20th. Word Cloud during 3 days of outbreak using JMP text explorer are shown below.

Therefore, following key words are chosen for analysis.

• Given: flu, fever, chill(s), sweat(s), aches, pain(s), fatigue, cough(ing), breathing difficulty, nausea, vomit(ing), diarrhea, enlarged lymph nodes
• Enhanced: cold, headache, sick, shortness of breath, declining health, hurts to move, aching muscles, sore throat, runny nose, problems breathing, pneumonia

3. Contagion Flag

Text containing above 23 words and phrases are chosen from dataset "Microblog_Final.csv".

Diagnosis Flag is text containing diagnosis words: "flu" and "cold". Symptom Flag is text containing at least 2 of all other key words apart from diagnosis words. Any text containing at least 1 of Diagnosis words or at least 2 of Symptom words are classified as Contagion Flag (20,466 rows) which is used for Visualization analysis. "Contagion_Flag.csv"