ISSS608 2017-18 T1 Assign FOO CELONG RAYMOND/MakingSenseOfTheChatter

< Background

Home

Preliminary >

Exploring and Organising the Data

---

First thing first, I checked if there is any dirty data. True enough, there are 21 microblogs with problems with the time values in the date. I cleaned up the time and set them to midnight (00:00).

The number of microblog is massive. I looked through to see the distribution of posts over time.

The data will need to be organised in some manner so that they can be easily analysed later. The obvious choice is by city zones. I carefully started to group the microblog by the zones by creating a column to store the zone in which the microblog was transmitted.

Next, I will also create an indicator column for microblogs that were transmitted area near the various points of interest.

That should organised the geolocation information. Now that that is done, the next thing is to organise the texts by their content. I classify the microblogs by their similarities to each other, using the latent class analysis that is parameterised to generate 10 clusters.

•  Clusters 1 to 5 are common topics.
•  Class 6 & 7 contains messages about people suffering from the illness but there are many post where it is from a third person point of view, or it is not clear if the author is talking about himself. They might be useful to track the spread but less so to trace the source by tracking the whereabout of the author.
•  Class 8 are microblogs in one or more foreign languages.
•  Class 9 are messages about conventions and election debates. These gatherings may be hotspots for the transmission of the disease.
•  Class 10 is interesting. It contains conversations that uses the phrase "lose my mind" in addition of their symptoms. Most of the authors describes the symptoms from first person which makes this group suitable for tracing of the source.

Indicators if symptoms keywords are used in the microblogs' text are added. But because these are ordinary citizens and not doctors they might use non medical terms to describe their condition. A survey of the microblogs yields the following keywords and their classification.

Classified as	Searched Terms
flu	flu, runny nose
fever	fever, temp, temperature
chills	chill
sweats	sweat
aches	ache, aching, sore, cramp
pains	pain, hurt
fatigue	fatigue
coughing	cough, pnenomia
breathing difficulty	breath
nausea	naus
vomiting	vomit, throwing up, puk
diarrhea	diarrhea
enlarged lymph node	lymph

After exploring the visualisations, the following keywords are tracks as well.

Classified as	Searched Terms
Accident	acci
Convention	conven
Explosion	explo
Truck	truck

< Background

Home

Preliminary >