Twitter Analytics: Documentation

Home

Project Overview

Project Management

Documentation

Findings

About Me

Data Data is collected manually from Twitter with Python and stored in SQLite database. Several keywords have been tried and retrieved such as “#ippt”, “#gaza” and “#MH17”. However, the data collected is deemed to be unrepresentative as it is seasonal (“#ippt” and “MH17”) which spikes high during a short period of time. On the other hand,“#gaza” keyword retrieves a lot of tweets within a short period of time which makes a better data. However, we may need to gather more data in terms of time frame and its granularity to find the suitable forecasting. Based on the processing speed limitation of R, this project will only look into 10,000 rows of data for efficiency. However, more data can be analyzed if time is not a constraint to the project. From the data gathered, various attributes are collected. However, the below will be the focus of this project: User name Post date Location Tweet content Data Cleansing Methodology Upon data exploration, the following methodologies for data cleansing is proposed: Extract tweets from SQLite Convert tweets to a data frame Convert tweets to corpus Change tweets to lower case Remove punctuations Remove stopwords Stemming words to retrieve their radicals Remove links from tweets Remove Retweet and Mentions Data Exploration Findings Several findings from data exploration to find pattern: “@..” can be used to identify the relationship between users and classified as one user “mentions” another user in the post “RT” can be used to indicate retweet of content by another user to indicate influencers Location is not recommended, as a selection features as users’ preference polarity exist. Some tends to disable their location tracking in their device while others may not. Hence, by separating the groups, there is a likelihood that only the same group of users are analyzed Project Approach The analytics project delivery and development utilizes the agile and iterative implementation approach. Hence, frequent communication with clients and teaching staff to gather inputs for model development and refinement will be emphasized in various stages Forecasting Approach Adapted from: http://www.analyticbridge.com/profiles/blogs/time-series-analysis-using- r-forecast-package Forecasting Approach Adapted from: http://sivaanalytics.wordpress.com/2013/10/10/sentiment-analysis-on- twitter-data-using-r-part-i/

Tools

Python

Python is a widely used high-programming language that emphasize on code readability and scalability. It syntax allows programmers to code in fewer lines of code as compared to C. Python is also much better for text mining/ web scraping/ file manipulation/ XML. Features in Python such as generators is able to make processing large number of flies an ease as compared to others

SQLite

SQLite is a free in-process library that implements a portable and no server solution as it writes directly to common media. It works well with R and Python and scalable enough for the needs of the project and client’s needs as compared to utilizing a common csv file. Moreover, data security can be monitored easily as compared to cloud solution.

R

R is an open source programming language that is developed by practicing statisticians and researchers for statistical analysis. R is also compatible with other tools such as SAS, SPSS, Oracle, SQLite, etc. There are available packages that meet the client’s requirement such as sentiment analysis and time series forecasting.

NodeXL

NodeXL is an extendible toolkit for network overview and exploration, which can be implemented as an add-in feature in Microsoft Excel spreadsheet software. NodeXL is able to combine analysis and visualization functions with familiar spreadsheet layout for data handling. NodeXL is explored to get a better understanding of available open source tool in the market for Social Network Analysis