AY1516 T2 Team AP Analysis PostInterimPlan

HOME

OVERVIEW

ANALYSIS

PROJECT MANAGEMENT

DOCUMENTATION

Data Retrieval & Manipulation		Findings		Post interim plan

Facebook Graph API (Post Interim Plan)

Apart from analysing one of SGAG's popular social network Twitter, we plan to leverage the Facebook Graph API. Drawing from our experience using the twitter API, we are looking to crawl Facebook data in a similar fashion, crawling, retrieving and aggregating post-level Facebook data. Hopefully, this process can yield conclusive results about the SGAG's social network (likes, shares, etc) on Facebook.

Approach (Post Interim Plan)

Step	Expected Result	Notes
1	Collect all post data	Get all posts of SGAG on SGAG page, preferably all posts from 2 years ago to date For each post, look out for 'Like' count, and when Graph API 2.6 is out, look out for the other Facebook reactions as well
2	All user objects for each like, for every post	For each follower, explore whether privacy options by users will limit us to further classify these user objects Analyse 'Like' count per post, and if possible chart 'Likers' social network on a post-level basis Manually categorize posts, like what we did for Twitter, and possibly prune them, to further refine insights
3	"Comment-Level" per post and number of shares on a "user-level"	Analyse posts by its comments Are posts that are highly commented popular? Analyze users who actually "Share" sgag's posts, and their connection with other users

Data Retrieval

Constructing the graph from scratch involved the usage of python code to retrieve posts from SGAG's Facebook account for posts dating back 10 months. This involved connecting to the Facebook graph API programatically to formulate a csv file that resembles this structure:

Each user ID in List of Likers and List of Commenters are separated by a semicolon, and tagged to each post.

Post ID	List of Likers	List of Commenters
378167172198277_1187053787976274	10206930900524483;1042647259126948;10204920589409318; ...	10153979571077290;955321504523847;1701864973403904; ...

After crawling the Facebook API for ~4.5 Hours, the result is 1600++ posts dating 10 Months ago, with a CSV file size of ~38MB. Entire code can be viewed here.

Code snippet of likers & commenters retrieval

Code snippet of conversion of CSV into GraphML format

Initial import report in GraphML

Subsequently, we wanted to visualize the data using the Gephi tool. Hence, additional python code was used to read the CSV file, programmatically reading each row of the CSV, and attaching each post ID to likers and commenters respectively. This is done so that we can construct the .graphml graph formatted file, which gephi is able to read. Entire code can be viewed here.

The resultant file (~211MB) is uploaded here for reference.

Gephi Analysis

The initial import into gephi constant crashes when using computers with < 8 GBs of RAM. We eventually managed to leverage a computer with 16GB of RAM to begin the initial import of the huge .graphml file. The initial report indicated that we have 296,063 Nodes and 2,142,208 edges.

Upon successful load of the file and running the Force Atlas 2 Algorithm, we achieved the below graph.

As expected, the graph is extremely unreadable, and furthermore, even with a high performance machine, the Gephi application is extremely slow and painful to work with under these circumstances.

Hence, we performed a filter of the the number of degrees on the graph to obtain a condensed version below.

From a high level view of the graph, we noticed that for some clusters(located at the side of the graph) have a mixture of both commenters(Green) and likers(Red), with a large proportion of interaction being "likes". However, even though with the filter applied, working with Gephi is extremely laggy and unusable to get concrete findings, and thus we decided to look for alternatives in order to obtain additional insights.