Difference between revisions of "AY1516 T2 Team AP Analysis PostInterimPlan"

From Analytics Practicum
Jump to navigation Jump to search
 
(2 intermediate revisions by the same user not shown)
Line 27: Line 27:
 
{| style="background-color:#ffffff; margin: 3px auto 0 auto" width="55%"
 
{| style="background-color:#ffffff; margin: 3px auto 0 auto" width="55%"
 
|-  
 
|-  
−
! style="font-size:15px; text-align: center; border-top:solid #ffffff; border-bottom:solid #ffffff" width="150px"| [[AY1516 T2 Team AP_Analysis| <span style="color:#3d3d3d">Data Retrieval & Manipulation</span>]]
+
! style="font-size:15px; text-align: center; border-top:solid #ffffff; border-bottom:solid #ffffff" width="150px"| [[AY1516 T2 Team AP_Analysis| <span style="color:#3d3d3d">Data Retrieval & Manipulation (Pre Interim)</span>]]
 
! style="font-size:15px; text-align: center; border-top:solid #ffffff; border-bottom:solid #ffffff" width="20px"|
 
! style="font-size:15px; text-align: center; border-top:solid #ffffff; border-bottom:solid #ffffff" width="20px"|
  
−
! style="font-size:15px; text-align: center; border-top:solid #ffffff; border-bottom:solid #ffffff" width="150px"| [[AY1516 T2 Team AP_Analysis_Findings| <span style="color:#3d3d3d">Findings</span>]]
+
! style="font-size:15px; text-align: center; border-top:solid #ffffff; border-bottom:solid #ffffff" width="150px"| [[AY1516 T2 Team AP_Analysis_Findings| <span style="color:#3d3d3d">Pre interim findings</span>]]
 +
! style="font-size:15px; text-align: center; border-top:solid #ffffff; border-bottom:solid #ffffff" width="20px"|
 +
 
 +
! style="font-size:15px; text-align: center; border-top:solid #ffffff; border-bottom:solid #ffffff" width="150px"| [[AY1516 T2 Team AP_Analysis_PostInterimTwitterFindings| <span style="color:#3d3d3d"> Post interim twitter findings</span>]]
 
! style="font-size:15px; text-align: center; border-top:solid #ffffff; border-bottom:solid #ffffff" width="20px"|
 
! style="font-size:15px; text-align: center; border-top:solid #ffffff; border-bottom:solid #ffffff" width="20px"|
  
Line 36: Line 39:
 
! style="font-size:15px; text-align: center; border-top:solid #ffffff; border-bottom:solid #ffffff" width="20px"|
 
! style="font-size:15px; text-align: center; border-top:solid #ffffff; border-bottom:solid #ffffff" width="20px"|
  
−
! style="font-size:15px; text-align: center; border-top:solid #ffffff; border-bottom:solid #2e2e2e" width="150px"| [[AY1516 T2 Team AP_Analysis_PostInterimFindings| <span style="color:#3d3d3d"> Post interim plan</span>]]
+
! style="font-size:15px; text-align: center; border-top:solid #ffffff; border-bottom:solid #ffffff" width="150px"| [[AY1516 T2 Team AP_Analysis_PostInterimFindings| <span style="color:#3d3d3d"> Post interim findings</span>]]
 
! style="font-size:15px; text-align: center; border-top:solid #ffffff; border-bottom:solid #ffffff" width="20px"|
 
! style="font-size:15px; text-align: center; border-top:solid #ffffff; border-bottom:solid #ffffff" width="20px"|
 
|}
 
|}
Line 76: Line 79:
  
 
</table>
 
</table>
−
 
−
==<div style="background: #232AE8; line-height: 0.3em; font-family:helvetica;  border-left: #6C7A89 solid 15px;"><div style="border-left: #FFFFFF solid 5px; padding:15px;font-size:15px;"><font color= "#ffffff"><strong>Data Retrieval</strong></font></div></div>==
 
−
 
−
Constructing the graph from scratch involved the usage of python code to retrieve posts from SGAG's Facebook account for posts dating back 10 months. This involved connecting to the Facebook graph API programatically to formulate a csv file that resembles this structure:
 
−
 
−
<table rules="all" width="80%" cellpadding="6" cellspacing="3" style="margin: 1em auto 1em auto; font-weight: normal;  border-style: solid">
 
−
<tr style="background-color:#E1F4FF;"><th>Post ID</th><th>List of Likers</th><th>List of Commenters</th></tr>
 
−
 
−
<tr><td width="10%" border="1" align="center">378167172198277_1187053787976274</td>
 
−
<td width="30%" border="1" align="center">10206930900524483;1042647259126948;10204920589409318; ...</td>
 
−
<td width="60%" align="center">10153979571077290;955321504523847;1701864973403904; ...</td></tr>
 
−
''Each user ID in List of Likers and List of Commenters are separated by a semicolon, and tagged to each post.''
 
−
</table>
 
−
 
−
After crawling the Facebook API for ~4.5 Hours, the result is 1600++ posts dating 10 Months ago, with a CSV file size of ~38MB. Entire code can be viewed [https://drive.google.com/a/smu.edu.sg/file/d/0B4ESKidr4zkINlNQaWxlZXByZFU/view?usp=sharing here].
 
−
 
−
[[File:Screen Shot 2016-04-09 at 5.22.10 pm.png|thumbnail|''Code snippet of likers & commenters retrieval'']]
 
−
[[File:Screen Shot 2016-04-09 at 5.34.50 pm.png|thumbnail|''Code snippet of conversion of CSV into GraphML format'']]
 
−
[[File:Screen Shot 2016-04-09 at 5.59.00 pm.png|thumbnail|''Initial import report in GraphML'']]
 
−
 
−
Subsequently, we wanted to visualize the data using the Gephi tool. Hence, additional python code was used to read the CSV file, programmatically reading each row of the CSV, and attaching each post ID to likers and commenters respectively. This is done so that we can construct the .graphml graph formatted file, which gephi is able to read. Entire code can be viewed [https://drive.google.com/a/smu.edu.sg/file/d/0B4ESKidr4zkIdWJNdGJWV3BVVVU/view?usp=sharing here].
 
−
 
−
The resultant file (~211MB) is uploaded [https://drive.google.com/a/smu.edu.sg/file/d/0B4ESKidr4zkIbnhPTWlnSU5sUms/view?usp=sharing here] for reference.
 
−
 
−
==<div style="background: #232AE8; line-height: 0.3em; font-family:helvetica;  border-left: #6C7A89 solid 15px;"><div style="border-left: #FFFFFF solid 5px; padding:15px;font-size:15px;"><font color= "#ffffff"><strong>Gephi Analysis</strong></font></div></div>==
 
−
 
−
The initial import into gephi constant crashes when using computers with < 8 GBs of RAM. We eventually managed to leverage a computer with 16GB of RAM to begin the initial import of the huge .graphml file. The initial report indicated that we have <b>296,063</b> Nodes and <b>2,142,208</b> edges.
 
−
 
−
Upon successful load of the file and running the Force Atlas 2 Algorithm, we achieved the below graph.
 
−
 
−
[[File:Screen Shot 2016-04-09 at 6.15.07 pm.png|900px]]
 
−
 
−
As expected, the graph is extremely unreadable, and furthermore, even with a high performance machine, the Gephi application is extremely slow and painful to work with under these circumstances.
 
−
 
−
Hence, we performed a filter of the the number of degrees on the graph to obtain a condensed version below.
 
−
 
−
[[File:Screen Shot 2016-04-09 at 6.24.36 pm.png|900px]]
 
−
 
−
From a high level view of the graph, we noticed that for some clusters(located at the side of the graph) have a mixture of both commenters(Green) and likers(Red), with a large proportion of interaction being "likes".
 
−
However, even though with the filter applied, working with Gephi is extremely laggy and unusable to get concrete findings, and thus we decided to look for alternatives in order to obtain additional insights.
 
−
 
−
==<div style="background: #232AE8; line-height: 0.3em; font-family:helvetica;  border-left: #6C7A89 solid 15px;"><div style="border-left: #FFFFFF solid 5px; padding:15px;font-size:15px;"><font color= "#ffffff"><strong>NetworkX approach</strong></font></div></div>==
 

Latest revision as of 19:17, 9 April 2016

Team ap home white.png HOME

Team ap overview white.png OVERVIEW

Team ap analysis white.png ANALYSIS

Team ap project management white.png PROJECT MANAGEMENT

Team ap documentation white.png DOCUMENTATION


Data Retrieval & Manipulation (Pre Interim) Pre interim findings Post interim twitter findings Post interim plan Post interim findings


Facebook Graph API

Apart from analysing one of SGAG's popular social network Twitter, we plan to leverage the Facebook Graph API. Drawing from our experience using the twitter API, we are looking to crawl Facebook data in a similar fashion, crawling, retrieving and aggregating post-level Facebook data. Hopefully, this process can yield conclusive results about the SGAG's social network (likes, shares, etc) on Facebook.

Approach

StepExpected ResultNotes
1 Collect all post data
  • Get all posts of SGAG on SGAG page, preferably all posts from 2 years ago to date
  • For each post, look out for 'Like' count, and when Graph API 2.6 is out, look out for the other Facebook reactions as well
2 All user objects for each like, for every post
  • For each follower, explore whether privacy options by users will limit us to further classify these user objects
  • Analyse 'Like' count per post, and if possible chart 'Likers' social network on a post-level basis
  • Manually categorize posts, like what we did for Twitter, and possibly prune them, to further refine insights
3 "Comment-Level" per post and number of shares on a "user-level"
  • Analyse posts by its comments
  • Are posts that are highly commented popular?
  • Analyze users who actually "Share" sgag's posts, and their connection with other users
Retrieved from ‘https://wiki.smu.edu.sg/ANLY482/index.php?title=AY1516_T2_Team_AP_Analysis_PostInterimPlan&oldid=7066’