Difference between revisions of "ANLY482 AY2016-17 T2 Group7: Text Mining"

From Analytics Practicum
Jump to navigation Jump to search
m
m
Line 129: Line 129:
 
[[File:lawnet iskandar rahmart.png|500px]]<br/>
 
[[File:lawnet iskandar rahmart.png|500px]]<br/>
 
When we searched “rahmart” or “iskandar bin rahmart”in lawnet we could not find anything as the correct name of the case should be “rahmat”, but SAS Text Miner grouped “rahmart” and “iskandar”, “bin” together so we speculate that many students searched for “iskandar bin rahmart” and found nothing. A recommendation system which will automatically link “rahmart” and the “iskandar bin rahmat” case would be welcomed.<br/>
 
When we searched “rahmart” or “iskandar bin rahmart”in lawnet we could not find anything as the correct name of the case should be “rahmat”, but SAS Text Miner grouped “rahmart” and “iskandar”, “bin” together so we speculate that many students searched for “iskandar bin rahmart” and found nothing. A recommendation system which will automatically link “rahmart” and the “iskandar bin rahmat” case would be welcomed.<br/>
 
 
<div style="background:#307FBB; line-height:0.3em; font-family:sans-serif; font-size:120%; border-left:#bbdefb solid 15px;"><div style="border-left:#fff solid 5px; padding:15px;"><font color="#fff"><strong>Interim Gap Analysis</strong></font></div></div>
 
 
<big>'''Excessive System Logging of Search Queries'''</big>
 
 
In our EDA, we discovered that there exists a problem of excessive system logging of search queries. We have found 2 examples of such occurrence:
 
 
{| class="wikitable"
 
|-
 
! '''Time''' !! '''Search Query Logged'''
 
|-
 
| 12:55:02PM || Re
 
|-
 
| 12:55:04PM || Resol
 
|-
 
| 12:55:06PM || Resoluti
 
|-
 
| 12:55:08PM || Resolution
 
|}
 
Example 1: Log data is logged every 2 second
 
 
{| class="wikitable"
 
|-
 
! Key Press!! Search Query Logged
 
|-
 
| 1st Key Press: T || T
 
|-
 
| 2nd Key Press: r || Tr
 
|-
 
| 3rd Key Press: u || Tru
 
|-
 
| 4th Key Press: m || Trum
 
|-
 
| 5th Key Press: p || Trump
 
|}
 
Example 2: Log Data is logged with every key press
 
 
In our analysis, these presents a problem to us in the form of how do we determine which is the actual search query that a User is searching for? As illustrated by the example by ‘User A’ below, in a single session logged by ‘User A’, there may be multiple search queries searched by users. In this case, we used 3 search queries as an example. The challenge to us is to sieve out which are the search queries (eg. Jack, Singapore) that User A is searching for when it is not the end of the session for him.
 
 
Eg. List of 3 Search Queries being logged with every key press by User A:
 
 
[ Start of Session for User A ]
 
 
Re
 
 
Regu
 
 
Regula
 
 
Regulati
 
 
Regulation
 
 
Ja
 
 
Jack
 
 
Si
 
 
Sing
 
 
Singap
 
 
Singapor
 
 
Singapore
 
 
[ End of Session for User A ]
 
 
We decided that this shortfall not only affects us as project analysts, but to other stakeholders as well.
 
 
<big>'''Interim Gap Analysis by Stakeholders'''</big>
 
 
The '''Actual Performance''' in this case would be if everything remains status quo, meaning the problem of multiple logging of search queries would persist.
 
 
The '''Desired Performance''' in this case would be if this problem does not exist and 1 line of logging is created for 1 full, actual search query.
 
 
{| class="wikitable"
 
|-
 
! Stakeholders Involved/Impact of Performance !! Actual Performance !! Desired Performance
 
|-
 
| ''Our Team as Project Analysts'' || Presents a problem whereby we need to find out how to determine which line of search query logged is the actual, full search query by end-users so that we can begin the analysis from there || Every line of search query would be the actual, full search query by end-users so we need not clean the dataset even further, thereby reducing the amount of work we have to do and saves time which can be better spent in progressing the analysis
 
|-
 
| ''End-Users of Library’s e-Resources'' || Presents a problem whereby end-users may experience unnecessary lag in obtaining the results from their search queries || No lag when completing searches would mean a better overall user experience. Furthermore, such seamless experience would mean that the system do not stand in the way of the intensive research that students have to do in their course of study, but rather serving as an effective aid to them.
 
|-
 
| ''Library Team as Project Sponsors for this Practicum'' || Presents a problem whereby the project sponsors run a risk of the project analysts not being able to sieve out the line of search queries which are full, actual and useful to determine the accurate search queries that users are actually searching for || No such problem as whatever the search query is, it would be logged as exactly that.
 
|-
 
| ''Library Team in charge of ensuring that the EzProxy server serves the users in the best possible way'' || Wastage of resources and can potentially slow down the servers when multiple logs are triggered and recorded before searches are completed. This utilizes processing RAM of the server unnecessarily and takes up precious memory space when being recorded as a line of search query. || There would be no wastage of server’s processing RAM and memory space as 1 line of logging would be created for 1 full, actual search query entered by users.
 
|}
 
  
  

Revision as of 23:51, 6 April 2017

Home

Team

Project Overview

Project Findings

Project Management

Documentation


With reference to Chart 11 in the Exploratory Data Analysis, we have selected 2 databases, Lawnet and Euromonitor, to focus on for this interim phase. This is due to the fact that these 2 databases are the most commonly used amongst the Law and Business students respectively, as these 2 schools are the 2 biggest contributors to the searches during the Term.

From the following actions applied to these 2 databases, we could then repeat these steps for the rest of the databases.

Text Analytics


Text1.png
Chart 12: Text analytics data preparation

Firstly, we need to format the search queries to lowercase form for standardization purposes. We do that by using Tableau’s ‘LOWER()’ function, filtering out two data sets: euromonitor’s data being “euromonitor_text_data” and lawnet’s data being “lawnet_text_data”.


Text2.png
Chart 13: Text mining process

After which we use SAS Enterprise Miner 14.1 to carry out text analytics. We import ‘euromonitor_text_data’ and ‘lawnet_text_data’ respectively by using the File Import function and running though the text mining process in Chart 13: Text mining process.


Text3.png
Chart 14: Text Parsing Configuration

We configure text parsing so that Parts of Speech such as ‘Aux’, ‘Conj’, ‘Det’, ‘Interj’, ‘Part’, ‘Prep’, ‘Pron’ and Types of Attributes including ‘Num’ and ‘Punct’ are all ignored.

Euromonitor


Chart17.png
Chart 15: Search Count in Euromonitor by Schools & Admission Years

Subject Matter: Contrasting BBM against Bsc(IS) users across Admission Years
Thought Process: As our team consists of BBM and Bsc(IS) students, we discussed among ourselves and then with our peers of our faculties about how often we use Euromonitor in our research. Amidst our sharings, we found out that more often than not, BSc(IS) students do not use Euromonitor as much as their BBM counterparts. However, some of the BSc(IS) students shared that they have used Euromonitor rather intensively in their 1st - 2nd years, mainly for researching on the University Core modules which they have to take (eg. TWC, BGS).

Thus, we attempted to verify this discussion through the analysis of the data.

Analysis: From Chart 15, we observe that the number of searches performed by BSc(IS) users across all admission years are significantly lower than their BBM counterparts. Thus, this could possibly verify our thoughts that BSc(IS) users indeed use Euromonitor for research lesser than their BBM counterparts.

Most interestingly, BSc(IS) users in AY_2016 have performed the most number of searches as compared to their faculty users from the other admission years. The first year of the SMU BSc(IS) curriculum usually consists of University Core Modules such as BGS (Business, Government & Society) and TWC (Technology and World Change) which are by nature, research-intensive modules. Thus, it would be more probable that BSc(IS) users in AY_2016, meaning they are in their first year in 2016, are performing such high number of searches because they are enrolled in such research-intensive modules. The number of research-intensive modules in the curriculum decreases significantly as the typical BSc(IS) user moves into his/her 2nd year and thereafter. This could be shown by the low number of searches performed by BSc(IS) users in AY_2015 (1st/2nd Year in 2016), AY_2014 (2nd/3rd Year in 2016) and AY_2013 (3rd/4th Year in 2016.

Contrasting with BBM users, the number of searches across all academic years remains high. This could be due to the nature of the BBM curriculum which consists of research-intensive modules throughout.

Plots (30720).png
Among 30720 cases, 8257 (26.88%) are dropped after parsing the data.

Plots (frequency w singapore).png
In addition to parsing of the data, we noticed that the Term “singapore” has the greatest frequency of 2175, followed by the Terms “consumer” and “tourism”.

Plots (concept linking Singapore).png
From the graph above we noticed that ‘singapore’ is linked to ‘hot drinks’, ‘hot’, ‘drink’, ‘singapore travel’, ‘consumer lifestyle’, ‘lifestyle’, ‘singapore consumer’ and ‘singapore airline’

Plots (concept linking consumer).png
From the graph above we noticed that ‘consumer’ is linked to ‘consumer health’, ‘consumer foodservice’, ‘electronics’, ‘’, ‘singapore consumer’, ‘trend’, ‘consumer electronics’ and ‘global’.

Plots (concept linking tourism).png
From the graph above, we noticed ‘tourism’ is linked to ‘medical’, ‘sport’, ‘cultural tourism’, ‘wellness’, ‘medical tourism’, ‘cultural’, ‘wellness tourism’ and ‘travel’.

Plots (text topic function).png
This is the result shown by function Text Topic.

Plots (text topic function enlarged).png
From the graph above, results from Text Topic function shows that “singapore”, “retail”, “beer”, “milk” and “juice” are of the same topic, “medical”, “tourism”, “technology” and “health” are of the same topic, and “lifestyle”, “consumer”, “singapore”, “japan” are of the same topic.

Lawnet

Plots (lawnet dashboard).png
Among 172363 cases, 36066 are dropped (20.92%). As compared to euromonitor, lawnet has a larger amount of searches.

Plots (concept linking slr).png
The most popular search Terms are ‘slr’, ‘ltd’ followed by ‘pte’. These stands for Singapore Law Review, Ltd as in Pte Ltd and Pte as in Pte Ltd respectively. This means that students search for Singapore Law Review a lot.

Plots (concept linking slr enlarged).png
From the graph above, we noticed that ‘slr’ which stands for Singapore Law Review, is linked to words which are presumably names such as ‘chum tat’, ‘ngiam’, ‘chiew’, ‘chiew hock’, ‘chum’ and the time period ‘1974-1976’. These could possibly tell us the popular cases associated with the Singapore Law Review and the time period for which cases took place in.

Plots (concept linking singapore lawnet).png
From the graph above, we noticed that ‘singapore’ is linked to words such as ‘overseas enterprise’, ‘pte’ (presumably pte in Pte Ltd, the short form for Private Limited), ‘global singapore’, ‘southeast’, ‘finance’, ‘institutional’, ‘law’, ‘ltd’ (presumably Ltd in Pte Ltd) and ‘development bank’.

Plots (text topic function lawnet).png
This is the result shown by function Text Topic.

Plots (text topic function lawnet enlarged).png
From the table above, results from Text Topic function shows that “slr”, “wlr”, “teck”, “attorney-general” are of the same topic, this is possibly because people who searched for singapore law review (slr), also searched for world law review (wlr) while the attorney-general is the legal advisor to the government and “teck” could be someone’s name. “sghc”, “bin”, “rahmart” and “iskandar” are of the same topic as “sghc” stands for singapore high court.

The name ‘Rahmart’, ‘bin’ and ‘Iskandar’ is an interesting search Term whereby it features a former policeman of the name ‘Iskandar bin Rahmat’ who was charged for committing double murder at Kovan MRT in 2013. This is a widely known local criminal case which most probably is being used as a prime example of criminal cases in the SMU Bachelor of Laws Curriculum, thereby explaining the popularity of these keywords. More interestingly, the ‘Rahmart’ is in fact a misspelling of the name ‘Rahmat’. This could possibly indicate that majority of the searches for ‘Rahmart’ were performed by users who are not of the Malay descent. Or this could possibly be due to a misspelling from the course material that was provided to the users, presumably Law students.

Lawnet iskandar.png Lawnet rahmart.png
Lawnet iskandar rahmart.png
When we searched “rahmart” or “iskandar bin rahmart”in lawnet we could not find anything as the correct name of the case should be “rahmat”, but SAS Text Miner grouped “rahmart” and “iskandar”, “bin” together so we speculate that many students searched for “iskandar bin rahmart” and found nothing. A recommendation system which will automatically link “rahmart” and the “iskandar bin rahmat” case would be welcomed.


[Back To Project Page]