Difference between revisions of "ANLY482 AY2016-17 T2 Group16: HOME/Interim"
| Line 36: | Line 36: | ||
<div><font face="Roboto"> | <div><font face="Roboto"> | ||
| − | <big>'''Overview'''</big><br/> | + | === <big>'''Overview'''</big><br/> === |
The objective of this project is to provide insights on eBook databases and their users for Li Ka Shing Library Analytics Team. The analysis is not limited to eBook databases but also studies the general traits of other databases. As much of analysis is done on proxy server request logs, data cleaning is a major component of this project. The analysis results will help Li Ka Shing Library understand the usage pattern of its users, and better serve SMU community with increasing demand for professional knowledge. | The objective of this project is to provide insights on eBook databases and their users for Li Ka Shing Library Analytics Team. The analysis is not limited to eBook databases but also studies the general traits of other databases. As much of analysis is done on proxy server request logs, data cleaning is a major component of this project. The analysis results will help Li Ka Shing Library understand the usage pattern of its users, and better serve SMU community with increasing demand for professional knowledge. | ||
| Line 45: | Line 45: | ||
There are users other than students (e.g. alumni, staff, visiting students and anonymous users), but the scope of this project is only limited to students because of the availability and insightfulness of student data. | There are users other than students (e.g. alumni, staff, visiting students and anonymous users), but the scope of this project is only limited to students because of the availability and insightfulness of student data. | ||
| − | === Exploratory Data Analysis === | + | === <big>'''Exploratory Data Analysis'''</big><br/> === |
Exploratory data analysis (EDA) was done on student data and request log data respectively. Through EDA, we hope to: | Exploratory data analysis (EDA) was done on student data and request log data respectively. Through EDA, we hope to: | ||
* understand data volume and dimensions | * understand data volume and dimensions | ||
| Line 52: | Line 52: | ||
* determine proper analysis approaches | * determine proper analysis approaches | ||
| − | ===='''Student'''==== | + | ===='''Student'''==== |
: 1. '''Understanding the data''' : | : 1. '''Understanding the data''' : | ||
The headers of student data are not self-explanatory. We studied the values of each columns and determined the meeting for “statistical categories” as is shown in the table below. | The headers of student data are not self-explanatory. We studied the values of each columns and determined the meeting for “statistical categories” as is shown in the table below. | ||
Revision as of 18:20, 22 April 2017
Contents
Overview
The objective of this project is to provide insights on eBook databases and their users for Li Ka Shing Library Analytics Team. The analysis is not limited to eBook databases but also studies the general traits of other databases. As much of analysis is done on proxy server request logs, data cleaning is a major component of this project. The analysis results will help Li Ka Shing Library understand the usage pattern of its users, and better serve SMU community with increasing demand for professional knowledge.
Data Overview
The data we will work with is request log data (a.k.a. digital trace) and student data. Request log is a NCSA Common Log Format (CLF) data with billions of record captured by the library’s URL rewriting proxy server. This dataset captures all user request to external databases. The record attributes are user ID, request time, http request line (method, URL, and protocol), response time, and user agent. The student data, specifying faculty, admission year, graduation year, and degree program, is also provided in csv format for the team. For non-disclosure reason, the user identifier - emails - are obfuscated by hashing to a 64-digit long hexadecimal numbers. The hashed ID will be used to link up two tables. Please refer to appendix for the complete data dimensions and samples. The request log records are filed by months. The monthly numbers of records in request log data vary from 3 million to 6 million and the file sizes around 2 GB. Student dataset contains 22,427 records for students not only limited to full-time but also postgraduates and exchange students. There are users other than students (e.g. alumni, staff, visiting students and anonymous users), but the scope of this project is only limited to students because of the availability and insightfulness of student data.
Exploratory Data Analysis
Exploratory data analysis (EDA) was done on student data and request log data respectively. Through EDA, we hope to:
- understand data volume and dimensions
- assess data quality, including completeness, validity, consistency and accuracy
- formulate hypotheses for analysis and
- determine proper analysis approaches
Student
- 1. Understanding the data :
The headers of student data are not self-explanatory. We studied the values of each columns and determined the meeting for “statistical categories” as is shown in the table below.
- 2. Data quality :
By comparing the number of student IDs with unique number of students, we confirmed there is not duplicate identifier. Besides, there is no missing values under faculty, degree name and admission year. We filled in the missing values in graduation year “in process”.
- 2. Student program distribution :
To find out the faculty distribution, we plotted bar chart Chart 1. From the bar chart, we observed that most of the students who visited the school library website for research are business students followed by “Others”. We investigated who make up the “Other” category by summarising categories in “Degree” column.
Based on Table 2, we analysed the “Others” category and found out that most of them belong to International Exchange Programme, followed by Local Exchange Programme student and H3 Program students.
