Maximum Project Findings
The pre-survey dataset had 1,455 records, and the post-survey dataset had 414 records. On merging, we only had 292 records where the pre and post surveys were both done by the respondents.
We conducted the following cleaning procedures:
- Irrelevant and Duplicate Fields: Removed from our dataset.
- Missing Data: Removed from our data set, null values were replaced with 0s to facilitate our analysis.
- Rectifying Discrepancies: Ensured data from pre and post survey were in comparable formats.
- Data Transformation: Transformed pre-survey training questions to numerical format.
- Standardisation: Standardised naming conventions for the variables in the merged data.
We conducted the following preparation procedures:
- Exploratory Data Analysis: Created calculated fields for divergent bar charts
- Chi-squared tests: Split the data into different skills
- Text Analysis: Comments were transformed to show unique words and their frequencies
Divergent Stacked Bar Graphs:
Visualisations give quick insights from data, allowing targeted analysis. For Likert data, using divergent stacked bar graphs is especially appropriate. It facilitates a visual comparison of respondents’ answers to the survey (Heiberger and Robbins, 2011). For our purpose, divergent stacked bar graphs allow us to see the general levels of confidence for the different research skills. By comparing the graphs using pre and post confidence data, we can understand whether there was an improvement in the responses.
Chi-Squared Tests for Independence:
To statistically determine whether the change, specifically improvements, in confidence were significant, the chi-squared test for independence was used. The chi-squared tests allow for us to conclude whether the distribution of the categorical variable (confidence) is related to the variable of our groups (training) (Kim, 2017).
There is ongoing debate as to the appropriateness of a chi-squared test on paired data. Some discuss using paired t-test or Wilcoxon test when working with paired data (Derrick and White, 2017). However, these approaches assume an equal spacing between the categories on the Likert scale, which is spurious for the confidence categories in our data. Furthermore, in conducting the chi-squared analysis, we obtain contingency tables that help us see detail in any improvements in confidence.
Word Frequency Analysis: Text comments from the end of the survey can be analysed to find out respondents’ concerns not captured by the survey. This is done with the underlying assumption words that appear more frequently indicate an issue that students care more about (Stemler, 2001).
Percentage of students who received training in library research skills
| Skill | Highest Majority | Lowest Majority |
|---|---|---|
Effect of Training of Confidence Levels
Confidence level before and after the semester of SMU students who did not receive any training in conducting overall library research skills:
There is a positive shift in the confidence of students in the skills of ‘Searching the Internet Using Google’, ‘Evaluating Information’, ‘Scoping your Topic’ and ‘Searching Keywords’. The confidence for ‘Citing References’, ‘Creating Reference Lists’ and ‘Searching Library Online Databases’ shows a negative shift.
Confidence level before and after the semester of SMU students who received training in conducting overall library research skills:
There is a positive shift in the confidence of students in all the skills.
The same process was repeated for each of the skills and the following are the insights discovered:
| Skill | Change in confidence without training | Change in confidence with training |
|---|---|---|
To establish whether the observed improvements were statistically significant, we conducted chi-squared goodness of fit tests. For each research skill category, pre levels of confidence were measured against postlevels. The sample chosen to conduct the analysis was of students who had gone through training during the semester to allow us to conclude if the trainings had any real impact on confidence levels.
H0: Pre-confidence and post-confidence are independent.
H1: H0 is false.
The significance level was set at 5%
Positive and significant results were obtained for ‘Creating Reference Lists’. From the contingency table, we can see an overall improvement in confidence, implying effective training for this skill. For the skills ‘Searching Library Online Databases’, ‘Evaluating Information, Citing References’, and ‘Searching using Keywords’ the results were positive and significant as well. However, results from the contingency table were mixed and indicate groups who's confidence decreased after having gone through training. Thus, these are areas in which the library's trainings are somewhat effective, but may be improved.
Results for the skills ‘Searching the Internet using Google’ and ‘Scoping Your Topic’ were not significant. It may be concluded that for these skills, trainings had no notable impact and the library should focus the most on training for these skills.
We used tableau to conduct text analysis on a total of 53 comments. The following chart was created to show the word frequencies. Only words with a frequency greater than 1 were included. As the number of unique words that are representative of how students feel about the library were very few, furthur analysis could not be performed.
We sought to provide the library with analysis that would help them provide better training for the suite of research skills. We reccommend the library to focus on certain specific matters. Students from the school of law are experiencing a reduction in confidence in skills such as creating reference lists after receiving training.The library training being catered to all the schools is unable to match up to the needs of the students of the law school, leading to a drop in their confidence levels despite the training.
Particular skills like Searching the Library Online Databases and Searching using Keywords showed significance in the relationship between training and confidence level. However, several students showed a negative trend in confidence after the passage of a semester. Particularly, as noted, results for Scoping Your Topic and Searching the Internet Using Google were insignificant. This implies that there is no evidence that the trainings that were provided had any impact on these skills.
Thinking into the future, we recommend the library to continue tracking the confidence of students. With more data over time, analysis may be modelled to more precisely measure the effects of library trainings.
Response Rate
We had a relatively small sample of responses for both the surveys. The analysis can be greatly improved, if more students participated in both surveys. With more data points, the results will be more robust and convincing. For example, the library could follow the example of the end of term course evaluation survey that rewards students with e-dollars for their participation.
Training Data
It was challenging to give recommendations on improving research skills. Respondents only answered about whether or not they attended trainings, but no further information was asked of them about trainings. Moreover, our analysis showed comments that were given were inconsequential to library research skills or training. The chi-squared test analysis gave some insight into the effectiveness of trainings, but more information is needed for studying specific areas of weakness for library trainings.
- Derrick, B. and White, P. (2017) Comparing two samples from an individual Likert question. International Journal of Mathematics and Statistics, 18 (3). ISSN 0974-7117 http://eprints.uwe.ac.uk/30814
- Heiberger M., Robbins, N B. (2011). Plotting Likert and Other Rating Scales. Proceedings of the 2011 Joint Statistical Meeting
- Kim HY. (May 2017). Statistical notes for clinical researchers: Chi-squared test and Fisher's exact test. Restor Dent Endod. 42(2):152-155. https://doi.org/10.5395/rde.2017.42.2.152
- Stemler, Steve. (2001). An overview of content analysis. Practical Assessment, Research & Evaluation, 7(17). http://PAREonline.net/getvn.asp?v=7&n=17
