HeaderSIS.jpg

IS480 Team wiki: 2017T1 Ravenous User Testing 1

From IS480
Jump to navigation Jump to search
Ravenouslogo.png


Iconh1.png
Iconau1.png
Iconpo1.png
Iconpm1.png
Icond.png

 

Ravenousinternaltest.png Ravenousuat12.png Ravenousuat2.png Ravenousuat3.png


General

About UAT 1

  • Date: 10th Aug 2017, Thursday
  • Time: 7:30P.M.
  • Duration: Approx 90 minutes per user
  • Venue: SMU Labs
  • Number of participants: 5


Objectives

  • To ensure that users are able to understand and interpret the charts
  • To gather feedback on the charts used for visualization
  • To check whether the flow of tasks for EvBot is logical and intutive
  • To check whether the behaviour of EvBot matches the user commands
  • To gather feedback to improve the intent and behaviour of the EvBot
  • To observe user's behaviour when using FaBot to manage facilities
  • To improve FaBot based on user feedback


Procedure

Before each test, Team Ravenous will introduce the application to the testers. The testing will be done in this order - FaBot, EvBot, Analytics Dashboard. For each test, they are asked to think aloud as they follow instruction from each document. Team Ravenous will be noting down the participants’ behaviours and any critical incidents. Participants are to leave their feedback at the end of each application test. At the end of each application test, Team Ravenous will asking the testers questions with regards to their behaviours and thought process as they navigate through the application. As the test may be long, testers are encourage to take breaks in between each application test

Conclusion of Analytics Dashboard User Testing 1

Test Plan

Click here for test instructions

Goals Reached

S/N Goals Reached? Remarks
1 Participants should be able to accurately gather statistics for claim rate and active rate across all agencies and PSD Yes All participants managed to interpret the data correctly and entered the correct values
2 Participants should be able to correctly draw an inference from the 4 claim and active rate charts Yes All participants were able to compare data from their own department to all agencies correctly. They were able to identify that PSD had a higher claim rate as compared to all agencies
3 Participants should be able to gather the statistics for number of logins on PC and mobile for different dates Yes All participants managed to interpret the data correctly and entered the correct values
4 Participants should understand what the "Group Activity" graph is trying to express Yes 2 out of 5 participants gave the answer we were looking for. The other participants did not give the answer we were looking for, but were able to correctly infer what the graph was trying to depict
5 Participants should be able to gather statistic for number of posts, shares, comments and likes in previous week Yes All participants managed to interpret the data correctly and entered the correct values
6 Participants should be able to draw Inference from the table of posts, shares, comments and likes Yes 2 out of 5 participants gave the answer we were looking for. The other participants did not give the answer we were looking for, but were able to correctly infer what the graph was trying to depict
7 Participants should be able to identify what “Story” the overview dashboard is trying to express. Yes "This is a very broad question. 2 out of 5 participants were able to 100% describe what the dashboard was about. The rest were able to identify parts of the dashboard story. Generally, all participants were able to understand the story of the dashboard."
8 Participants should correctly infer what the chart and the table “Engagement Score Across PSD Group Charts” are trying to depict. Yes All participants were able to correctly infer that the graph is trying to show
9 Participants should correctly infer what the doughnut chart is trying to depict Yes "Most participants inferred the chart correctly. 1 participant inferred the chart wrongly."
10 Participants should correctly infer what the chart “Group Activity Chart” is trying to depict. No Most participants could not interpret the graph correctly. Some could not understand what the graph was trying to tell.
11 Participants should be able to correctly infer both the charts together and know what both the charts are trying to depict Yes "Most of the participants were able to infer that Workplace Mobile is used more than WorkChat Mobile. Although some participants did not give us the exact answer we were looking out for, they were still able to understand the chart"
12 Participants should be able to correctly infer what the chart “Login Analysis” is trying to depict. No "Most participants could not interpret the graph correctly. Some could not understand what the graph was trying to tell. Furthermore, one participant was confused if the login is used for workplace or workchat. This is of concern to us."
13 Participants should be able to identify the top 3 day-and-time users are actively posting and commenting on posts Yes "All participants managed to interpret the data correctly and entered the correct values. 1 participant misinterpreted the question"
14 Participants should be able to decide when is the best time to post an article on Workplace@FB No All 5 participants had different answers. None of the participants got the correct answer
15 Participants should be able to Infer what the chart “Content on Workplace” is trying to depict. Yes All participants were able to correctly infer that the graph is trying to depict
16 Participants should be able to infer what the chart “Word Cloud” is trying to depict. Yes "Most participants were able to understand what the word cloud is for. 1 participant was not able to figure out what the word cloud is for"
17 Participants should be able to list down the top 5 words that capture their attention. No Most participants had different answers. Answers were accurate up to top 7 words

Key Findings

Function Observations/Users' Comments Changes to be made
Group Activity Chart - Group The phrase "moving" was not intuitive for participants, so participants could not interpret the chart accurately Rework the calculation of Group Activity Chart and the visualization of the chart.
Login Analysis - People The phrase "moving" was not intuitive for participants, so participants could not interpret the chart accurately Rework the calculation of Login Analysis Chart and the visualization of the chart.
Login Analysis - People Participants were confused if the data refers to logins specific to Workplace or the WorkChat app Explore helper tips to make the function clearer or, look at repositioning the chart to a more intuitive location
Post-Time & Comment-Time Analysis Chart - Content Non-intuitive naming of graph. Come up with a more intuitive header for this chart
Post-Time & Comment-Time Analysis Chart - Content Graph UI is too small. Difficult for users to read at a glance. Poor color choice too. Redesign/resize the graph for friendlier UI/UX
Word Cloud - Content Participants had trouble identifying the top 5 words used Redesign word cloud UI

- No overlap
- Shade/Coloring

Overall Results/Comments

  • Improvements can be made to User Interface in terms of sizing, color choice and spacing between charts
  • Participants suggested placing a short description of the chart in the form of a helper tip beside the chart header
  • Participants gave an average of 5.2 out of 6 in terms of usefulness of the charts and all participants were able to interpret 90% of the charts and graphs accurately without any prompts or helper descriptions. Thus, we still consider UAT 1 to be a success

Full Analytics Dashboard results can be found here

Conclusion of EvBot User Testing 1

Test Plan

Click here for test instructions

Goals Reached

S/N Goals Reached? Remarks
1 Creating a new private event Yes -
2 Register your Workplace Event with EvBot No EvBot was wrongly trained with incorrect trigger words. Corrective measures to retrain the bot is needed
3 View the Events EvBot is currently helping you with Yes -
4 Add questions to your event Yes -
5 View added questions Yes 1 participant had trouble getting the bot to respond correctly due to NLP intent mapping to respective english/grammar/adjective word

Needs to be relooked as we feel the participants choice of words is valid (survey, survey questions)

6 Remove a question Yes -
7 Closing an event Yes -
8 Sending out survey Yes -

Key Findings

Function Observations/Users' Comments Changes to be made
Register Workplace Event Participants were redirected to register attendance instead of registering an event Retrain EvBot AI to understand register workplace event and to forget register attendance. Need to think about other adjectives for taking attendance.
View survey questions EvBot prompted one participant to do a survey instead of view questions when participant keyed in 'survey' & 'survey questions' Figure out a way to distinctly separate the two, as both are valid phrases for the intended intent

Overall Results/Comments

  • Based on our post-test survey, all participants rated our chatbot highly in terms of understanding their intents and choice of key words. However, EvBot needs to be retrained in certain functions to have distinct triggers for closely related intents.
  • The team considers the UAT 1 for EvBot to be a success as participants were able to follow and carry out most of the intended instructions without any problems, with the exception of registering an event.

Full EvBot results can be found here

Conclusion of FaBot User Testing 1

Test Plan

Click here for test instructions

Goals Reached

S/N Goals Reached? Remarks
1 Participant should be able to make a facility booking for 24th August 2017 from 2:00pm-3:30pm for facility 1001. Yes Learning points: Better communication among team and more detailed instruction within the team is needed if preparation for user testing is needed beforehand
2 Participant should be able to find the ‘View bookings’ tab and the facility booking made on 2nd Sept 2017. Yes -
3 Participant should be able to delete the booking made on 2nd Sept 2017. Yes -
4 Participant should be able to find the search available facility menu and find a facility for 30th August 2017 from 10:00am-12:00am for 10 people. Yes Learning points: Better communication among team and more detailed instruction within the team is needed if preparation for user testing is needed beforehand
5 Participant should be able to book the Seminar Room with the capacity of 43 people. No Learning points: Better communication among team and more detailed instruction within the team is needed if preparation for user testing is needed beforehand
6 Participant should be able to find and delete the booking made 25th August 2017. Yes -

Key Findings

Function Observations/Users' Comments Changes to be made
Book Facility Time Validation FaBot seems to accept invalid time formats at random period and continue prompting user To debug the logic error in FaBot
Search Available Facility Search Available Facility function was not easily found by some participants Explore ways to make the function more visible
Book Facility Confirmation Participants suggested that it can be useful to have a confirmation before making a booking Explore ways to enforce a confirmation message before users book a facility

Overall Results/Comments

  • Certain tests were not able to be carried out as test data were not cleared after a test session. Better communication among team and more detailed instructions within the team is needed whenever preparation for user testing is required.
  • The team considers the UAT 1 for FaBot to be a success as most participants were able to follow and carry out most of the intended instructions without any problems, apart from those where test data were not cleared

Full FaBot results can be found here