HeaderSIS.jpg

Difference between revisions of "IS480 Team wiki: 2013T1 ThunderBolt Project Documentation Usability Test"

From IS480
Jump to navigation Jump to search
 
(18 intermediate revisions by the same user not shown)
Line 48: Line 48:
 
</td>
 
</td>
 
</tr></table>
 
</tr></table>
 +
===<div style="background: #d5a10d; padding: 15px; font-weight: bold; line-height: 0.3em"><font color= #FFFFFF>Internal Testing</font></div>===
 +
Internal testing was conducted using regression method at end of each iteration. Refer to our test plan, test case and results below.<br>
 +
'''Click [https://www.dropbox.com/s/7o3heujyzj23na2/test_plan.docx here] for our test plan'''<br>
 +
'''Click [https://www.dropbox.com/s/f6prulqjh3n3f6p/Test%20cases%20and%20results.xlsx here] for our test cases by iterations'''<br>
 +
 
===<div style="background: #d5a10d; padding: 15px; font-weight: bold; line-height: 0.3em"><font color= #FFFFFF>Usability Test 1</font></div>===
 
===<div style="background: #d5a10d; padding: 15px; font-weight: bold; line-height: 0.3em"><font color= #FFFFFF>Usability Test 1</font></div>===
 
====<font color="#200772">Date and Venue</font>====
 
====<font color="#200772">Date and Venue</font>====
Line 259: Line 264:
 
- 15 Students completed IS480 and students doing IS480 next term<br>
 
- 15 Students completed IS480 and students doing IS480 next term<br>
 
[[Image:UT2Photo.png|400px]][[Image:UT1VSUT2participantprofile.png|530px]]<br><br>
 
[[Image:UT2Photo.png|400px]][[Image:UT1VSUT2participantprofile.png|530px]]<br><br>
 +
|}
 +
====<font color="#200772">Scope of Test</font>====
 +
{| class="wikitable" style="text-align: center; height:50px"
 +
|+
 +
|-
 +
! scope="col" width="150" style="background-color:#200772"| <font color="#ffffff">Function</font>
 +
! scope="col" width="50" style="background-color:#200772"| <font color="#ffffff">Admin</font>
 +
! scope="col" width="130" style="background-color:#200772"| <font color="#ffffff">Course Coordinator</font>
 +
! scope="col" width="50" style="background-color:#200772"| <font color="#ffffff">Supervisor/Reviewer</font>
 +
! scope="col" width="50" style="background-color:#200772"| <font color="#ffffff">TA</font>
 +
! scope="col" width="50" style="background-color:#200772"| <font color="#ffffff">Student</font>
 +
|-
 +
 +
! scope="row" style="background-color:#eeeeee"|Single Sign On
 +
|style="text-align: center;|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|-
 +
! scope="row" style="background-color:#eeeeee"|Role switching
 +
|style="text-align: center;|-
 +
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;"|-
 +
|style="text-align: center;"|-
 +
|style="text-align: center;"|-
 +
|-
 +
 +
! scope="row" style="background-color:#eeeeee"|Create schedule
 +
|style="text-align: center;|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;"|-
 +
|style="text-align: center;"|-
 +
|style="text-align: center;"|-
 +
|-
 +
 +
! scope="row" style="background-color:#eeeeee"|View schedule
 +
|style="text-align: center;|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|-
 +
 +
! scope="row" style="background-color:#FBF8C7"|Edit schedule
 +
|style="text-align: center;background-color:#FBF8C7;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;background-color:#FBF8C7;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;background-color:#FBF8C7;"|-
 +
|style="text-align: center;background-color:#FBF8C7;"|-
 +
|style="text-align: center;background-color:#FBF8C7;"|-
 +
|-
 +
 +
! scope="row" style="background-color:#FBF8C7"|Archive schedule
 +
|style="text-align: center;background-color:#FBF8C7;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;background-color:#FBF8C7;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;background-color:#FBF8C7;"|-
 +
|style="text-align: center;background-color:#FBF8C7;"|-
 +
|style="text-align: center;background-color:#FBF8C7;"|-
 +
|-
 +
 +
! scope="row" style="background-color:#eeeeee"|Create booking
 +
|style="text-align: center;|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;"|-
 +
|style="text-align: center;"|-
 +
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|-
 +
 +
! scope="row" style="background-color:#FBF8C7"|Add optional attendees
 +
|style="text-align: center;background-color:#FBF8C7;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;background-color:#FBF8C7;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;background-color:#FBF8C7;"|-
 +
|style="text-align: center;background-color:#FBF8C7;"|-
 +
|style="text-align: center;background-color:#FBF8C7;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|-
 +
 +
! scope="row" style="background-color:#eeeeee"|View booking
 +
|style="text-align: center;|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|-
 +
 +
! scope="row" style="background-color:#eeeeee"|Edit booking
 +
|style="text-align: center;|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;"|-
 +
|style="text-align: center;"|-
 +
|style="text-align: center;"|-
 +
|-
 +
 +
! scope="row" style="background-color:#eeeeee"|Delete booking
 +
|style="text-align: center;|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;"|-
 +
|style="text-align: center;"|-
 +
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|-
 +
 +
! scope="row" style="background-color:#eeeeee"|Approve/Reject booking
 +
|style="text-align: center;|-
 +
|style="text-align: center;"|-
 +
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;"|-
 +
|style="text-align: center;"|-
 +
|-
 +
 +
 +
! scope="row" style="background-color:#FBF8C7"|Reject booking with reason
 +
|style="text-align: center;background-color:#FBF8C7;"|-
 +
|style="text-align: center;background-color:#FBF8C7;"|-
 +
|style="text-align: center;background-color:#FBF8C7;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;background-color:#FBF8C7;"|-
 +
|style="text-align: center;background-color:#FBF8C7;"|-
 +
|-
 +
 +
! scope="row" style="background-color:#eeeeee"|View my bookings
 +
|style="text-align: center;|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|-
 +
 +
! scope="row" style="background-color:#eeeeee"|Supervisor Availability
 +
|style="text-align: center;|-
 +
|style="text-align: center;"|-
 +
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;"|-
 +
|style="text-align: center;"|-
 +
|-
 +
 +
! scope="row" style="background-color:#FBF8C7"|TA sign up for filming
 +
|style="text-align: center;background-color:#FBF8C7;"|-
 +
|style="text-align: center;background-color:#FBF8C7;"|-
 +
|style="text-align: center;background-color:#FBF8C7;"|-
 +
|style="text-align: center;background-color:#FBF8C7;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;background-color:#FBF8C7;"|-
 +
|-
 +
 +
! scope="row" style="background-color:#eeeeee"|Add default milestones
 +
|style="text-align: center;|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;"|-
 +
|style="text-align: center;"|-
 +
|style="text-align: center;"|-
 +
|-
 +
 +
! scope="row" style="background-color:#eeeeee"|Add required attendees for different milestones
 +
|style="text-align: center;|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;"|-
 +
|style="text-align: center;"|-
 +
|style="text-align: center;"|-
 +
|-
 +
 +
! scope="row" style="background-color:#eeeeee"|Set presentation duration
 +
|style="text-align: center;|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;"|-
 +
|style="text-align: center;"|-
 +
|style="text-align: center;"|-
 +
|-
 +
|}
 +
<br>
 +
'''User Testing 2 has been expanded from User Testing 1 to include 5 more features, which are highlighted in yellow.'''
 +
 +
====<font color="#200772">Testing Method</font>====
 +
{| style="border: 1px solid darkgray; text-align: center;font-siz:14px; height:50px"
 +
|scope="row"  width="1000" style="text-align: left; background: #eeeeee"|
 +
A list of tasks based on different scenarios were given to participants and they need to follow the scenarios and figure out how to complete them in the system.
 +
Respective facilitators who sat beside them need to observe and record down how they were using the system to complete the task assigned.<br>
 +
Upon completion of all tasks, participants were given a link to complete a questionnaire with regards to their experiences of using our system and feedback any issues they faced during testing or improvements they would like to see in the future.
 +
|}
 +
====<font color="#200772">Test Results</font>====
 +
'''Comparison of User Testing 1 survey result and User Testing 2 survey result. Overall, MORE users agree with our system!<br>'''
 +
{| style="border: 1px solid darkgray; text-align: center;font-siz:14px; height:50px"
 +
|scope="row"  width="1000" style="text-align: left; background: #eeeeee"|
 +
 +
[[Image:UT2SurveyResult1.png|470px]][[Image:UT2SurveyResult2.png|490px]]<br><br>
 +
[[Image:UT2SurveyResult3.png|470px]][[Image:UT2SurveyResult4.png|490px]]<br><br>
 +
'''Most common feedback from User Testing 2'''<br>
 +
- Admin and Course Coordinator want to allow drag-n-drop when editing a booking<br>
 +
- Admin and Course Coordinator also want to request for reason when students delete booking<br>
 +
- Supervisors want their time slots to be pre-populate as unavailable instead of available<br>
 +
- Students are not sure if their supervisor/reviewers receive email to approve their booking<br>
 +
- TA wants to see other TAs' sign on on their availability page as well, so that they can swap with each other if can't make it last min<br>
 +
'''Click here to see our Response Plan:''' [https://www.dropbox.com/s/w8mxojxaaziyp31/Response%20Plan%20to%20User%20Testing%202.xlsx]<br>
 +
[[Image:UserTesting2MostCommonFeedback.png|450px]][[Image:UserTesting2MostCommonFeedback_2.png|450px]]<br><br>
 +
[[Image:UserTesting2MostCommonFeedback_3.png|450px]][[Image:UserTesting2MostCommonFeedback_4.png|450px]]<br><br>
 +
[[Image:UserTesting2MostCommonFeedback_5.png|450px]][[Image:UserTesting2MostCommonFeedback_6.png|450px]]<br><br>
 +
|}
 +
 +
====<font color="#200772">Documentation</font>====
 +
{| style="border: 1px solid darkgray; text-align: center;font-siz:14px; height:50px"
 +
|scope="row"  width="330" style="text-align: left; background: #eeeeee"|
 +
:1. '''Test Plan'''
 +
:#Test plan: [https://www.dropbox.com/s/e9x0ly6loav73mp/Usability%20Test%202%20Plan.pdf]<br>
 +
:2. '''Scenario Task Lists'''
 +
:#Admin task: [https://www.dropbox.com/s/vgmdjkyxew3fguz/Task%20List%20for%20Admin.pdf]<br>
 +
:#Course Coordinator task: [https://www.dropbox.com/s/gxbwylkunaxpoan/Task%20List%20for%20Course%20Coordinator.pdf]<br>
 +
:#Supervisor/Reviewer task: [https://www.dropbox.com/s/rsxg54wa06ef5nj/Task%20List%20for%20Supervisor.pdf]<br>
 +
:#TA task: [https://www.dropbox.com/s/4a9vxd850pzr6vx/Task%20List%20for%20TA.pdf] <br>
 +
:#Student task: [https://www.dropbox.com/s/qro3qll9pvyxz1l/Task%20List%20for%20Student.pdf]<br>
 +
:3. '''Qualtrics Survey Questions'''<br>
 +
:#Survey Link: [http://smusg.asia.qualtrics.com/SE/?SID=SV_6KmS3VTUpoGHdml]<br>
 +
:#Survey Raw Result:[https://www.dropbox.com/s/k0th7w46ocpfp8g/User%20Testing%202%20survey%20raw%20result.pdf]<br>
 +
:4. '''Participant Informed Consent Form'''<br>
 +
:#Consent Form: [https://www.dropbox.com/s/6ca0aaoew2n2zcs/Participant%20Informed%20Constent%20Form.pdf]
 +
|}
 +
 +
===<div style="background: #d5a10d; padding: 15px; font-weight: bold; line-height: 0.3em"><font color= #FFFFFF>Usability Test 3</font></div>===
 +
====<font color="#200772">Date and Venue</font>====
 +
{| style="border: 1px solid darkgray; text-align: center;font-siz:14px; height:50px"
 +
|scope="row"  width="1000" style="text-align: left; background: #eeeeee"|
 +
'''Date: 18 Nov 2013'''<br>
 +
'''Time: 2pm - 7pm'''<br>
 +
'''Venue: SIS MR4.3'''<br>
 +
|}
 +
 +
====<font color="#200772">Participants</font>====
 +
{| style="border: 1px solid darkgray; text-align: center;font-siz:14px; height:50px"
 +
|scope="row"  width="1000" style="text-align: left; background: #eeeeee"|
 +
'''No. of Participants: 25'''<br>
 +
'''Participant Profile:'''<br>
 +
- Admin<br>
 +
- 4 Supervisor/Reviewer<br>
 +
- 3 TA<br>
 +
- 17 Students<br>
 +
[[Image:UT1-3ParticipantProfile.png|530px]]<br><br>
 
|}
 
|}
 
====<font color="#200772">Scope of Test</font>====
 
====<font color="#200772">Scope of Test</font>====
Line 304: Line 540:
  
 
! scope="row" style="background-color:#eeeeee"|Edit schedule
 
! scope="row" style="background-color:#eeeeee"|Edit schedule
−
|style="text-align: center;|[[Image:transparent-green-checkmark.png|24px]]
+
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 
|style="text-align: center;"|-
 
|style="text-align: center;"|-
Line 312: Line 548:
  
 
! scope="row" style="background-color:#eeeeee"|Archive schedule
 
! scope="row" style="background-color:#eeeeee"|Archive schedule
−
|style="text-align: center;|[[Image:transparent-green-checkmark.png|24px]]
+
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 
|style="text-align: center;"|-
 
|style="text-align: center;"|-
Line 328: Line 564:
  
 
! scope="row" style="background-color:#eeeeee"|Add optional attendees
 
! scope="row" style="background-color:#eeeeee"|Add optional attendees
−
|style="text-align: center;|[[Image:transparent-green-checkmark.png|24px]]
+
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 
|style="text-align: center;"|-
 
|style="text-align: center;"|-
Line 369: Line 605:
  
 
! scope="row" style="background-color:#eeeeee"|Reject booking with reason
 
! scope="row" style="background-color:#eeeeee"|Reject booking with reason
−
|style="text-align: center;|-
+
|style="text-align: center;"|-
 
|style="text-align: center;"|-
 
|style="text-align: center;"|-
 
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 
|style="text-align: center;"|-
 
|style="text-align: center;"|-
 
|style="text-align: center;"|-
 
|style="text-align: center;"|-
 +
|-
 +
 +
! scope="row" style="background-color:#FBF8C7"|Cancel booking
 +
|style="text-align: center;background-color:#FBF8C7;"|-
 +
|style="text-align: center;background-color:#FBF8C7;"|-
 +
|style="text-align: center;background-color:#FBF8C7;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;background-color:#FBF8C7;"|-
 +
|style="text-align: center;background-color:#FBF8C7;"|-
 
|-
 
|-
  
Line 384: Line 628:
 
|-
 
|-
  
 +
! scope="row" style="background-color:#FBF8C7"|Presentation subscription
 +
|style="text-align: center;background-color:#FBF8C7"|-
 +
|style="text-align: center;background-color:#FBF8C7"|-
 +
|style="text-align: center;background-color:#FBF8C7"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;background-color:#FBF8C7"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;background-color:#FBF8C7"|[[Image:transparent-green-checkmark.png|24px]]
 +
|-
 
! scope="row" style="background-color:#eeeeee"|Supervisor Availability
 
! scope="row" style="background-color:#eeeeee"|Supervisor Availability
 
|style="text-align: center;|-
 
|style="text-align: center;|-
Line 393: Line 644:
  
 
! scope="row" style="background-color:#eeeeee"|TA sign up for filming
 
! scope="row" style="background-color:#eeeeee"|TA sign up for filming
−
|style="text-align: center;|-
+
|style="text-align: center;"|-
 
|style="text-align: center;"|-
 
|style="text-align: center;"|-
 
|style="text-align: center;"|-
 
|style="text-align: center;"|-
 
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 
|style="text-align: center;"|[[Image:transparent-green-checkmark.png|24px]]
 
|style="text-align: center;"|-
 
|style="text-align: center;"|-
 +
|-
 +
 +
! scope="row" style="background-color:#FBF8C7"|TA swap slots
 +
|style="text-align: center;background-color:#FBF8C7;"|-
 +
|style="text-align: center;background-color:#FBF8C7;"|-
 +
|style="text-align: center;background-color:#FBF8C7;"|-
 +
|style="text-align: center;background-color:#FBF8C7;"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;background-color:#FBF8C7;"|-
 
|-
 
|-
  
Line 422: Line 681:
 
|style="text-align: center;"|-
 
|style="text-align: center;"|-
 
|style="text-align: center;"|-
 
|style="text-align: center;"|-
 +
|-
 +
 +
! scope="row" style="background-color:#FBF8C7"|CSV file upload
 +
|style="text-align: center;background-color:#FBF8C7"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;background-color:#FBF8C7"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;background-color:#FBF8C7"|-
 +
|style="text-align: center;background-color:#FBF8C7"|-
 +
|style="text-align: center;background-color:#FBF8C7"|-
 +
|-
 +
 +
! scope="row" style="background-color:#FBF8C7"|Manage users
 +
|style="text-align: center;background-color:#FBF8C7"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;background-color:#FBF8C7"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;background-color:#FBF8C7"|-
 +
|style="text-align: center;background-color:#FBF8C7"|-
 +
|style="text-align: center;background-color:#FBF8C7"|-
 +
|-
 +
 +
! scope="row" style="background-color:#FBF8C7"|Generate reports
 +
|style="text-align: center;background-color:#FBF8C7"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;background-color:#FBF8C7"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;background-color:#FBF8C7"|-
 +
|style="text-align: center;background-color:#FBF8C7"|-
 +
|style="text-align: center;background-color:#FBF8C7"|-
 +
|-
 +
 +
! scope="row" style="background-color:#FBF8C7"|Admin/Faculty Dashboard
 +
|style="text-align: center;background-color:#FBF8C7"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;background-color:#FBF8C7"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;background-color:#FBF8C7"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;background-color:#FBF8C7"|-
 +
|style="text-align: center;background-color:#FBF8C7"|-
 +
|-
 +
 +
! scope="row" style="background-color:#FBF8C7"|TA sign up dashboard
 +
|style="text-align: center;background-color:#FBF8C7"|[[Image:transparent-green-checkmark.png|24px]]
 +
|style="text-align: center;background-color:#FBF8C7"|-
 +
|style="text-align: center;background-color:#FBF8C7"|-
 +
|style="text-align: center;background-color:#FBF8C7"|-
 +
|style="text-align: center;background-color:#FBF8C7"|-
 
|-
 
|-
 
|}
 
|}
 +
<br>
 +
 +
'''User Testing 3 has been expanded from User Testing 2 to include 8 more features, which are highlighted in yellow.'''
  
 
====<font color="#200772">Testing Method</font>====
 
====<font color="#200772">Testing Method</font>====
Line 430: Line 732:
 
A list of tasks based on different scenarios were given to participants and they need to follow the scenarios and figure out how to complete them in the system.
 
A list of tasks based on different scenarios were given to participants and they need to follow the scenarios and figure out how to complete them in the system.
 
Respective facilitators who sat beside them need to observe and record down how they were using the system to complete the task assigned.<br>
 
Respective facilitators who sat beside them need to observe and record down how they were using the system to complete the task assigned.<br>
−
Upon completion of all tasks, participants were given a link to complete a questionnaire with regards to their experiences of using our system and feedback any issues they faced during testing or improvements they would like to see in the future.
+
Upon completion of all tasks, participants were given a link to complete a questionnaire with regards to the quality of emails they received from the system.
 
|}
 
|}
 +
 
====<font color="#200772">Test Results</font>====
 
====<font color="#200772">Test Results</font>====
 
{| style="border: 1px solid darkgray; text-align: center;font-siz:14px; height:50px"
 
{| style="border: 1px solid darkgray; text-align: center;font-siz:14px; height:50px"
 
|scope="row"  width="1000" style="text-align: left; background: #eeeeee"|
 
|scope="row"  width="1000" style="text-align: left; background: #eeeeee"|
  
−
[[Image:UT2SurveyResult1.png|470px]][[Image:UT2SurveyResult2.png|490px]]<br><br>
+
'''Critical issues discovered during User Testing 3'''<br>
−
[[Image:UT2SurveyResult3.png|470px]][[Image:UT2SurveyResult4.png|490px]]<br><br>
+
- Unable to handle conflicting concurrent actions. This happened when admin was trying to remove a faculty from the system while she was perform tasks in the system<br>
−
'''Most common feedback from User Testing 2'''<br>
+
- Email ID mismatch. One of the faculty couldn't login to the system due to multiple different SSO IDs he has for different systems<br>
−
- Admin and Course Coordinator want to allow drag-n-drop when editing a booking<br>
 
−
- Admin and Course Coordinator also want to request for reason when students delete booking<br>
 
 
- Supervisors want their time slots to be pre-populate as unavailable instead of available<br>
 
- Supervisors want their time slots to be pre-populate as unavailable instead of available<br>
−
- Students are not sure if their supervisor/reviewers receive email to approve their booking<br>
+
- Some still complain that number of emails received were too much. The email on adding an optional attendee can be reduced<br>
−
- TA wants to see other TAs' sign on on their availability page as well, so that they can swap with each other if can't make it last min<br>
+
[[Image:importanceofemail.PNG|400px]][[Image:languageofemail.PNG|400px]]<br>
−
'''Click here to see our Response Plan:''' [https://www.dropbox.com/s/w8mxojxaaziyp31/Response%20Plan%20to%20User%20Testing%202.xlsx]<br>
 
−
[[Image:UserTesting2MostCommonFeedback.png|450px]][[Image:UserTesting2MostCommonFeedback_2.png|450px]]<br><br>
 
−
[[Image:UserTesting2MostCommonFeedback_3.png|450px]][[Image:UserTesting2MostCommonFeedback_4.png|450px]]<br><br>
 
−
[[Image:UserTesting2MostCommonFeedback_5.png|450px]][[Image:UserTesting2MostCommonFeedback_6.png|450px]]<br><br>
 
 
|}
 
|}
  
Line 454: Line 751:
 
|scope="row"  width="330" style="text-align: left; background: #eeeeee"|
 
|scope="row"  width="330" style="text-align: left; background: #eeeeee"|
 
:1. '''Test Plan'''
 
:1. '''Test Plan'''
−
:#Test plan: [https://www.dropbox.com/s/e9x0ly6loav73mp/Usability%20Test%202%20Plan.pdf]<br>
+
:#Test plan: [https://www.dropbox.com/s/oqw1gz0l05ikbxu/Usability%20Test%203%20Plan.pdf]<br>
 
:2. '''Scenario Task Lists'''
 
:2. '''Scenario Task Lists'''
−
:#Admin task: [https://www.dropbox.com/s/vgmdjkyxew3fguz/Task%20List%20for%20Admin.pdf]<br>
+
:#Supervisor/Reviewer task: [https://www.dropbox.com/s/nektjwquzloblyp/Task%20List%20for%20Supervisor.pdf]<br>
−
:#Course Coordinator task: [https://www.dropbox.com/s/gxbwylkunaxpoan/Task%20List%20for%20Course%20Coordinator.pdf]<br>
+
:#Student task: [https://www.dropbox.com/s/qxlkc8wmv90c6zr/Task%20List%20for%20Student.pdf]<br>
−
:#Supervisor/Reviewer task: [https://www.dropbox.com/s/rsxg54wa06ef5nj/Task%20List%20for%20Supervisor.pdf]<br>
+
:#TA task: [https://www.dropbox.com/s/21eg48tpyx9x6t7/Task%20List%20for%20TA.pdf]<br>
−
:#TA task: [https://www.dropbox.com/s/4a9vxd850pzr6vx/Task%20List%20for%20TA.pdf] <br>
 
−
:#Student task: [https://www.dropbox.com/s/qro3qll9pvyxz1l/Task%20List%20for%20Student.pdf]<br>
 
 
:3. '''Qualtrics Survey Questions'''<br>
 
:3. '''Qualtrics Survey Questions'''<br>
−
:#Survey Link: [http://smusg.asia.qualtrics.com/SE/?SID=SV_6KmS3VTUpoGHdml]<br>
+
:#Survey Link: [http://smusg.asia.qualtrics.com/SE/?SID=SV_2fOb3G7Mh4ieRZr]<br>
−
:#Survey Raw Result:[https://www.dropbox.com/s/k0th7w46ocpfp8g/User%20Testing%202%20survey%20raw%20result.pdf]<br>
+
:#Survey result: [https://www.dropbox.com/s/fn0ohswd5v4bd42/Survey%20Key%20Feedback.xlsx]<br>
 
:4. '''Participant Informed Consent Form'''<br>
 
:4. '''Participant Informed Consent Form'''<br>
−
:#Consent Form: [https://www.dropbox.com/s/6ca0aaoew2n2zcs/Participant%20Informed%20Constent%20Form.pdf]
+
:#Consent Form: [https://www.dropbox.com/s/imfibqb9moy7vtm/Participant%20Informed%20Constent%20Form.pdf]
 
|}
 
|}

Latest revision as of 03:15, 26 November 2013

Wikiprofilepic.jpg

Home

  About Us   Project Overview   Project Management   Project Documentation
Diagrams Prototype Usability Test Meeting Minutes Downloads

Internal Testing

Internal testing was conducted using regression method at end of each iteration. Refer to our test plan, test case and results below.
Click here for our test plan
Click here for our test cases by iterations

Usability Test 1

Date and Venue

Date: 2 Sep 2013
Time: 1.30pm - 3pm
Venue: SIS MR4.7

Participants

No. of Participants: 10
Participant Profile:
- Admin
- Course Coordinator
- Supervisor/Reviewer
- Students doing FYP next semester

UT1Photo.png
UT1Photo2.png
UT1Photo3.png

Scope of Test

Function Admin Course Coordinator Supervisor/Reviewer Student
Single Sign On Transparent-green-checkmark.png Transparent-green-checkmark.png Transparent-green-checkmark.png Transparent-green-checkmark.png
Role switching - Transparent-green-checkmark.png - -
Create schedule Transparent-green-checkmark.png Transparent-green-checkmark.png - -
View schedule Transparent-green-checkmark.png Transparent-green-checkmark.png Transparent-green-checkmark.png Transparent-green-checkmark.png
Create booking Transparent-green-checkmark.png Transparent-green-checkmark.png - Transparent-green-checkmark.png
View booking Transparent-green-checkmark.png Transparent-green-checkmark.png Transparent-green-checkmark.png Transparent-green-checkmark.png
Edit booking Transparent-green-checkmark.png Transparent-green-checkmark.png - -
Delete booking Transparent-green-checkmark.png Transparent-green-checkmark.png - Transparent-green-checkmark.png
Approve/Reject booking - - Transparent-green-checkmark.png -
View booking history Transparent-green-checkmark.png Transparent-green-checkmark.png Transparent-green-checkmark.png Transparent-green-checkmark.png
Supervisor Availability - - Transparent-green-checkmark.png -
Add default milestones Transparent-green-checkmark.png Transparent-green-checkmark.png - -
Add required attendees for different milestones Transparent-green-checkmark.png Transparent-green-checkmark.png - -
Set presentation duration Transparent-green-checkmark.png Transparent-green-checkmark.png - -

Testing Method

A list of tasks based on different scenarios were given to participants and they need to follow the scenarios and figure out how to complete them in the system. Respective facilitators who sat beside them need to observe and record down how they were using the system to complete the task assigned.
Upon completion of all tasks, participants were given a link to complete a questionnaire with regards to their experiences of using our system and feedback any issues they faced during testing or improvements they would like to see in the future.

Test Results

When asked about what do they like about our system:
- 9 out of 10 said that UI is clean and intuitive
- 1 out of 10 said that new system, better than nothing

What asked what they dislike about the system:
- 2 out of 10 said that they are not sure how to check profile information
- 1 out of 10 said that system is slow
- 1 out of 10 not sure they can check booking history
- 1 out of 10 said that no confirmation when delete booking
- 1 out of 10 said that create schedule for Admin and Course Coordinator is not intuitive
- 1 out of 10 said that abit too many clicks
- 1 out of 10 encountered errors while performing tasks

Some of the issues they faced when performing tasks are:
- 2 out of 10 said that they have problem checking their role in the system
- 1 out of 10 face DB connection error
- 1 out of 10 said that Safari browser unable to click down arrow
- 1 out of 10 said that date and time field for edit booking is hard to use
- 1 out of 10 said that they cant use SSO auto login, need to have manual login
- 1 out of 10 said that email content is not sufficient

Improvements they would like to have to have a better user experience:
- 1 out of 10 wants the system to be faster
- 1 out of 10 want to improve on buttons e.g Booking history
- 1 out of 10 wants email header to be color-coded, easier for user to identify the status of the booking
- 1 out of 10 wants lesser clicks
- '1 out of 10 wants tool-tip/Breadcrumbs

Click here to see our Response Plan: [1]

SurveyResult1.PNG
SurveyResult2.PNG

SurveyResult3.PNG
SurveyResult4.PNG

Most Common Feedback for User Testing 1:
UserTesting1MostCommonFeedback.PNGUserTesting1MostCommonFeedback 2.PNG

UserTesting1MostCommonFeedback 3.PNGUserTesting1MostCommonFeedback 3 fixed.PNG

Documentation

1. Test Plan
  1. Test plan: [2]
2. Scenario Task Lists
  1. Admin task: [3]
  2. Course Coordinator task: [4]
  3. Supervisor/Reviewer task: [5]
  4. Student task: [6]
3. Qualtrics Survey Questions
  1. Survey Link: [7]
  2. Survey Raw Result:[8]
4. Participant Informed Consent Form
  1. Consent Form: [9]

Usability Test 2

Date and Venue

Date: 4 Oct 2013
Time: 3.30pm - 7pm
Venue: SIS Seminar Rm 2-2

Participants

No. of Participants: 21
Participant Profile:
- Admin
- Course Coordinator
- 3 Supervisor/Reviewer
- 2 TA
- 15 Students completed IS480 and students doing IS480 next term
UT2Photo.pngUT1VSUT2participantprofile.png

Scope of Test

Function Admin Course Coordinator Supervisor/Reviewer TA Student
Single Sign On Transparent-green-checkmark.png Transparent-green-checkmark.png Transparent-green-checkmark.png Transparent-green-checkmark.png Transparent-green-checkmark.png
Role switching - Transparent-green-checkmark.png - - -
Create schedule Transparent-green-checkmark.png Transparent-green-checkmark.png - - -
View schedule Transparent-green-checkmark.png Transparent-green-checkmark.png Transparent-green-checkmark.png Transparent-green-checkmark.png Transparent-green-checkmark.png
Edit schedule Transparent-green-checkmark.png Transparent-green-checkmark.png - - -
Archive schedule Transparent-green-checkmark.png Transparent-green-checkmark.png - - -
Create booking Transparent-green-checkmark.png Transparent-green-checkmark.png - - Transparent-green-checkmark.png
Add optional attendees Transparent-green-checkmark.png Transparent-green-checkmark.png - - Transparent-green-checkmark.png
View booking Transparent-green-checkmark.png Transparent-green-checkmark.png Transparent-green-checkmark.png Transparent-green-checkmark.png Transparent-green-checkmark.png
Edit booking Transparent-green-checkmark.png Transparent-green-checkmark.png - - -
Delete booking Transparent-green-checkmark.png Transparent-green-checkmark.png - - Transparent-green-checkmark.png
Approve/Reject booking - - Transparent-green-checkmark.png - -
Reject booking with reason - - Transparent-green-checkmark.png - -
View my bookings Transparent-green-checkmark.png Transparent-green-checkmark.png Transparent-green-checkmark.png Transparent-green-checkmark.png Transparent-green-checkmark.png
Supervisor Availability - - Transparent-green-checkmark.png - -
TA sign up for filming - - - Transparent-green-checkmark.png -
Add default milestones Transparent-green-checkmark.png Transparent-green-checkmark.png - - -
Add required attendees for different milestones Transparent-green-checkmark.png Transparent-green-checkmark.png - - -
Set presentation duration Transparent-green-checkmark.png Transparent-green-checkmark.png - - -


User Testing 2 has been expanded from User Testing 1 to include 5 more features, which are highlighted in yellow.

Testing Method

A list of tasks based on different scenarios were given to participants and they need to follow the scenarios and figure out how to complete them in the system. Respective facilitators who sat beside them need to observe and record down how they were using the system to complete the task assigned.
Upon completion of all tasks, participants were given a link to complete a questionnaire with regards to their experiences of using our system and feedback any issues they faced during testing or improvements they would like to see in the future.

Test Results

Comparison of User Testing 1 survey result and User Testing 2 survey result. Overall, MORE users agree with our system!

UT2SurveyResult1.pngUT2SurveyResult2.png

UT2SurveyResult3.pngUT2SurveyResult4.png

Most common feedback from User Testing 2
- Admin and Course Coordinator want to allow drag-n-drop when editing a booking
- Admin and Course Coordinator also want to request for reason when students delete booking
- Supervisors want their time slots to be pre-populate as unavailable instead of available
- Students are not sure if their supervisor/reviewers receive email to approve their booking
- TA wants to see other TAs' sign on on their availability page as well, so that they can swap with each other if can't make it last min
Click here to see our Response Plan: [10]
UserTesting2MostCommonFeedback.pngUserTesting2MostCommonFeedback 2.png

UserTesting2MostCommonFeedback 3.pngUserTesting2MostCommonFeedback 4.png

UserTesting2MostCommonFeedback 5.pngUserTesting2MostCommonFeedback 6.png

Documentation

1. Test Plan
  1. Test plan: [11]
2. Scenario Task Lists
  1. Admin task: [12]
  2. Course Coordinator task: [13]
  3. Supervisor/Reviewer task: [14]
  4. TA task: [15]
  5. Student task: [16]
3. Qualtrics Survey Questions
  1. Survey Link: [17]
  2. Survey Raw Result:[18]
4. Participant Informed Consent Form
  1. Consent Form: [19]

Usability Test 3

Date and Venue

Date: 18 Nov 2013
Time: 2pm - 7pm
Venue: SIS MR4.3

Participants

No. of Participants: 25
Participant Profile:
- Admin
- 4 Supervisor/Reviewer
- 3 TA
- 17 Students
UT1-3ParticipantProfile.png

Scope of Test

Function Admin Course Coordinator Supervisor/Reviewer TA Student
Single Sign On Transparent-green-checkmark.png Transparent-green-checkmark.png Transparent-green-checkmark.png Transparent-green-checkmark.png Transparent-green-checkmark.png
Role switching - Transparent-green-checkmark.png - - -
Create schedule Transparent-green-checkmark.png Transparent-green-checkmark.png - - -
View schedule Transparent-green-checkmark.png Transparent-green-checkmark.png Transparent-green-checkmark.png Transparent-green-checkmark.png Transparent-green-checkmark.png
Edit schedule Transparent-green-checkmark.png Transparent-green-checkmark.png - - -
Archive schedule Transparent-green-checkmark.png Transparent-green-checkmark.png - - -
Create booking Transparent-green-checkmark.png Transparent-green-checkmark.png - - Transparent-green-checkmark.png
Add optional attendees Transparent-green-checkmark.png Transparent-green-checkmark.png - - Transparent-green-checkmark.png
View booking Transparent-green-checkmark.png Transparent-green-checkmark.png Transparent-green-checkmark.png Transparent-green-checkmark.png Transparent-green-checkmark.png
Edit booking Transparent-green-checkmark.png Transparent-green-checkmark.png - - -
Delete booking Transparent-green-checkmark.png Transparent-green-checkmark.png - - Transparent-green-checkmark.png
Approve/Reject booking - - Transparent-green-checkmark.png - -
Reject booking with reason - - Transparent-green-checkmark.png - -
Cancel booking - - Transparent-green-checkmark.png - -
View my bookings Transparent-green-checkmark.png Transparent-green-checkmark.png Transparent-green-checkmark.png Transparent-green-checkmark.png Transparent-green-checkmark.png
Presentation subscription - - Transparent-green-checkmark.png Transparent-green-checkmark.png Transparent-green-checkmark.png
Supervisor Availability - - Transparent-green-checkmark.png - -
TA sign up for filming - - - Transparent-green-checkmark.png -
TA swap slots - - - Transparent-green-checkmark.png -
Add default milestones Transparent-green-checkmark.png Transparent-green-checkmark.png - - -
Add required attendees for different milestones Transparent-green-checkmark.png Transparent-green-checkmark.png - - -
Set presentation duration Transparent-green-checkmark.png Transparent-green-checkmark.png - - -
CSV file upload Transparent-green-checkmark.png Transparent-green-checkmark.png - - -
Manage users Transparent-green-checkmark.png Transparent-green-checkmark.png - - -
Generate reports Transparent-green-checkmark.png Transparent-green-checkmark.png - - -
Admin/Faculty Dashboard Transparent-green-checkmark.png Transparent-green-checkmark.png Transparent-green-checkmark.png - -
TA sign up dashboard Transparent-green-checkmark.png - - - -


User Testing 3 has been expanded from User Testing 2 to include 8 more features, which are highlighted in yellow.

Testing Method

A list of tasks based on different scenarios were given to participants and they need to follow the scenarios and figure out how to complete them in the system. Respective facilitators who sat beside them need to observe and record down how they were using the system to complete the task assigned.
Upon completion of all tasks, participants were given a link to complete a questionnaire with regards to the quality of emails they received from the system.

Test Results

Critical issues discovered during User Testing 3
- Unable to handle conflicting concurrent actions. This happened when admin was trying to remove a faculty from the system while she was perform tasks in the system
- Email ID mismatch. One of the faculty couldn't login to the system due to multiple different SSO IDs he has for different systems
- Supervisors want their time slots to be pre-populate as unavailable instead of available
- Some still complain that number of emails received were too much. The email on adding an optional attendee can be reduced
Importanceofemail.PNGLanguageofemail.PNG

Documentation

1. Test Plan
  1. Test plan: [20]
2. Scenario Task Lists
  1. Supervisor/Reviewer task: [21]
  2. Student task: [22]
  3. TA task: [23]
3. Qualtrics Survey Questions
  1. Survey Link: [24]
  2. Survey result: [25]
4. Participant Informed Consent Form
  1. Consent Form: [26]
Retrieved from ‘https://wiki.smu.edu.sg/is480/index.php?title=IS480_Team_wiki:_2013T1_ThunderBolt_Project_Documentation_Usability_Test&oldid=172142’