Internal Testing
Internal testing was conducted using regression method at end of each iteration. Refer to our test plan, test case and results below.
Click here for our test plan
Click here for our test cases by iterations
Usability Test 1
Date and Venue
Date: 2 Sep 2013
Time: 1.30pm - 3pm
Venue: SIS MR4.7
|
Participants
No. of Participants: 10
Participant Profile:
- Admin
- Course Coordinator
- Supervisor/Reviewer
- Students doing FYP next semester
|
Scope of Test
Function
|
Admin
|
Course Coordinator
|
Supervisor/Reviewer
|
Student
|
Single Sign On
|
|
|
|
|
Role switching
|
-
|
|
-
|
-
|
Create schedule
|
|
|
-
|
-
|
View schedule
|
|
|
|
|
Create booking
|
|
|
-
|
|
View booking
|
|
|
|
|
Edit booking
|
|
|
-
|
-
|
Delete booking
|
|
|
-
|
|
Approve/Reject booking
|
-
|
-
|
|
-
|
View booking history
|
|
|
|
|
Supervisor Availability
|
-
|
-
|
|
-
|
Add default milestones
|
|
|
-
|
-
|
Add required attendees for different milestones
|
|
|
-
|
-
|
Set presentation duration
|
|
|
-
|
-
|
Testing Method
A list of tasks based on different scenarios were given to participants and they need to follow the scenarios and figure out how to complete them in the system.
Respective facilitators who sat beside them need to observe and record down how they were using the system to complete the task assigned.
Upon completion of all tasks, participants were given a link to complete a questionnaire with regards to their experiences of using our system and feedback any issues they faced during testing or improvements they would like to see in the future.
|
Test Results
When asked about what do they like about our system:
- 9 out of 10 said that UI is clean and intuitive
- 1 out of 10 said that new system, better than nothing
What asked what they dislike about the system:
- 2 out of 10 said that they are not sure how to check profile information
- 1 out of 10 said that system is slow
- 1 out of 10 not sure they can check booking history
- 1 out of 10 said that no confirmation when delete booking
- 1 out of 10 said that create schedule for Admin and Course Coordinator is not intuitive
- 1 out of 10 said that abit too many clicks
- 1 out of 10 encountered errors while performing tasks
Some of the issues they faced when performing tasks are:
- 2 out of 10 said that they have problem checking their role in the system
- 1 out of 10 face DB connection error
- 1 out of 10 said that Safari browser unable to click down arrow
- 1 out of 10 said that date and time field for edit booking is hard to use
- 1 out of 10 said that they cant use SSO auto login, need to have manual login
- 1 out of 10 said that email content is not sufficient
Improvements they would like to have to have a better user experience:
- 1 out of 10 wants the system to be faster
- 1 out of 10 want to improve on buttons e.g Booking history
- 1 out of 10 wants email header to be color-coded, easier for user to identify the status of the booking
- 1 out of 10 wants lesser clicks
- '1 out of 10 wants tool-tip/Breadcrumbs
Click here to see our Response Plan: [1]
Most Common Feedback for User Testing 1:
|
Documentation
- 1. Test Plan
- Test plan: [2]
- 2. Scenario Task Lists
- Admin task: [3]
- Course Coordinator task: [4]
- Supervisor/Reviewer task: [5]
- Student task: [6]
- 3. Qualtrics Survey Questions
- Survey Link: [7]
- Survey Raw Result:[8]
- 4. Participant Informed Consent Form
- Consent Form: [9]
|
Usability Test 2
Date and Venue
Date: 4 Oct 2013
Time: 3.30pm - 7pm
Venue: SIS Seminar Rm 2-2
|
Participants
No. of Participants: 21
Participant Profile:
- Admin
- Course Coordinator
- 3 Supervisor/Reviewer
- 2 TA
- 15 Students completed IS480 and students doing IS480 next term
|
Scope of Test
Function
|
Admin
|
Course Coordinator
|
Supervisor/Reviewer
|
TA
|
Student
|
Single Sign On
|
|
|
|
|
|
Role switching
|
-
|
|
-
|
-
|
-
|
Create schedule
|
|
|
-
|
-
|
-
|
View schedule
|
|
|
|
|
|
Edit schedule
|
|
|
-
|
-
|
-
|
Archive schedule
|
|
|
-
|
-
|
-
|
Create booking
|
|
|
-
|
-
|
|
Add optional attendees
|
|
|
-
|
-
|
|
View booking
|
|
|
|
|
|
Edit booking
|
|
|
-
|
-
|
-
|
Delete booking
|
|
|
-
|
-
|
|
Approve/Reject booking
|
-
|
-
|
|
-
|
-
|
Reject booking with reason
|
-
|
-
|
|
-
|
-
|
View my bookings
|
|
|
|
|
|
Supervisor Availability
|
-
|
-
|
|
-
|
-
|
TA sign up for filming
|
-
|
-
|
-
|
|
-
|
Add default milestones
|
|
|
-
|
-
|
-
|
Add required attendees for different milestones
|
|
|
-
|
-
|
-
|
Set presentation duration
|
|
|
-
|
-
|
-
|
User Testing 2 has been expanded from User Testing 1 to include 5 more features, which are highlighted in yellow.
Testing Method
A list of tasks based on different scenarios were given to participants and they need to follow the scenarios and figure out how to complete them in the system.
Respective facilitators who sat beside them need to observe and record down how they were using the system to complete the task assigned.
Upon completion of all tasks, participants were given a link to complete a questionnaire with regards to their experiences of using our system and feedback any issues they faced during testing or improvements they would like to see in the future.
|
Test Results
Comparison of User Testing 1 survey result and User Testing 2 survey result. Overall, MORE users agree with our system!
Documentation
- 1. Test Plan
- Test plan: [11]
- 2. Scenario Task Lists
- Admin task: [12]
- Course Coordinator task: [13]
- Supervisor/Reviewer task: [14]
- TA task: [15]
- Student task: [16]
- 3. Qualtrics Survey Questions
- Survey Link: [17]
- Survey Raw Result:[18]
- 4. Participant Informed Consent Form
- Consent Form: [19]
|
Usability Test 3
Date and Venue
Date: 18 Nov 2013
Time: 2pm - 7pm
Venue: SIS MR4.3
|
Participants
No. of Participants: 25
Participant Profile:
- Admin
- 4 Supervisor/Reviewer
- 3 TA
- 17 Students
|
Scope of Test
Function
|
Admin
|
Course Coordinator
|
Supervisor/Reviewer
|
TA
|
Student
|
Single Sign On
|
|
|
|
|
|
Role switching
|
-
|
|
-
|
-
|
-
|
Create schedule
|
|
|
-
|
-
|
-
|
View schedule
|
|
|
|
|
|
Edit schedule
|
|
|
-
|
-
|
-
|
Archive schedule
|
|
|
-
|
-
|
-
|
Create booking
|
|
|
-
|
-
|
|
Add optional attendees
|
|
|
-
|
-
|
|
View booking
|
|
|
|
|
|
Edit booking
|
|
|
-
|
-
|
-
|
Delete booking
|
|
|
-
|
-
|
|
Approve/Reject booking
|
-
|
-
|
|
-
|
-
|
Reject booking with reason
|
-
|
-
|
|
-
|
-
|
Cancel booking
|
-
|
-
|
|
-
|
-
|
View my bookings
|
|
|
|
|
|
Presentation subscription
|
-
|
-
|
|
|
|
Supervisor Availability
|
-
|
-
|
|
-
|
-
|
TA sign up for filming
|
-
|
-
|
-
|
|
-
|
TA swap slots
|
-
|
-
|
-
|
|
-
|
Add default milestones
|
|
|
-
|
-
|
-
|
Add required attendees for different milestones
|
|
|
-
|
-
|
-
|
Set presentation duration
|
|
|
-
|
-
|
-
|
CSV file upload
|
|
|
-
|
-
|
-
|
Manage users
|
|
|
-
|
-
|
-
|
Generate reports
|
|
|
-
|
-
|
-
|
Admin/Faculty Dashboard
|
|
|
|
-
|
-
|
TA sign up dashboard
|
|
-
|
-
|
-
|
-
|
User Testing 3 has been expanded from User Testing 2 to include 8 more features, which are highlighted in yellow.
Testing Method
A list of tasks based on different scenarios were given to participants and they need to follow the scenarios and figure out how to complete them in the system.
Respective facilitators who sat beside them need to observe and record down how they were using the system to complete the task assigned.
Upon completion of all tasks, participants were given a link to complete a questionnaire with regards to the quality of emails they received from the system.
|
Test Results
Critical issues discovered during User Testing 3
- Unable to handle conflicting concurrent actions. This happened when admin was trying to remove a faculty from the system while she was perform tasks in the system
- Email ID mismatch. One of the faculty couldn't login to the system due to multiple different SSO IDs he has for different systems
- Supervisors want their time slots to be pre-populate as unavailable instead of available
- Some still complain that number of emails received were too much. The email on adding an optional attendee can be reduced
|
Documentation
- 1. Test Plan
- Test plan: [20]
- 2. Scenario Task Lists
- Supervisor/Reviewer task: [21]
- Student task: [22]
- TA task: [23]
- 3. Qualtrics Survey Questions
- Survey Link: [24]
- Survey result: [25]
- 4. Participant Informed Consent Form
- Consent Form: [26]
|