HeaderSIS.jpg

IS480 Team wiki: 2012T1 Timber Werkz UsabilityTest

From IS480
Jump to navigation Jump to search
HOME   PROJECT OVERVIEW     PROJECT MANAGEMENT   DOCUMENTATION  
         


<< MID-TERM WIKI          << FINAL WIKI


Usability Test 1

Objectives

1. Determine that the usage of the features in the application are consistent to the expectations of the users (acting as of Casting Managers & Artistes)

2. Obtain feedback from users such as to improve the usability (learnability, efficiency, errors, satisfaction) and aesthetics of our application



Scope of Test

Usability Test 1: A basic set up of the test at the office of Oak3 Films
S/N Features Artiste Prod. House
1 Registration / Log in / Log out Greentick.png Greentick.png
2 Change / Reset password Greentick.png Greentick.png
3 View / Edit Artiste Portfolio Greentick.png -
4 Search for casting calls / View suggested casting calls / Favourite a casting call Greentick.png -
5 Apply for a casting call / View all applications Greentick.png -
6 View / Edit Production Portfolio - Greentick.png
7 Search for artistes / View suggested artistes / Favourite an artiste - Greentick.png
8 Create / Edit a Casting Call / View all applicants of a Casting Call - Greentick.png
9 Invite an artiste to a Casting Call - Greentick.png

Session 1 (Casting Managers)


Session 1 was successfully conducted on 7 Sept 2012, 2 – 6pm at the office of Oak3 Films.
A total of 7 staff members from Oak3 Films assumed the role of a Casting Manager participated: #Dennis (Head of Production)

  1. Jonathan (Producer)
  2. Jeremy (Production Assistant)
  3. PeiHua (Production Assistant)
  4. Jazz (Graphics)
  5. Dinesh (IT Executive)
  6. Ana (Secretary)



Feedback from Session 1 of the Usability Test is recorded in the User Feedback Document
View also our Response Plan to Feedback (Casting Mgrs).

Most common feedback:

Solution: Display all tooltips by default to users on the first time they create casting calls
Solution: Automatically scroll to bottom of page when date input is selected
Solution: Bring users directly to "edit portfolio" page the first time they sign up


Results from Casing Managers End-of-Test Qualtrics Survey (Extract):

Chart 1: 2 candidates expressed possibility that the system is unnecessarily complex
Watch how we conducted Session 1 HERE

Session 1 Conclusion

Inferring from the above table, we identified that some functions of the application may not be sufficiently intuitive (E.g. Items #2, 3 and 4). Perhaps this explains why some testers felt that the system is slightly complex (as shown in Chart 1).

One of the solutions to get around this is to introduce some simple tooltips to allow the user to get familiarized in using the system. We have also implemented some changes to improve the overall usability and intuitiveness of the application. View our Response Plan to Feedback (Casting Mgrs).

Nevertheless, many users expressed that on the whole, the system is easy to use, streamlined, is “time-saving” and user-friendly.




Session 2 (Artistes)

Session 2 was conducted from 9 - 14 Sept 2012
A total of 18 testers who assumed the role as Artistes participated.

Feedback from Session 2 of the Usability Test: Artiste User Feedback
View also our Response Plan to Feedback (Artistes).

Most common feedback:

Solution: Bring user to edit page on the first visit to the account and change all edit buttons to blue colour
Solution: Add label to each information row
Solution: Use 3 conventional drop down number selection
Solution: Change the layout to look more like a form


Results from Artiste End-of-Test Qualtrics Survey (Extract):

Chart 3: around 8 candidates felt that the system is unnecessarily complex
Chart 4: 6 candidates believed that they need assistance to use this system

Session 2 Conclusion

Session 2 provided us with much insights especially to the overall intuitiveness of using the system. The top few feedback involves how candidates had difficulties in navigating and using the system; and we have since made changes to the feedback given. Charts 3 and 4 further support this issue, where many candidates believe that the system is either slightly complex or they would need the help of a person to fully exploit the system.
View our Response Plan to Feedback (Artistes)



Testing Methodology

a. Collecting qualitative metrics

Users of the test is categorised into two groups:
  1. Peers/Family members assuming the Artiste role (Artiste Task List) and
  2. Staff members of Oak3 Films assuming the Casting Manager role (Casting Managers Task List)
Users are encouraged to think out loud while accomplishing each task so as to better allow the facilitators to record down accurate observations and significant details of the testers’ thought processes. Facilitators will also be observing for errors (e.g. false clicks) and navigation issues. Upon completion of all the tasks, the tester will be presented a questionnaire (Artistes/CastingMgrs) in which he will be asked what he dislikes / likes about the application and their satisfaction level of the application.
The testing process will also be recorded via video using Screenflow and will complement the facilitators’ observations. Facilitators are to record the timing in which a note-worthy observation (e.g. body language response) occurred.
1346957489 Schedule File.png


b. Collecting quantitative metrics

Quantitative metrics to be measured include:
  • The amount of time to complete each task
  • The number of clicks taken for completing each task (and of which how many are false clicks)


Supporting Documents UT1

Category Links
Pre-Testing
  1. Usability Test Plan v3
During-Testing
1. Scenario Task Lists
a. For Artistes
b. For Casting Managers
c. Data Sheet for Casting Managers
2. Qualtrics Survey Questions
a. For Artistes
b. For Casting Managers
3. Facilitator Script
4. User Consent Form
Post-Testing
1. User Feedback Results
a. from Artistes
b. from Casting Managers
2. Qualtrics Report
a. Artistes
b. Casting Managers
3. Proposed Solutions to Feedback
a. Response Plan to Feedback (Artistes)
b. Response Plan to Feedback (Casting Managers)



Usability Test 2

Objectives

1. Determine that the usage of the features in the application are consistent to the expectations of real users (Casting Managers & Artistes)

2. Obtain feedback from users such as to improve the usability (learnability, efficiency, errors, satisfaction) and aesthetics of our application


Scope of Test

The scope of the test has been expanded from Usability Test 1 to include 6 more features:

S/N Features Artiste Prod. House Casting Manager
1 Registration / Log in / Log out Greentick.png Greentick.png Greentick.png
2 Change / Reset password Greentick.png Greentick.png Greentick.png
3 View / Edit Artiste Portfolio Greentick.png - -
4 Search for casting calls / View suggested casting calls / Favourite a casting call Greentick.png - -
5 Apply for a casting call / View all applications Greentick.png - -
6 Accept/ reject an invitation Greentick.png - -
7 Select an audition/ View audition date Greentick.png - -
8 View / Edit Production Portfolio - Greentick.png -
9 Invite Casting Manager - Greentick.png -
10 View / Edit Casting Manager Portfolio - - Greentick.png
11 Search for artistes / View suggested artistes / Favourite an artiste - - Greentick.png
12 Create / Edit a Casting Call / View all applicants of a Casting Call - - Greentick.png
13 Invite an artiste to a Casting Call - - Greentick.png
14 Create / Edit / View Auditions - - Greentick.png
15 Manage Account Greentick.png - Greentick.png


Plan for Usability Test 2


initially, we planned to have both a lab test and field test for Usability Test 2. The lab test will be a more controlled one, whereby we could sit beside the users and observe their usage of our system. The field test, however, will be carried out at the actual environment (e.g. artistes at their homes and casting managers at their offices) to determine if they are capable of using the system at their own comfort without our guidance and interference.

Also, we wanted to carry out a pilot test before the actual test to verify if our planned test goes according to our expectations, and to make any necessary changes before the actual test.

The following chart shows our original plan for Usability Test 2:

TW UT2Actual.png

Actual test and results

However, as there wasn't any ongoing casting calls at the point where we carried out Usability Test 2, we decided to proceed with only a lab test and leave the field test for later. With Oak3's help and support, we gathered artistes with the following profiles:

TW UT2 participants.png
TW UT2 combined.jpg


Testing Methodology

Collecting qualitative metrics (Lab Test)

Users are encouraged to think out loud while accomplishing each task so as to better allow the facilitators to record down accurate observations and significant details of the testers’ thought processes. Facilitators will also be observing for errors (e.g. false clicks) and navigation issues. Upon completion of all the tasks, the tester will be presented a questionnaire in which he will be asked what he dislikes / likes about the application and their satisfaction level of the application.
1351780541 27-Edit Text.png


The testing process will also be recorded via video using Screenflow and will complement the facilitators’ observations. Facilitators are to record the timing in which a note-worthy observation (e.g. body language response) occurred.

Supporting Documents UT2

Category Links
Test Documents
1. UT2 Plan
2. Artiste Observed Feedback and Response Plan
3. Artiste Survey Link
4. Artiste Survey Results Report
5. Casting Manager Observed Feedback and Response Plan
6. Casting Manager Task List
7. Artiste Task List


Extract of Solutions to Most Common Feedback

UT21.png


TW UT22.png



Field Test

Objectives

1. Determine if users are capable of handling the application on their own (without any guidance from us) in the actual environment where they would be using the application

2. Obtain feedback from users such as to improve the usability (learnability, efficiency, errors, satisfaction) and aesthetics of our application



Field Test Details & Outcome


Field Test was conducted from 28 November 2012 to 1 December 2012 remotely at the artistes' own comfort. With the help and support of Oak3, we managed to get some artistes to sign up as a user of Casting3, and to apply for an ongoing casting call, "Koffee did it". They will be provided with a link to an evaluation form, where they will asked to give us their feedback.

TW FieldTestMethod.png

As of the submission date of this wiki page (28 Nov), the field test is ongoing and results have yet to be collated. The results will be later concluded in the following links:

Feedback Links:

1. Profile of artistes who have signed up
2. Survey for Artistes and their feedback
3. Response Plan to Artiste Feedback