HeaderSIS.jpg

IS480 Team wiki: 2012T1 Timber Werkz UsabilityTest

From IS480
Jump to navigation Jump to search
HOME   PROJECT OVERVIEW     PROJECT MANAGEMENT   DOCUMENTATION  
         


<< MID-TERM WIKI


Usability Test 1

Objectives

1. Determine that the usage of the features in the application are consistent to the expectations of the users (acting as of Casting Managers & Artistes)

2. Obtain feedback from users such as to improve the usability (learnability, efficiency, errors, satisfaction) and aesthetics of our application



Scope of Test

Usability Test 1: A basic set up of the test at the office of Oak3 Films
S/N Features Artiste Prod. House
1 Registration / Log in / Log out Greentick.png Greentick.png
2 Change / Reset password Greentick.png Greentick.png
3 View / Edit Artiste Portfolio Greentick.png -
4 Search for casting calls / View suggested casting calls / Favourite a casting call Greentick.png -
5 Apply for a casting call / View all applications Greentick.png -
6 View / Edit Production Portfolio - Greentick.png
7 Search for artistes / View suggested artistes / Favourite an artiste - Greentick.png
8 Create / Edit a Casting Call / View all applicants of a Casting Call - Greentick.png
9 Invite an artiste to a Casting Call - Greentick.png


Session 1 (Casting Managers)


Session 1 was successfully conducted on 7 Sept 2012, 2 – 6pm at the office of Oak3 Films.
A total of 7 staff members from Oak3 Films assumed the role of a Casting Manager participated: #Dennis (Head of Production)

  1. Jonathan (Producer)
  2. Jeremy (Production Assistant)
  3. PeiHua (Production Assistant)
  4. Jazz (Graphics)
  5. Dinesh (IT Executive)
  6. Ana (Secretary)



Feedback from Session 1 of the Usability Test is recorded in the User Feedback Document
View also our Response Plan to Feedback (Casting Mgrs).

Most common feedback:

Solution: Display all tooltips by default to users on the first time they create casting calls
Solution: Automatically scroll to bottom of page when date input is selected
Solution: Bring users directly to "edit portfolio" page the first time they sign up


Results from Casing Managers End-of-Test Qualtrics Survey (Extract):

Chart 1: 2 candidates expressed possibility that the system is unnecessarily complex
Watch how we conducted Session 1 HERE

Session 1 Conclusion

Inferring from the above table, we identified that some functions of the application may not be sufficiently intuitive (E.g. Items #2, 3 and 4). Perhaps this explains why some testers felt that the system is slightly complex (as shown in Chart 1).

One of the solutions to get around this is to introduce some simple tooltips to allow the user to get familiarized in using the system. We have also implemented some changes to improve the overall usability and intuitiveness of the application. View our Response Plan to Feedback (Casting Mgrs).

Nevertheless, many users expressed that on the whole, the system is easy to use, streamlined, is “time-saving” and user-friendly.




Session 2 (Artistes)

Session 2 was conducted from 9 - 14 Sept 2012
A total of 18 testers who assumed the role as Artistes participated.

Feedback from Session 2 of the Usability Test: Artiste User Feedback
View also our Response Plan to Feedback (Artistes).

Most common feedback:

Solution: Bring user to edit page on the first visit to the account and change all edit buttons to blue colour
Solution: Add label to each information row
Solution: Use 3 conventional drop down number selection
Solution: Change the layout to look more like a form


Results from Artiste End-of-Test Qualtrics Survey (Extract):

Chart 3: around 8 candidates felt that the system is unnecessarily complex
Chart 4: 6 candidates believed that they need assistance to use this system

Session 2 Conclusion

Session 2 provided us with much insights especially to the overall intuitiveness of using the system. The top few feedback involves how candidates had difficulties in navigating and using the system; and we have since made changes to the feedback given. Charts 3 and 4 further support this issue, where many candidates believe that the system is either slightly complex or they would need the help of a person to fully exploit the system.
View our Response Plan to Feedback (Artistes)



Testing Methodology

a. Collecting qualitative metrics

Users of the test is categorised into two groups:
  1. Peers/Family members assuming the Artiste role (Artiste Task List) and
  2. Staff members of Oak3 Films assuming the Casting Manager role (Casting Managers Task List)
Users are encouraged to think out loud while accomplishing each task so as to better allow the facilitators to record down accurate observations and significant details of the testers’ thought processes. Facilitators will also be observing for errors (e.g. false clicks) and navigation issues. Upon completion of all the tasks, the tester will be presented a questionnaire (Artistes/CastingMgrs) in which he will be asked what he dislikes / likes about the application and their satisfaction level of the application.
The testing process will also be recorded via video using Screenflow and will complement the facilitators’ observations. Facilitators are to record the timing in which a note-worthy observation (e.g. body language response) occurred.
1346957489 Schedule File.png


b. Collecting quantitative metrics

Quantitative metrics to be measured include:
  • The amount of time to complete each task
  • The number of clicks taken for completing each task (and of which how many are false clicks)


Supporting Documents UT1

Category Links
Pre-Testing
  1. Usability Test Plan v3
During-Testing
1. Scenario Task Lists
a. For Artistes
b. For Casting Managers
c. Data Sheet for Casting Managers
2. Qualtrics Survey Questions
a. For Artistes
b. For Casting Managers
3. Facilitator Script
4. User Consent Form
Post-Testing
1. User Feedback Results
a. from Artistes
b. from Casting Managers
2. Qualtrics Report
a. Artistes
b. Casting Managers
3. Proposed Solutions to Feedback
a. Response Plan to Feedback (Artistes)
b. Response Plan to Feedback (Casting Managers)



Usability Test 2


Supporting Documents UT2

Category Links
Test Documents
1. UT2 Plan
2. Artiste Observed Feedback
3. Artiste Survey Template
4. Artiste Survey Actual
5. Artiste Survey Results Report