Objectives
1. Determine that the usage of the features in the application are consistent to the expectations of the users (acting as of Casting Managers & Artistes)
2. Obtain feedback from users such as to improve the usability (learnability, efficiency, errors, satisfaction) and aesthetics of our application
|
Scope of Test
Usability Test 1: A basic set up of the test at the office of Oak3 Films
S/N
|
Features
|
Artiste
|
Prod. House
|
1
|
Registration / Log in / Log out
|
|
|
2
|
Change / Reset password
|
|
|
3
|
View / Edit Artiste Portfolio
|
|
-
|
4
|
Search for casting calls / View suggested casting calls / Favourite a casting call
|
|
-
|
5
|
Apply for a casting call / View all applications
|
|
-
|
6
|
View / Edit Production Portfolio
|
-
|
|
7
|
Search for artistes / View suggested artistes / Favourite an artiste
|
-
|
|
8
|
Create / Edit a Casting Call / View all applicants of a Casting Call
|
-
|
|
9
|
Invite an artiste to a Casting Call
|
-
|
|
Session 1 (Casting Managers)
Session 1 was successfully conducted on 7 Sept 2012, 2 – 6pm at the office of Oak3 Films.
A total of 7 staff members from Oak3 Films assumed the role of a Casting Manager participated:
#Dennis (Head of Production)
- Jonathan (Producer)
- Jeremy (Production Assistant)
- PeiHua (Production Assistant)
- Jazz (Graphics)
- Dinesh (IT Executive)
- Ana (Secretary)
Feedback from Session 1 of the Usability Test is recorded in the User Feedback Document
View also our Response Plan to Feedback (Casting Mgrs).
Most common feedback:
Solution: Display all tooltips by default to users on the first time they create casting calls
|
Solution: Automatically scroll to bottom of page when date input is selected
|
Solution: Bring users directly to "edit portfolio" page the first time they sign up
|
Results from Casing Managers End-of-Test Qualtrics Survey (Extract):
Chart 1: 2 candidates expressed possibility that the system is unnecessarily complex
|
Watch how we conducted Session 1 HERE
|
Session 1 Conclusion
Inferring from the above table, we identified that some functions of the application may not be sufficiently intuitive (E.g. Items #2, 3 and 4). Perhaps this explains why some testers felt that the system is slightly complex (as shown in Chart 1).
One of the solutions to get around this is to introduce some simple tooltips to allow the user to get familiarized in using the system. We have also implemented some changes to improve the overall usability and intuitiveness of the application. View our Response Plan to Feedback (Casting Mgrs).
Nevertheless, many users expressed that on the whole, the system is easy to use, streamlined, is “time-saving” and user-friendly.
|
Session 2 (Artistes)
Session 2 was conducted from 9 - 14 Sept 2012
A total of 18 testers who assumed the role as Artistes participated.
Feedback from Session 2 of the Usability Test: Artiste User Feedback
View also our Response Plan to Feedback (Artistes).
Most common feedback:
Solution: Bring user to edit page on the first visit to the account and change all edit buttons to blue colour
|
Solution: Add label to each information row
|
Solution: Use 3 conventional drop down number selection
|
Solution: Change the layout to look more like a form
|
Results from Artiste End-of-Test Qualtrics Survey (Extract):
Chart 3: around 8 candidates felt that the system is unnecessarily complex
|
Chart 4: 6 candidates believed that they need assistance to use this system
|
Session 2 Conclusion
Session 2 provided us with much insights especially to the overall intuitiveness of using the system. The top few feedback involves how candidates had difficulties in navigating and using the system; and we have since made changes to the feedback given.
Charts 3 and 4 further support this issue, where many candidates believe that the system is either slightly complex or they would need the help of a person to fully exploit the system.
View our Response Plan to Feedback (Artistes)
|
Testing Methodology
a. Collecting qualitative metrics
- Users of the test is categorised into two groups:
- Peers/Family members assuming the Artiste role (Artiste Task List) and
- Staff members of Oak3 Films assuming the Casting Manager role (Casting Managers Task List)
- Users are encouraged to think out loud while accomplishing each task so as to better allow the facilitators to record down accurate observations and significant details of the testers’ thought processes. Facilitators will also be observing for errors (e.g. false clicks) and navigation issues. Upon completion of all the tasks, the tester will be presented a questionnaire (Artistes/CastingMgrs) in which he will be asked what he dislikes / likes about the application and their satisfaction level of the application.
- The testing process will also be recorded via video using Screenflow and will complement the facilitators’ observations. Facilitators are to record the timing in which a note-worthy observation (e.g. body language response) occurred.
b. Collecting quantitative metrics
- Quantitative metrics to be measured include:
- The amount of time to complete each task
- The number of clicks taken for completing each task (and of which how many are false clicks)
|
Supporting Documents UT1
|