HOME
|
|
PROJECT OVERVIEW
|
|
|
PROJECT MANAGEMENT
|
|
DOCUMENTATION
|
|
<< MID-TERM WIKI << FINAL WIKI
Usability Test 1
Objectives
1. Determine that the usage of the features in the application are consistent to the expectations of the users (acting as of Casting Managers & Artistes)
2. Obtain feedback from users such as to improve the usability (learnability, efficiency, errors, satisfaction) and aesthetics of our application
|
Scope of Test
Usability Test 1: A basic set up of the test at the office of Oak3 Films
S/N
|
Features
|
Artiste
|
Prod. House
|
1
|
Registration / Log in / Log out
|
|
|
2
|
Change / Reset password
|
|
|
3
|
View / Edit Artiste Portfolio
|
|
-
|
4
|
Search for casting calls / View suggested casting calls / Favourite a casting call
|
|
-
|
5
|
Apply for a casting call / View all applications
|
|
-
|
6
|
View / Edit Production Portfolio
|
-
|
|
7
|
Search for artistes / View suggested artistes / Favourite an artiste
|
-
|
|
8
|
Create / Edit a Casting Call / View all applicants of a Casting Call
|
-
|
|
9
|
Invite an artiste to a Casting Call
|
-
|
|
Session 1 (Casting Managers)
Session 1 was successfully conducted on 7 Sept 2012, 2 – 6pm at the office of Oak3 Films.
A total of 7 staff members from Oak3 Films assumed the role of a Casting Manager participated:
#Dennis (Head of Production)
- Jonathan (Producer)
- Jeremy (Production Assistant)
- PeiHua (Production Assistant)
- Jazz (Graphics)
- Dinesh (IT Executive)
- Ana (Secretary)
Feedback from Session 1 of the Usability Test is recorded in the User Feedback Document
View also our Response Plan to Feedback (Casting Mgrs).
Most common feedback:
Solution: Display all tooltips by default to users on the first time they create casting calls
|
Solution: Automatically scroll to bottom of page when date input is selected
|
Solution: Bring users directly to "edit portfolio" page the first time they sign up
|
Results from Casing Managers End-of-Test Qualtrics Survey (Extract):
Chart 1: 2 candidates expressed possibility that the system is unnecessarily complex
|
Watch how we conducted Session 1 HERE
|
Session 1 Conclusion
Inferring from the above table, we identified that some functions of the application may not be sufficiently intuitive (E.g. Items #2, 3 and 4). Perhaps this explains why some testers felt that the system is slightly complex (as shown in Chart 1).
One of the solutions to get around this is to introduce some simple tooltips to allow the user to get familiarized in using the system. We have also implemented some changes to improve the overall usability and intuitiveness of the application. View our Response Plan to Feedback (Casting Mgrs).
Nevertheless, many users expressed that on the whole, the system is easy to use, streamlined, is “time-saving” and user-friendly.
|
Session 2 (Artistes)
Session 2 was conducted from 9 - 14 Sept 2012
A total of 18 testers who assumed the role as Artistes participated.
Feedback from Session 2 of the Usability Test: Artiste User Feedback
View also our Response Plan to Feedback (Artistes).
Most common feedback:
Solution: Bring user to edit page on the first visit to the account and change all edit buttons to blue colour
|
Solution: Add label to each information row
|
Solution: Use 3 conventional drop down number selection
|
Solution: Change the layout to look more like a form
|
Results from Artiste End-of-Test Qualtrics Survey (Extract):
Chart 3: around 8 candidates felt that the system is unnecessarily complex
|
Chart 4: 6 candidates believed that they need assistance to use this system
|
Session 2 Conclusion
Session 2 provided us with much insights especially to the overall intuitiveness of using the system. The top few feedback involves how candidates had difficulties in navigating and using the system; and we have since made changes to the feedback given.
Charts 3 and 4 further support this issue, where many candidates believe that the system is either slightly complex or they would need the help of a person to fully exploit the system.
View our Response Plan to Feedback (Artistes)
|
Testing Methodology
a. Collecting qualitative metrics
- Users of the test is categorised into two groups:
- Peers/Family members assuming the Artiste role (Artiste Task List) and
- Staff members of Oak3 Films assuming the Casting Manager role (Casting Managers Task List)
- Users are encouraged to think out loud while accomplishing each task so as to better allow the facilitators to record down accurate observations and significant details of the testers’ thought processes. Facilitators will also be observing for errors (e.g. false clicks) and navigation issues. Upon completion of all the tasks, the tester will be presented a questionnaire (Artistes/CastingMgrs) in which he will be asked what he dislikes / likes about the application and their satisfaction level of the application.
- The testing process will also be recorded via video using Screenflow and will complement the facilitators’ observations. Facilitators are to record the timing in which a note-worthy observation (e.g. body language response) occurred.
b. Collecting quantitative metrics
- Quantitative metrics to be measured include:
- The amount of time to complete each task
- The number of clicks taken for completing each task (and of which how many are false clicks)
|
Supporting Documents UT1
|
Usability Test 2
Objectives
1. Determine that the usage of the features in the application are consistent to the expectations of real users (Casting Managers & Artistes)
2. Obtain feedback from users such as to improve the usability (learnability, efficiency, errors, satisfaction) and aesthetics of our application
|
Scope of Test
S/N
|
Features
|
Artiste
|
Prod. House
|
Casting Manager
|
1
|
Registration / Log in / Log out
|
|
|
|
2
|
Change / Reset password
|
|
|
|
3
|
View / Edit Artiste Portfolio
|
|
-
|
-
|
4
|
Search for casting calls / View suggested casting calls / Favourite a casting call
|
|
-
|
-
|
5
|
Apply for a casting call / View all applications
|
|
-
|
-
|
6
|
Accept/ reject an invitation
|
|
-
|
-
|
7
|
Select an audition/ View audition date
|
|
-
|
-
|
8
|
View / Edit Production Portfolio
|
-
|
|
-
|
9
|
Invite Casting Manager
|
-
|
|
-
|
10
|
View / Edit Casting Manager Portfolio
|
-
|
-
|
|
11
|
Search for artistes / View suggested artistes / Favourite an artiste
|
-
|
-
|
|
12
|
Create / Edit a Casting Call / View all applicants of a Casting Call
|
-
|
-
|
|
13
|
Invite an artiste to a Casting Call
|
-
|
-
|
|
14
|
Create / Edit / View Auditions
|
-
|
-
|
|
15
|
Manage Account
|
|
-
|
|
Types of Testing Conducted
initially, we planned to have both a lab test and field test for Usability Test 2. The lab test will be a more controlled one, whereby we could sit beside the users and observe their usage of our system. The field test, however, will be carried out at the actual environment (e.g. artistes at their homes and casting managers at their offices) to determine if they are capable of using the system at their own comfort without our guidance and interference.
Also, we wanted to carry out a pilot test before the actual test to verify if our planned test goes according to our expectations, and to make any necessary changes before the actual test.
The following chart shows our original plan for Usability Test 2:
Participants Profiles
Testing Methodology
a. Collecting qualitative metrics (Lab Test)
- Users are encouraged to think out loud while accomplishing each task so as to better allow the facilitators to record down accurate observations and significant details of the testers’ thought processes. Facilitators will also be observing for errors (e.g. false clicks) and navigation issues. Upon completion of all the tasks, the tester will be presented a questionnaire in which he will be asked what he dislikes / likes about the application and their satisfaction level of the application.
- The testing process will also be recorded via video using Screenflow and will complement the facilitators’ observations. Facilitators are to record the timing in which a note-worthy observation (e.g. body language response) occurred.
b. Collecting qualitative metrics (Field Test)
- Upon beginning the process, the tester will be presented a questionnaire in which he will be asked what he dislikes / likes about the application and their satisfaction level of the application. At the end of the user test, the results collected would be examined to observe for errors.
|
Supporting Documents UT2
|
Field Test
Objectives
1. Determine if users are capable of handling the application on their own (without any guidance from us) in the actual environment where they would be using the application
2. Obtain feedback from users such as to improve the usability (learnability, efficiency, errors, satisfaction) and aesthetics of our application
|
Field Test Details & Outcome
Field Test was conducted from 28 November 2012 to 1 December 2012 remotely at the artistes' own comfort. With the help and support of Oak3, we managed to get some artistes to sign up as a user of Casting3, and to apply for an ongoing casting call, "Koffee did it". They will be provided with a link to an evaluation form, where they will asked to give us their feedback.
As of the submission date of this wiki page (28 Nov), the field test is ongoing and results have yet to be collated. The results will be later concluded in the following links:
Feedback Links:
- 1. Profile of artistes who have signed up
- 2. Survey for Artistes and their feedback
- 3. Response Plan to Artiste Feedback
|