IS480 Team wiki: 2012T1 M.O.O.T/Technical Applications/UT2
|User Testing 1||User Testing 2||Development Technologies||Hardware||Primary Research||Secondary Research|
User Testing 2
Our User Testing 2 (UT2) took place on 5 November from 12pm to 6pm.
- Verify if users can react to AlterSense without prompting
- Test the intuitiveness of our user interface (photo taking) and hand gestures
- Gather user opinion about using AlterSense in a mall setting
- Test the accuracy of gender recognition in a mall setting
- Level 4 of new extension between Plaza Singapura and Dhoby Ghaut
Unlike our previous user testing UT1, there would not be any demarcation marking the recommended standing position for shopper (about 4 metres away from the Kinect sensor) on the floor. AlterSense has been improved to use real world values; hence it does not matter where the shopper stands as long as the shopper’s whole figure is captured by Kinect.
- General public
- We had a total of 53 testers - 29 females and 24 males.
1. Ask shopper if they have a smartphone and to download Microsoft Tag if they have one. If they are unwilling to download, we lend them our phones
2. Let user use AlterSense by himself/herself
3. Observe user if he/she has any difficulties responding to AlterSense
4. Measure the time taken for user to raise his/her right hand and to scan the Microsoft tag
5. Let user complete a survey form
Facilitators will be observing for errors or problems that the testers faced. Upon completion of all the tasks, the tester will be presented a survey form in which he/she will be asked about their opinion and satisfaction level of the application.
- Time taken for shopper to raise both hands
- Time taken to choose a hat
- Time taken to raise right hand
- No. of failed detections of raising of both hands
- No. of failed detections of choosing a hat
- No. of failed detections of raising his/her right hand
- Whether male shoppers had picked the top hat and female shoppers had pick the ladies’ hat
- Whether shopper has been able to complete the whole flow (i.e. shopper did not leave halfway)
- Response time to scan the Microsoft Tag
- No. of failed attempts to scan the Microsoft Tag
|Pre-Testing|| Test Plan |
Location of UT2
|During testing||Observation Form|
|Post-Testing|| Observation Form Responses |
Survey Form Results
Response Plan to Problems
Observation Form Findings
After switching to using real world values for this UT2, there was fortunately not a very significant drop in the accuracy of gender recognition when compared to our UT1. We have managed to overcome the initial problem from our UT1 that gender recognition would only be accurate when tester stands at a fixed spot from the Kinect sensor.
The accuracy of having shopper choose his/her gender-related hat is disappointingly low. This is because several shoppers are not particular with the choice of hat. For some, they found that the instruction to pick a hat was unclear and thus the gesture of using right hand to pick a hat as the gesture was detected immediately once shopper raise their right hand even though the shopper meant to raise the hand to pick the hat on the left. For others, they found the hat a little too small and could not really see the details of the hat. Thus they could not really differentiate between the two hats and hence just anyhow pick.
With this, we have decided to drop this functionality of using the hat-picking scene for unsupervised learning of Neural Network because the current way of using hats to verify the actual gender of shopper against the gender predicted from AlterSense is very inaccurate.
However the functionality of choosing hat will still be around to allow shopper to have an accessory to pose with for photo taking.
The button to allow for unsupervised learning will also still be around so that CMA can use it if they are able to find a better way to verify gender for unsupervised learning.
Problems Found And Their Solutions
|No.||Problem faced||At which functionality it occurred||Proposed Solution||Status|
|1||Instruction not present||Stand still when shopper starts interacting with application||Display this instruction||Solved|
|2||Instruction not present due to bugs||Choose door by walking to the left or right||Solve the bug||Solved|
|3||2 sets of instructions present, which confused the shopper [Instruction to raise both hands to be teleported remained though it was not supposed to be]||Use right hand to pick a hat||Fix this bug||Solved|
|4||Because instruction was not clear, there were many false positives for the gesture of using right hand to pick a hat as the gesture was detected immediately once shopper raise their right hand||Pick a hat by using right hand||
|5||Instruction (arrow) not present due to bugs||Scan the Microsoft Tag to retrieve gender-related content||Fix this bug||Solved|
|6||Shopper had difficulty using right hand to pick a hat because it requires them to raise their right elbow above their shoulders||Use right hand to pick a hat||Re-calibrate the gesture such that it only requires them to raise their palm above their shoulders||Solved|
|7||Photo of shopper is too pixelated; there is hair loss.||Photo taking||Opt for a more ideal background - plain background with less noise and dimmer lighting||N.A|
|8||Font size of instruction was too small for some people||At all instructions||Enlarge the font size||Solved|
Time taken to choose a door is relatively longer than the rest because of the following reasons:
- Some people enjoyed interacting with the stickman and continued to stand there, hence increasing their time taken to choose a door
- Instruction to choose a door by walking to the left or right was not very clear for some people
- Some, especially the older generation, are not very familiar with following the instructions on the screen
- Some are not used to using hand gestures to make a choice
But the majority of the people eventually got the hang of it and became quicker in following instructions.
Survey Form Results
The stickman interaction is the most popular segment of our application. Many people were amazed and intrigued by the stickman imitating their actions. Some even stayed longer just to interact with the stickman. Note that both the 'like' and 'dislike' questions are not mandatory, hence shoppers could choose not to answer these two questions.
Only a small number of shoppers choose to answer this question, as shown from the graph above that the responses for this question was significantly lesser than the responses for the 'like' question. Out of those who dislike, most did not enjoy the hat-picking scene the most.
Only a small number of shoppers found our instructions to be unclear. Out of those who found it unclear, most of them found the instructions in 'Picking of a hat' scene to be the most confusing.
Some shoppers found the action of walking towards their chosen door to be confusing and not intuitive. This may be because they are not sure whether they are supposed to move forward till they're right in front of the door or just moving towards their right or left will do.
Many shoppers found the user interface of AlterSense to be quite presentable. The average score of the presentability of our user interface is 3.53.
Most of the shoppers were quite pleased with their overall experience with AlterSense and found it to be rather fun. The average score of the fun factor of AlterSense is 3.92.
The majority of the shoppers were supportive of our application; they do not mind using it again and would also recommend it to their friends and family.
Photos from Photo Taking
|Fellow SMU student giving us his support||Shoppers amused with the Kinect||Shopper having fun|
|Shopper finding the hat too big for her||Another round of photo taking; now with her friend||Lawrence & Esther (the one on the right) from CMA came over too|
|Shopper smiling brightly for the camera||Couple shot||Team shot with Zhennan from CMA (Kenny had to leave halfway through though as he had lessons)|
|A closer look at our poster||Standing still for photo taking||Inez engaging the young shoppers|
|Shopper trying out AlterSense||Shopper successfully detected as female||At the photo-taking scene|
|Russell explaining to shopper||At the doors scene||Filling up the survey form|
|Lawrence taking a photo||Explaining the whole flow to Lawrence and Esther|
|Zhennan amused by the stick man||Inez posing with the poster||Perhaps admiring our poster?|
|Tired but happy faces. Many thanks to Zhennan for helping us carry the heavy box back to SMU||Russell's not tired, still has the energy to look at the camera||Walking back to SMU|