HeaderSIS.jpg

Difference between revisions of "IS480 Team wiki: 2012T1 M.O.O.T/Technical Applications/UT2"

From IS480
Jump to navigation Jump to search
 
Line 211: Line 211:
 
[[Image:MOOTUT2likechart2.png | 550px]]
 
[[Image:MOOTUT2likechart2.png | 550px]]
  
The stickman interaction is the most popular segment of our application. Many people were amazed and intrigued by the stickman imitating their actions. Some even stayed longer just to interact with the stickman.
+
The stickman interaction is the most popular segment of our application. Many people were amazed and intrigued by the stickman imitating their actions. Some even stayed longer just to interact with the stickman. Note that both the 'like' and 'dislike' questions are not mandatory, hence shoppers could choose not to answer these two questions.
  
 
[[Image:MOOTUT2dislikechart2.png | 550px]]
 
[[Image:MOOTUT2dislikechart2.png | 550px]]
 +
 +
Only a small number of shoppers choose to answer this question, as shown from the graph above that the responses for this question was significantly lesser than the responses for the 'like' question. Out of those who dislike, most did not enjoy the hat-picking scene the most.
  
 
[[Image:MOOTUT2instructionchart.png  | 550px]]
 
[[Image:MOOTUT2instructionchart.png  | 550px]]

Latest revision as of 11:11, 23 November 2012

Home

Team/Project Partners

Project Overview

Project Management

Design Specifications

Technical Applications


User Testing 1 User Testing 2 Development Technologies Hardware

Project Management & Documentation

Primary Research Secondary Research

User Testing 2

Our User Testing 2 (UT2) took place on 5 November from 12pm to 6pm.

Objectives

  • Verify if users can react to AlterSense without prompting
  • Test the intuitiveness of our user interface (photo taking) and hand gestures
  • Gather user opinion about using AlterSense in a mall setting
  • Test the accuracy of gender recognition in a mall setting

Location

  • Level 4 of new extension between Plaza Singapura and Dhoby Ghaut
MOOT UT2 setup.jpg
MOOT UT2 setup2.jpg

Unlike our previous user testing UT1, there would not be any demarcation marking the recommended standing position for shopper (about 4 metres away from the Kinect sensor) on the floor. AlterSense has been improved to use real world values; hence it does not matter where the shopper stands as long as the shopper’s whole figure is captured by Kinect.

Testers

  • General public
  • We had a total of 53 testers - 29 females and 24 males.
MOOTUT2genderoftesters.png MOOTUT2ageoftesters.png

Execution Flow

1. Ask shopper if they have a smartphone and to download Microsoft Tag if they have one. If they are unwilling to download, we lend them our phones
2. Let user use AlterSense by himself/herself
3. Observe user if he/she has any difficulties responding to AlterSense
4. Measure the time taken for user to raise his/her right hand and to scan the Microsoft tag
5. Let user complete a survey form

Qualitative metric

Facilitators will be observing for errors or problems that the testers faced. Upon completion of all the tasks, the tester will be presented a survey form in which he/she will be asked about their opinion and satisfaction level of the application.

Quantitative metric

  • Time taken for shopper to raise both hands
  • Time taken to choose a hat
  • Time taken to raise right hand
  • No. of failed detections of raising of both hands
  • No. of failed detections of choosing a hat
  • No. of failed detections of raising his/her right hand
  • Whether male shoppers had picked the top hat and female shoppers had pick the ladies’ hat
  • Whether shopper has been able to complete the whole flow (i.e. shopper did not leave halfway)
  • Response time to scan the Microsoft Tag
  • No. of failed attempts to scan the Microsoft Tag

Supporting Documents

Category Links
Pre-Testing Test Plan
Location of UT2
During testing Observation Form
Survey Form
Post-Testing Observation Form Responses
Survey Form Results
Response Plan to Problems

Observation Form Findings

MOOTUT2accuracy2.png

After switching to using real world values for this UT2, there was fortunately not a very significant drop in the accuracy of gender recognition when compared to our UT1. We have managed to overcome the initial problem from our UT1 that gender recognition would only be accurate when tester stands at a fixed spot from the Kinect sensor.

The accuracy of having shopper choose his/her gender-related hat is disappointingly low. This is because several shoppers are not particular with the choice of hat. For some, they found that the instruction to pick a hat was unclear and thus the gesture of using right hand to pick a hat as the gesture was detected immediately once shopper raise their right hand even though the shopper meant to raise the hand to pick the hat on the left. For others, they found the hat a little too small and could not really see the details of the hat. Thus they could not really differentiate between the two hats and hence just anyhow pick.

With this, we have decided to drop this functionality of using the hat-picking scene for unsupervised learning of Neural Network because the current way of using hats to verify the actual gender of shopper against the gender predicted from AlterSense is very inaccurate.

However the functionality of choosing hat will still be around to allow shopper to have an accessory to pose with for photo taking.

The button to allow for unsupervised learning will also still be around so that CMA can use it if they are able to find a better way to verify gender for unsupervised learning.

Problems Found And Their Solutions

No. Problem faced At which functionality it occurred Proposed Solution Status
Instruction
1 Instruction not present Stand still when shopper starts interacting with application Display this instruction Solved
2 Instruction not present due to bugs Choose door by walking to the left or right Solve the bug Solved
3 2 sets of instructions present, which confused the shopper [Instruction to raise both hands to be teleported remained though it was not supposed to be] Use right hand to pick a hat Fix this bug Solved
4 Because instruction was not clear, there were many false positives for the gesture of using right hand to pick a hat as the gesture was detected immediately once shopper raise their right hand Pick a hat by using right hand
  • Change the gesture such that shopper would raise either hand to pick female hat on the right or the male hat on the left
  • Increase the size of the hat
  • Improve the clarity of the instruction by stating that they have to ‘raise their hand’ instead of saying ‘using your hand’
Solved
5 Instruction (arrow) not present due to bugs Scan the Microsoft Tag to retrieve gender-related content Fix this bug Solved
Gestures
6 Shopper had difficulty using right hand to pick a hat because it requires them to raise their right elbow above their shoulders Use right hand to pick a hat Re-calibrate the gesture such that it only requires them to raise their palm above their shoulders Solved
Overall feedback
7 Photo of shopper is too pixelated; there is hair loss. Photo taking Opt for a more ideal background - plain background with less noise and dimmer lighting N.A
8 Font size of instruction was too small for some people At all instructions Enlarge the font size Solved

MOOTUT2timetaken.png

Time taken to choose a door is relatively longer than the rest because of the following reasons:

  • Some people enjoyed interacting with the stickman and continued to stand there, hence increasing their time taken to choose a door
  • Instruction to choose a door by walking to the left or right was not very clear for some people
  • Some, especially the older generation, are not very familiar with following the instructions on the screen
  • Some are not used to using hand gestures to make a choice

But the majority of the people eventually got the hang of it and became quicker in following instructions.

Survey Form Results

MOOTUT2likechart2.png

The stickman interaction is the most popular segment of our application. Many people were amazed and intrigued by the stickman imitating their actions. Some even stayed longer just to interact with the stickman. Note that both the 'like' and 'dislike' questions are not mandatory, hence shoppers could choose not to answer these two questions.

MOOTUT2dislikechart2.png

Only a small number of shoppers choose to answer this question, as shown from the graph above that the responses for this question was significantly lesser than the responses for the 'like' question. Out of those who dislike, most did not enjoy the hat-picking scene the most.

MOOTUT2instructionchart.png

Only a small number of shoppers found our instructions to be unclear. Out of those who found it unclear, most of them found the instructions in 'Picking of a hat' scene to be the most confusing.

MOOTUT2gesturechart.png

Some shoppers found the action of walking towards their chosen door to be confusing and not intuitive. This may be because they are not sure whether they are supposed to move forward till they're right in front of the door or just moving towards their right or left will do.

MOOTUT2presentablechart.png

Many shoppers found the user interface of AlterSense to be quite presentable. The average score of the presentability of our user interface is 3.53.

MOOTUT2funchart.png

Most of the shoppers were quite pleased with their overall experience with AlterSense and found it to be rather fun. The average score of the fun factor of AlterSense is 3.92.

MOOTUT2useagainchart.png

The majority of the shoppers were supportive of our application; they do not mind using it again and would also recommend it to their friends and family.

Photos from Photo Taking

MOOTKinectSnapshot-00-42-55.png MOOTKinectSnapshot-03-17-09.png MOOTKinectSnapshot-03-30-22.png
Fellow SMU student giving us his support Shoppers amused with the Kinect Shopper having fun
MOOTKinectSnapshot-04-20-44.png MOOTKinectSnapshot-04-17-13.png MOOTKinectSnapshot-04-41-02.png
Shopper finding the hat too big for her Another round of photo taking; now with her friend Lawrence & Esther (the one on the right) from CMA came over too
MOOTKinectSnapshot-05-21-31.png MOOTKinectSnapshot-05-06-51.png MOOTKinectSnapshot-05-33-09.png
Shopper smiling brightly for the camera Couple shot Team shot with Zhennan from CMA (Kenny had to leave halfway through though as he had lessons)

Photos

MOOTUT2image15.png MOOTUT2image9.png MOOTUT2image19.png
A closer look at our poster Standing still for photo taking Inez engaging the young shoppers
MOOTUT2image4.png MOOTUT2image5.png MOOTUT2image6.png
Shopper trying out AlterSense Shopper successfully detected as female At the photo-taking scene
MOOTUT2image7.png MOOTUT2image8.png MOOTUT2image18.png
Russell explaining to shopper At the doors scene Filling up the survey form
MOOTUT2image10.png MOOTUT2image11.png MOOTUT2image20.png
Lawrence taking a photo Explaining the whole flow to Lawrence and Esther
MOOTUT2image12.png MOOTUT2inez.png MOOTUT2image17.png
Zhennan amused by the stick man Inez posing with the poster Perhaps admiring our poster?
MOOTUT2image1.png MOOTUT2image2.png MOOTUT2image3.png
Tired but happy faces. Many thanks to Zhennan for helping us carry the heavy box back to SMU Russell's not tired, still has the energy to look at the camera Walking back to SMU