2011T2 Bazinga UserTest2

From IS480
Jump to navigation Jump to search
Main Wiki Page   Project Management   Project Metrics   Technical Complexity & Deployment   User Test 2   Beta Test   BP Tech Fair   Team Reflections

Bazinga user test2.png

Actual Testing Dates: 21st to 28th of March 2012

Please go to our Schedule, Observations and User Test 2 Details that provides all the data that has been recorded.

UT 2 Summary

Our User Test 2 took place on 21/3/12and ended on 28/3/12.
A total of 35 users participated.
In order to ensure that our test results are not affected by any biasness, our users included students from different faculties and include a mixture of both non-Kinect and Kinect users so that we can anticipate the general public's receptiveness to our application once it is launched. The following documents were what we prepared for our User Testing 2.

Feel free to view the schedule and User Testing execution process that has been prepared, as shown in the first external link. The second link is the feedback form that all users will fill in after trying out the testing.

Prepared Documents Link
User Test 2 Schedule, Execution Process UT 2 Schedule and Execution process
User Test 2 Feedback Questions SMU Quadtrics Feedback Questions
Download UT 2 Feedback
User Test 2 Plan Download UT 2 Plan
User Test 2 Instructions for Users Download instructions
UT 2 Feedback From Users UT 2 Feedback Form Results
User Test 2 Schedule and Observations UT 2 Observation Results
UAT Results/Analysis UT 2 Analysis

UT 2 Introduction

After our MidTerm presentation, we decided to improve our User Testing process, especially since we are creating an application on the Kinect platform, which is still a new technology in the industry as compared to mobile phones. Thus, we decided to implement black box testing, which will enable us to get both qualitative and quantitative data from this User Test 2. This will enable us to understand what are the main issues faced by the users when placed in a quiet environment where they will not be able to ask the developers any questions pertaining to the list of tasks they will have to complete.

Black Box Testing

To ensure that our 2T1 meets our objectives in allowing the user to use the application without our help, we made it a non guided User Test. We wanted to verify if the users could pick up the gestures naturally and provide us opinion on how our group can further improve our application. Thus, we will place instructions at the side of the users (both left and right) and the user will follow the steps required to allow us to know which features require more help from the user, without the need of someone to provide immediate help.

UT2 Goals

Collect useful usability and gesture feedback from our users to better understand their needs and improve our existing application.

UT2 Objectives

  • Unguided Test for all users
  • Verify if the user finds it intuitive and easy to pick up hand gestures
  • Verify the usability and easy navigation of the application
  • Gather user opinion on Kinect Application feasibility in a retail setting

Difference from UT 1

UT 1 and 2 Differences
Bazinga ut 1 2 difference.png

UT 2 Setup

Room setup

Bazinga setup.png

Equipment Used

Bazinga equipment.png

Observers’ Tasks

Observer's points to note for each User Observer's procedures
Here are a list of points that we have to do to ensure that we captured all the relevant data from the user.
  • Time to perform a task
  • Percentage of task completed
  • Number of errors
  • Number of repetitions or failed commands
  • Number of features or commands not used
  • Time spent navigating and/or searching for information
  • Number of clicks or taps to perform a task
  • Quality/Quantity of information found
Collocate log files for each participant/scenario

Summarize each scenario across all participants, includes plots and trend analysis

  • Quantitative measures
  • Qualitative observations
  • Questionnaire answers

Make conclusions for each test goal

Observers’ Checklist

Here is the checklist that all observers will hold to record down any details they see during the testing.

Bazinga ut2 checklist.png

Importance of Tutorial Videos for User Test 2

How it started

Through the FYP project, our team realize that for Kinectify to be successful in a commercial setting, it requires users using the application to be able to know how to perform the different gestures. Due to this, our team has identify Tutorial Videos to be a core portion of the project where users will be able to call out for tutorial videos whenever they are unable to perform a certain gestures or when they get "stuck" using the application.

Purpose of Tutorial Videos

For the tutorial videos, we aim to let users learn how to use kinectify in the most engaging and easiest manner ever. The tutorial videos will be like a virtual assistant being there by your side to teach you how to perform gesture step by step. In the event that users get stuck through the application, they can invoke tutorials through a simple gesture.

Challenges We Face

1 Different from Mobile Application or Website
We realize how different the tutorial are from a mobile application or even a website. We realize we are unable to simply give instructions in a manual or just through a screen as our application requires some form of "teacher" to teach the users how the properly perform the gesture. This is interesting as our team are being engage mentally and physically on creating tutorial videos that can teach users without us being there. We realize how detail we must be in our tutorial video but at the same time not making it too long if not we will lose the attention of our users.
2 What the best way to show a tutorial video

The team has came a long way from creating the tutorial videos. From the first video to the most current video we have produce, we have experiment with many different ways to let users learn how to use gestures in our web application. This was something interesting and also challenging for us as we do not have a so call "industry standard" we can use given that Commercial Kinect applications are very new in the market. We simply have to create the tutorial and then test the market response to see if the videos are effective.

3 No Industry Standard

Given the Kinect has always been used as a gaming platform, there is not "Industry Standard" when it comes to tutorial videos in a commercial setting. We have seen many different applications where Kinect is used in a commercial setting. However, one thing we notice is how all these application are mostly prove of concepts or there are users present to guide users through. As such, to help give us direction for our tutorial video creation, our team use Kinect games like "Kinect Sports" or "Dance Central" as a guide where we see how they showcase their tutorial video. From there, we tweet it into a more commercial ready environment where users can learn gestures in the easiest way and also engaging manner.

Learning Process - Observe, Testing, Improve Approach

To ensure that our tutorial videos are engaging and also commercially feasible. We went through the road of observing the different ways commercial applications are push into the market and how tutorials are taught. The team even went to shops with establish touch screen base applications to see how they taught their users. Through these different findings, we always improve the tutorial videos. How can we gauge if the tutorial is effect? For that, our team will consult opinions or even through User Testing environment to see how users react to the tutorial video. From the feedback we receive, we improve the tutorial videos. Through each version of the tutorial videos, we realize how effectively we are teaching the users.

User Test 2 Video Tutorial Journey and Progress

To meet the goals we define to create our video tutorials, our team tackles two critical questions:

How should the tutorials be shown?

From this question, the team aims to tackle how tutorials should be display and perceive by the end users. Through this question, you will see how our team tackle challenges like how we approach in flimming our tutorials or how we even show case step by step approaches to end-users. Ideally, our team will like the users to see the videos and instantly know how to use our applications.

Kinectsports instructions 1.png Kinectsports instructions 2.png Kinect sports.png

From the screen shots above, our team took reference from established kinect games like "Kinect Sports Season 1". Before the start of every game, there is tutorial to users where the users are engage in a step by step tutorial to aid their learning. We thought it was highly applicable to our application and thus we attempt to produce tutorial videos that are similar. The reason why we took kinect games as our reference as this gaming companies has spend millions of dollars doing research as to how tutorials can be taught to users and what the best way to learn them. For us, we felt it would be a good starting base for our tutorials.

Tutorial Timeline

In total there are mainly 7 versions of the tutorial video that has been created. This excludes any minor changes we made along the way to any versions. The different versions can be summarize as shown:

Version 1


Tutv1a.png Tutv1b.png

The idea here is to show a simple videos where a highlight of a person is shown. The person will be facing the camera from the front. Simple instructions will appear when the user perform gestures like two handed navigation to help assist the user.


Tutv1b.png This screenshot showns how users will not be able to gauge the difference in hand position they have to place.

We realize that the video is hard to view and is not very intuitive for users. As it is a single color highlight, it will be hard for users to differentiate how an actual swipe should be or even how to properly do a 2 hand navigation.Furthermore, the instructions are not very clear. For example, terms like "Push Forward". What exactly is push forward as different users will have different interpretations of this meaning. This are things in the tutorial we had to improve on.

Version 2

For this version of the tutorial video. It was an improvement over the last video where the demonstrator is clearly shown. More meaningful instructions are added from just "Navigate" to "Navigate Downwards".

Tutv2a.png Tutv2b.png

Limitations from this video is that the mask worn by the demonstrator is too scary and might scare some of the users. Apart from that, the instructions and images are still not clear enough for users. In fact, they might be redundant as the demonstrator is already performing the same action. Another limitation is where this video suffers the same problem to the last tutorial video. As it is forward facing, it was hard for users to tell actions like 2 hand navigation as they are unable to tell how far their hands should be apart.

Version 3

More in-depth studies was done with reference to kinect games and proof of concept kinect applications online. Our team came out with this new tutorial video where it shows a step by step 3 column gesture screen. This time round, more visible changes were made to the tutorial.

Tutv3c.png For a start, a bright and striking title screen was included to let the users know what tutorial are they watching. Apart from this,the demonstrator is properly attired instead of having one color overlay or wearing a mask. Furthermore, the tutorial video breaks down the gestures into steps where each step shows a 45 degree view along with a front view. Depending on the gesture, the front view might be replace with a top view as the top view will be able to teach the user more effectively. We hope that through each step, users can see how this gesture is properly perform, especially with two different perspective. It should help the users greatly.

Tutv3a.png Tutv3b.png

Though the tutorials were a huge improvements over tutorial 2, we realize how users still struggle to use the application. The tutorial videos do not show the actual kinectify screenshots. Thus though users can see how the demonstrator does the action, they are unable to visualize how this actions will impact on the actual application itself. Apart from that, due to the choice of clothing by the demonstrator, we realize that the pale pink cardigan blends too well with the white background that it makes it hard to view certain gestures. A different outfit could have been better. The wordings to describe each step was still too vague. For example, things like ready position does not mean anything to user.

Version 4


More visual improvements were made. This time round, in view of the limitations mention in Version 3. We included the kinectify screenshot that show users the actual screen they will be interacting it. Apart from that, there is a main instruction shown the center that show users should do. There is also a improvement in the instructions given to users where it shows user for each step, what will be the purpose of the gesture. Example, Browse Up/Down is shown where the demonstrator browse with her right hand. The fancy title for each tutorial screen has been replace with our demonstrator performing the Help gesture and the title spell with bold font above.Overall, what we did this version was a huge improvement over the last version as users could now visualize how the different gestures interact with the kinectify screen.

Additional feature we added to this tutorial video was speech where every instruction was spoken to ensure users could hear it as they perform.

Tutv4a.png Tutv4b.png Tutv4c.png

Limitations Some limitations are as follows:

Tutv4c.png For the tutorial title, the font appears too kiddish or unprofessional. Furthermore, given the attire of the demonstrator, it blends too well with the white background that makes it hard for users to view properly.

Tutv4a.png For this screen, we can see center banners grab the attention of the users. As they are set to appear at the same time as the Steps below, it makes it very confusing for users to differentiate where to focus on. The font for the instructions can be improve and the demonstrator attire still blends too well to the background.

We also realize how our tutorial videos do not cover basic gestures like going back. This is something we like to explore for the next tutorial video.

The voice instruction was great but it was too robotic. We needed a more natural voice instruction. We also eliminated instructions like "excellent or great job" as it was redundant during the tutorial videos.

Version 5


From the shortcomings of the last version, the team went back to the drawing board and ask ourselves "Where can we improve from here?" During the brainstorming, our team took details into consideration like

1. When would instructions, screenshot and demonstrator appear so users can have clarity of instructions and not be confuse?
2. How fast can we get users to understand the instructions in each step?
3. The proper wordings for instructions to use to ensure they are clear to the users and ensure word is not too technical for users
4. The timing to flash each instructions to users.
We basically re-shot all our tutorial videos through a very elaborated setup where the demonstrator was flim along with the kinectify screen shot as shown below. We have broken down the different gestures to have different angles. For most gestures, we maintain a 45 degree posture and for certain gesture like the fast scroll for 2 hand navigation, we use a front shot approach as shown below. [image1] [image2]

This time round, we even choose an attire that stands out for the video (Black Cardigan with Velvet red top) and prevented it from blending with the white background.

Changes were also made to the tutorial title page, instead of just "How to Browse Menu", we have included a screenshot of what users will be expecting in the next screen.

A introduction quick tip was also added to the tutorial video as we wanted users to know how to go back or get help through the application. This will be played whenever the application is started.

A new natural voice instruction was recorded to give users a more natural experience and interaction.

For this tutorial video, all instructions would appear first and spoken to users before the demonstrator and screenshot appear to show users.

Background music was added to make tutorial video more fun and entertaining.

The end result is a sleek tutorial application. During UT2, this tutorial was place in our ut2 environment to see its effectiveness.

Tutv5a.png Tutv5b.png Tutv5c.png Tutv5d.png Tutv5e.png

It was through UT2 environment (4 days) that we saw the limitations of these tutorial video.

For a start, we realize how users do not recognize the help gestures as it seems like a title page instead of a gesture. As a result, 1st day of UT2, the users could not get help from the application.

Tutv5d.png From the screenshot above of the tutorial title, we realize that the tutorial title page was too short for users to notice what was actually going on. Furthermore, the animation of our demonstrator doing the help gesture confuse users. Another critical issue was also how each tutorial title look too similar that whenever it was played to users, they skip it instantly thinking it was the same tutorial video.

We also realize how lengthy our tutorial videos are at 1min plus. It was too lengthy and thus we needed to shorten it.

This time round, the tutorial was able to guide users properly but minor changes needed to be made and these changes were done through the 4 days UT2 testing.

Version 6

Through the UT2, we made several immediate changes to the tutorial video base on the user experiences. This was necessary as it was critical problems that users face and would have drastic impact on UT2. For a start, we streamline the videos in terms of duration and steps. For example, for 2 hand navigation gesture, we initially had 7 steps as we separate left scroll from right scroll. After streamlining, we left 5 core steps where we group gestures like scroll left and right together. This has greatly has reduce the overall video timings from 1 min plus to all less than a minute. For the tutorial title menu, we have crop the screenshots to focus on specific areas and a notepad is place there to tell users what sort of screen or browser are they interacting with. This helps give a different feel to each tutorial video. Apart from that, the quick tips segment was given a revamp where by it has its own title screen and a intro to kinectify. The quick tip will show users what they will be expecting in Kinectify and what are the main gestures they should remember.

Tutv6a.png Tutv6b.png Tutv6c.png Tutv6d.png

One limitation is that we realize that users do not know when the tutorial end so we have to find a way. Another limitation is that users do not remember what is the previous step about so we must find some way for them to recall what was the previous step they have done for the gesture. One other limitation was the background music that sounded repetitive and gave the impression that the tutorial video was similar. All these are solve in the next version.

Version 7


The most current version of the tutorial videos where we address the shortcomings from the last version. We have included a end screen for each tutorial video so users know when the video ends as shown in the screen below. Apart from that, we have rename each Step in the video to show what the step is about. Users can be reminded about what each step is to help them complete the gesture. Lastly, we use a more light base music where each tutorial video had a different starting point. It gave a fresh new look and feel to each tutorial so users can differentiate them easily.

Tutv7a.png Tutv7b.png

When should the tutorials be shown?

The challenge

From this question, we aim to tackle on exactly when should the tutorial video be display. This is a challenging question as a tutorial video can make or break an application. Consider the following two scenarios that we observe from UT2,

Scenario A

Tutorial videos are only shown when users invoke the help gesture. We notice how many users come to application and they simply get lost. Despite the help signs, many do not really want to invoke the tutorial. The tutorial was very important for the 2 hand navigation as without it, it was quite hard for any users to figure out how to use it.

Scenario B

We show the tutorial video every time user goes into a new page. We thought forcing the user into the tutorial scenario would be the best case. Apparently, it was not as effective as we thought as we saw users either getting irritated by the repetitive tutorials from the same gestures or they simply skip every single tutorial as they find it hindering their actions.

Scenario C (Something we thought off but did not try it out)

Having a session base tutorial where the first instance a user uses the application, they will see every tutorial once. When they return to a similar page, they will not be shown the tutorial video as it is part of the same session. One problem with this is that was that our application had no login. Simply to say, it was impossible to tell when a user session started or when it ended. A user could found what he needs and walk away half way during our application.

As shown, the different scenarios post different difficulties but are considerations we had while developing the tutorial video. A group might have the best tutorial in the world, but then, the ultimate question lies " Where and when exactly to put them?"

Our Solution

Based on the different scenarios, our team studied how users react to tutorial videos and then came out with what we thought was the best way to integrate tutorial videos into the application.

Whenever the application appears idle or no users are in front of it, a screensaver would appear. If more than 10seconds, we assume that a new user is here to use the application and a quick tip tutorial video will be played to introduce the system. If the user happens to be an existing user, he/she can simply skip the tutorial. Through each screen, we choose to "simplify" each screen with possible instructions to help users use the application. In the worst case scenario, they can do the help gesture to invoke the application.

Through this approach, we saw how user experience was greatly improve and how every user could learn the gestures we have in Kinectify in a fun and engaging way.

UT 2 Results

Users Demographics

UT2 breakdown.png

Type Number Percentage
SIS Students 25 71%
Non-IS Students 9 26%
Professors and Instructors 1 3%

Comments: Non-SIS students represent the user population from other faulties in SMU and their comments tend to be much less technical, reflecting most of the actual target public users. In addition, they also require much more guidance than SIS students, who are more experienced in application development and new technologies. Non SIS students provided us with an interesting dimension to the feedback collected, such as more of the business feasibility of the Kinect application, as compared to the SIS Students who will be more inclined towards the technology, and will tell more of which functions are more useful.

The instructor, on the other hand, is also very technical and provided us the best advice and opinions for a user to pick up gestures naturally, and how the application will be suitable for the public, based onher vast experiences with lots of applications.

In addition, we had a good mix of males and females, giving us a good mix of reviews, not bias only to one gender.


Type Number Percentage
Male 19 54%
Female 16 46%

User Testing 2 Outcome

Our team has received very positive feedbacks and suggestions on how we can improve our application. Most of the users feel that the Kinect application is cool, interactive, and still have some room to improve in the Augmented Reality function and the gestures used in the application. They see a potential in this application, with only a minority of 3 users who would not want to use the application again. We are very happy that 32 out of 35 users will want to use our application again, signifying the application has a huge future ahead. Many users told us verbally on how impressed they are, expressing the usefulness of this application.

We truly thank our users to help us for a 30 minute test even though it was the crucial week filled with deadlines.

We did a thorough analysis came up with a document that can be downloaded here.
It explains:

  • all the problems faced by our users
  • key findings we have observed
  • number of times a user had to watch a tutorial video
  • time difference between all the users

It has been truly awesome to review what our users have to say to us. Despite the late nights of reviewing their videos, we found the whole process extremely enriching and useful for the future of this application.

User Testing 2 Documents

For more information on our User Testing 2, you can download our documents below:

Documents Link
UT 2 Feedback From Users UT 2 Feedback Form Results
User Test 2 Schedule and Observations UT 2 Observation Results
UAT Results/Analysis UT 2 Analysis

A Big Thank You!!!

Bazinga postershot.jpg

We truly thank our users to help us for a 30 minute test even though it was the crucial week filled with deadlines!! :)

She gave us the best support we could ever wish for. Her persevering spirit for us has been truly amazing. She never failed to give us useful suggestions and ways for us to improve our User Testing process.
We TRULY THANK YOU from the bottom of our hearts! for all your effort and time!!

It has been truly awesome to review what our users have to say to us. Despite the late nights of reviewing their videos, we found the whole process extremely enriching and useful for the future of this application. We have learnt alot through this whole user testing 2 process.

Here is a picture below showing our tired faces after our whole User Testing 2 Process.

Bazinga ut2.jpg