HeaderSIS.jpg

Difference between revisions of "IS480 Team wiki: 2016T2 Team Sirius User Tests"

From IS480
Jump to navigation Jump to search
 
(48 intermediate revisions by the same user not shown)
Line 47: Line 47:
 
<!--Content Start-->
 
<!--Content Start-->
 
<br>
 
<br>
<div style="background: #1d4c5d; padding: 15px; font-weight: bold; line-height: 0.3em; text-indent: 15px;letter-spacing:-0.08em;font-size:20px;"><font color=#ffffff>User Test 1</font></div> We have recruited our product owner and sponsor from IDA as our participants.</div>
+
<div style="background: #1d4c5d; padding: 15px; font-weight: bold; line-height: 0.3em; text-indent: 15px;letter-spacing:-0.08em;font-size:20px; id:UT2"><font color=#ffffff>User Test 3</font></div><br/>
 +
<u><h2>User Test 3</h2></u>
 +
<b>Venue:</b> Singapore Management University <br/>
 +
<b>Date:</b> 16 and 17 Mar 2016, Wednesday and Thursday<br/>
 +
<b>Time:</b> 1.00pm to 6.00pm <br/>
 +
<b>Duration:</b> ~60 minutes <br/>
 +
<b>Number of Participant(s):</b> 29 <br/>
 +
<b>User Test:</b> [http://goo.gl/forms/SUXpT1mhjA Instruction here] <br/>
 +
 
 +
<h3>Objectives</h3>
 +
<ul><li>To gather feedback on usability of the user interface from existing users</li>
 +
<li>Identify usability issues user might face </li></ul>
 +
 
 +
<h3>Scope for User Testing 3</h3>
 +
<ul>
 +
<li>Login/Logout</li>
 +
<li>Create an experiment </li>
 +
<li>Add a new goal to an existing experiment</li>
 +
<li>Interpret results from the experiment</li>
 +
</ul>
 +
 
 +
<h3>Participants</h3>
 +
We have recruited 29 students with programming background as our participants.
 +
 
 +
<h3>Goals and Results</h3>
 +
 
 +
{| class="wikitable" style="background-color:#FFFFFF; width: 800px;" align="center"
 +
|-
 +
! style="color:#ecf0f1; background-color:#1d4c5d;" width="50pt" | S/N
 +
! style="color:#ecf0f1; background-color:#1d4c5d;" width="200pt" | Goals
 +
! style="color:#ecf0f1; background-color:#1d4c5d;" width="200pt" | Reached/Reached?
 +
|-
 +
|style="text-align: center;"| 1
 +
|| Participants should be able to login and identify the number of experiments running
 +
|style="background-color:#9CF740"| Goal reached. All participants were able to complete the tasks.
 +
|-
 +
|style="text-align: center;"| 2
 +
|| Participants should be able to create a new experiment
 +
|style="background-color:#f7f759"| Goal reached. All participants were able to complete the tasks. All participants took a longer time to complete the task.
 +
|-
 +
|style="text-align: center;"| 3
 +
|| Participants should be able add new goals to an existing experiment
 +
|style="background-color:#f7f759"|
 +
Goal reached. All participants were able to complete the tasks. However, some participants felt that the layout and the number steps required was an issue.
 +
|-
 +
|style="text-align: center;"| 4
 +
|| Participants should be able to understand the experiment results
 +
|style="background-color:#f7f759"|
 +
Goal reached. All participants were able to complete the tasks. However, some participants found that the labels used was an issue for them to understand the result.
 +
|-
 +
|}
 +
 
 +
<h3>Key Findings</h3>
 +
{| class="wikitable" style="background-color:#FFFFFF; width: 800px;" align="center"
 +
|-
 +
! colspan=2 style="background-color: #1d4c5d;color:#ecf0f1; background-color:#1d4c5d;" | Overall Findings Summary
 +
|-
 +
| rowspan=5 style="text-align: center;"| Quantitative Analysis
 +
|| The experiment list page is full of experiments, no filter provided.
 +
|-
 +
|| Participants felt there are no feedback given that an experiment should be stopped before making changes to it.
 +
|-
 +
|| Participants were confused with the goal selectors' colours.
 +
|-
 +
|| Participants were confused with the results tab. They were unable to find the statistical significance.
 +
|-
 +
|| Some of the buttons that were purposely placed to guide the user to complete the task without any guidance were unused.
 +
|-
 +
| rowspan=21 style="text-align: center;"| Possible changes before handover
 +
|| Revise the goal selector and "how it works"
 +
|-
 +
|| Provide a prompt to user if the experiment is not stopped before making edits
 +
|-
 +
|| Provide a filter to filter out the experiments of different status
 +
|-
 +
|| Revise results tab, changed "Raw Data" to "Statistical Significance"
 +
|}
 +
 
 +
<h3>Detailed Findings</h3>
 +
{| class="wikitable" style="background-color:#FFFFFF; width: 800px;" align="center"
 +
|-
 +
||
 +
Participants should be able to login and identify the number of experiments running
 +
|style="background-color:#9CF740"|Goal #1 was reached
 +
|-
 +
|}
 +
[[Image: sirius_ut3_task1.png]]
 +
*82% of the participants rated the usability of this function as very easy.
 +
*Most of the participants were able to identify the number of experiments running.
 +
*However, 18% of the participants found the ‘total number of experiments running’ label was not obvious enough.
 +
 
 +
{| class="wikitable" style="background-color:#FFFFFF; width: 800px;" align="center"
 +
|-
 +
|| Participants should be able to create a new experiment
 +
|style="background-color:#f7f759"|Goal #2 was partially reached
 +
|-
 +
|}
 +
[[Image:sirius_ut3_task2.png]]
 +
*42% of the participants rated the usability of this function as easy and above.
 +
*Some participants found the layout is simple that makes it easy to select and understand to create a experiment.
 +
*However, many participants found the CSS selector was hard to use, precision was required to select the elements.
 +
*Some of the participants were unable to locate the “How does it work?”. Hence, they were unable to complete the tasks without any guidance.
 +
 
 +
[[Image: sirius_ut3_task2p2.png]]
 +
*86% of the participants do not understand the difference between the ‘Original’ and ‘Variation’ tabs
 +
*They might not understand the concept of AB testing tools.
 +
*Most of the participants mentioned that the ‘Original’ is the previous layout of the page and ‘Variation’ is the edited layout after adding the JQuery code.
 +
*Some of the participants assumed that the ‘Original’ was verison A and ‘Variation’ was version B which can be technically correct.
 +
 
 +
{| class="wikitable" style="background-color:#FFFFFF; width: 800px;" align="center"
 +
|-
 +
|| Participants should be able add new goals to an existing experiment 
 +
|style="background-color:#f7f759"|Goal #3 was partially reached
 +
|-
 +
|}
 +
[[Image: sirius_ut3_task3.png]]
 +
*58% of the participants rated the usability of this function as <b>neutral and below</b>.
 +
*Some participants found that the instruction given was not comprehensive and intuitive.
 +
*Some participants commented that they does not know that they must a stop the experiment before editing. No prompt or instruction was given.
 +
*No confirmation message was given after adding a new goal on an existing experiment.
 +
 
 +
{| class="wikitable" style="background-color:#FFFFFF; width: 800px;" align="center"
 +
|-
 +
|| Participants should able to find goals of from existing experiments
 +
|style="background-color:#f7f759"|Goal #4 was partially reached
 +
|-
 +
|}
 +
*71% of the participants rated the usability of this function as neutral and below.
 +
*Some of the participants found the terms used were confusing.
 +
*Some participants found there was lack of guidance to help them understand the terms and graph displayed.
 +
*Some participants commented that the inconsistency of terms used (rounding of figures) has confused them
 +
<br>
 +
<div style="background: #1d4c5d; padding: 15px; font-weight: bold; line-height: 0.3em; text-indent: 15px;letter-spacing:-0.08em;font-size:20px; id:UT2"><font color=#ffffff>User Test 2</font></div><br/>
 +
 
 +
<u><h2>User Test 2</h2></u>
 +
<b>Venue:</b> IDA <br/>
 +
<b>Date:</b> 4 Feb 2016, Thursday <br/>
 +
<b>Time:</b> 9.30am <br/>
 +
<b>Duration:</b> ~45 minutes <br/>
 +
<b>Number of Participant(s):</b> 4 <br/>
 +
<b>User Test:</b> [https://docs.google.com/document/d/1WmMsjmpSdsuHr_XnbzGNBG5pRi93ZGgmhOvlL0F00a8/pub Instuction here]
 +
 
 +
<h3>Objectives</h3>
 +
<ul><li>To gather feedback on usability of the user interface from existing users</li>
 +
<li>Identify usability issues user might face </li></ul>
 +
 
 +
<h3>Scope for User Testing 2</h3>
 +
<ul>
 +
<li>Login/Logout</li>
 +
<li>Create an experiment </li>
 +
<li>Add a new goal to an existing experiment</li>
 +
<li>Interpret results from the experiment</li>
 +
</ul>
 +
 
 +
<h3>Participants</h3>
 +
We have recruited four developers from IDA as our participants.
 +
 
 +
<h3>Goals and Results</h3>
 +
 
 +
{| class="wikitable" style="background-color:#FFFFFF; width: 800px;" align="center"
 +
|-
 +
! style="color:#ecf0f1; background-color:#1d4c5d;" width="50pt" | S/N
 +
! style="color:#ecf0f1; background-color:#1d4c5d;" width="200pt" | Goals
 +
! style="color:#ecf0f1; background-color:#1d4c5d;" width="200pt" | Reached/Reached?
 +
|-
 +
|style="text-align: center;"| 1
 +
|| Participants should be able to login and logout successfully without any form of guidance
 +
|style="background-color:#9CF740"| Goal reached. All participants were able to complete the tasks.
 +
|-
 +
|style="text-align: center;"| 2
 +
|| Participants should be able to create a new experiment
 +
|style="background-color:#f7f759"| Goal reached. All participants were able to complete the tasks. All participants took a longer time to complete the task.
 +
|-
 +
|style="text-align: center;"| 3
 +
||Participants should be able add new goals to an existing experiment
 +
|style="background-color:#f7f759"|
 +
Goal reached. All participants were able to complete the tasks. However, some participants felt that the layout and the number steps required was an issue.
 +
|-
 +
|style="text-align: center;"| 4
 +
||
 +
Participants should be able to understand the experiment results
 +
|style="background-color:#f7f759"|
 +
Goal reached. All participants were able to complete the tasks. However, some participants found that the labels used was an issue for them to understand the result.
 +
|-
 +
|}
 +
 
 +
<h3>Key Findings</h3>
 +
{| class="wikitable" style="background-color:#FFFFFF; width: 800px;" align="center"
 +
|-
 +
! colspan=2 style="background-color: #1d4c5d;color:#ecf0f1; background-color:#1d4c5d;" | Overall Findings Summary
 +
|-
 +
| rowspan=4 style="text-align: center;"| Quantitative Analysis
 +
|| The labels used were inconsistent or confusing.
 +
|-
 +
|| Participants felt the site was dull and could be more ‘lively’ instead.
 +
|-
 +
|| Participants took a longer time to create, edit and view the result of an existing experiment.
 +
|-
 +
|| Some of the buttons that were purposely placed to guide the user to complete the task without any guidance were unused.
 +
|-
 +
| rowspan=21 style="text-align: center;"| Possible changes before User Testing 3
 +
|| Revise the red box and its label on the login page
 +
|-
 +
|| Revise the labels on landing page after the user has login
 +
|-
 +
||Unnatural spacing in goal form partial
 +
|-
 +
||Order of goal types wrt. help text
 +
|-
 +
||Awkward snippet code placement
 +
|-
 +
||Step help icon not noticed
 +
|-
 +
||Update weightage button
 +
|-
 +
||Modal dialog at last step to start experiment immediately
 +
|-
 +
||SG help trigger showing when goal type is not 'click'
 +
|-
 +
||Source free public domain icons
 +
|-
 +
||Implement step click navigation
 +
|-
 +
||Total visits showing up for every goal
 +
|-
 +
||Make create experiment buttons more concise
 +
|-
 +
||Remove 'click to select' text on snippet code dropdown
 +
|-
 +
||Update weightage modal too wide
 +
|-
 +
||Experiment list table header capitalisation consistency
 +
|-
 +
||Reviewer wanted: total visit over total conversion, pie-chart opportunity?
 +
|-
 +
||No feedback upon experiment creation/edit (state change in general)
 +
|-
 +
||Livelier colour scheme
 +
|-
 +
||Validation errors that occur further down the page are not apparent when they happen (user needs to scroll down to see)
 +
|-
 +
||Implement no-experiment view
 +
|}
 +
 
 +
<h3>Detailed Findings</h3>
 +
{| class="wikitable" style="background-color:#FFFFFF; width: 800px;" align="center"
 +
|-
 +
||
 +
Participants should be able to login and logout successfully without any form of guidance
 +
|style="background-color:#9CF740"|Goal #1 was reached
 +
|-
 +
|}
 +
 
 +
*All participants rated the usability of this function as easy.
 +
*However, participants were distracted by the red box (labelled ‘You need to sign in or sign up before continued’). Hence, they made more steps to explore what does the label means before trying to login.
 +
 
 +
{| class="wikitable" style="background-color:#FFFFFF; width: 800px;" align="center"
 +
|-
 +
|| Participants should be able to create a new experiment
 +
|style="background-color:#f7f759"|Goal #2 was partially reached
 +
|-
 +
|}
 +
 
 +
*All participants rated the usability of this function as <b>neutral</b>.
 +
*All participants made numerous steps to complete the task.
 +
*All participants met problems to edit the weightage. They could not locate where to edit weightage and one commented that the popup can be improved, mainly on the layout and labels used.
 +
*Some participants did not realized the help button to complete their task. The button is not obvious. And the term used in the help menu is different from the selection
 +
*All participants could not locate the sidebar of the given web page for the experiment. This is because when the screen is narrow and the page is web responsive, the participants could see the sidebar. However,  the slide button (purpose to help the participant to view the full page) is not obvious until the participants were told.
 +
*Some participants were unclear of the CSS selector. They seems to unsure of the colours of the selection and to understand the main function of it.
 +
 
 +
{| class="wikitable" style="background-color:#FFFFFF; width: 800px;" align="center"
 +
|-
 +
|| Participants should be able add new goals to an existing experiment 
 +
|style="background-color:#f7f759"|Goal #3 was partially reached
 +
|-
 +
|}
 +
 
 +
*All participants rated the usability of this function as <b>neutral</b>.
 +
*Some participants commented that there was too much vertical spacing for the non-click goal on the configuration variation page.
 +
*Some participants commented that there was too many steps to add a new goal.
 +
 
 +
 
 +
{| class="wikitable" style="background-color:#FFFFFF; width: 800px;" align="center"
 +
|-
 +
|| Participants should able to find goals of from existing experiments
 +
|style="background-color:#f7f759"|Goal #4 was partially reached
 +
|-
 +
|}
 +
 
 +
*All participants rated the usability of this function as <b>neutral</b>.
 +
*All participants took a longer time to interpret the result.
 +
*Some participants could not find the statistical significance figure.
 +
*Some participants felt that the term (e.g. tab labels, chart labels, Scroll goal and page goal) used were confusing. There could be improvement on the charts (e.g. single variable chart is meaningless, the layout of the charts)
 +
 
 +
<br>
 +
<!--UT 1-->
 +
<div style="background: #1d4c5d; padding: 15px; font-weight: bold; line-height: 0.3em; text-indent: 15px;letter-spacing:-0.08em;font-size:20px;id:UT1"><font color=#ffffff>User Test 1</font></div>
  
 
<u><h2>User Test 1</h2></u>
 
<u><h2>User Test 1</h2></u>
Line 55: Line 351:
 
<b>Time:</b> 3.00pm <br/>
 
<b>Time:</b> 3.00pm <br/>
 
<b>Duration:</b> ~25 minutes <br/>
 
<b>Duration:</b> ~25 minutes <br/>
<b>Number of Participant(s):</b> 2  
+
<b>Number of Participant(s):</b> 2 <br/>
 +
<b>User Test:</b> [https://docs.google.com/document/d/12wnsBwy-IDWiAbHDLHG8Vf281lYE68xY68Ht6j3WHhY/pub Instuction here]
  
 
<h3>Objectives</h3>
 
<h3>Objectives</h3>
Line 63: Line 360:
 
<h3>Scope for User Testing 1</h3>
 
<h3>Scope for User Testing 1</h3>
 
<ul>
 
<ul>
<li>Create an experiment</l1>
+
<li>Create an experiment</li>
 
<li>Edit an existing experiment </li>
 
<li>Edit an existing experiment </li>
<li>Start/Stop an experiment</l1>
+
<li>Start/Stop an experiment</li>
 
<li>Preview an existing experiment</li>
 
<li>Preview an existing experiment</li>
 
</ul>
 
</ul>
Line 73: Line 370:
  
 
<h3>Goals and Results</h3>
 
<h3>Goals and Results</h3>
<center>
+
 
 
{| class="wikitable" style="background-color:#FFFFFF; width: 800px;" align="center"
 
{| class="wikitable" style="background-color:#FFFFFF; width: 800px;" align="center"
 
|-
 
|-
Line 81: Line 378:
 
|-
 
|-
 
|style="text-align: center;"| 1
 
|style="text-align: center;"| 1
|style="text-align: center;"| Participants should be able complete task #1 without any guidance from our test facilitator
+
|| Participants should be able complete task #1 without any guidance from our test facilitator
|style="text-align: center;"| Goal partially reached. Both participants were confused about the process (navigating from page to page), unsure about what the inputs were for.
+
|style="background-color:#f7f759"| Goal partially reached. Both participants were confused about the process (navigating from page to page), unsure about what the inputs were for.
 +
|-
 +
|style="text-align: center;"| 2
 +
|| Participants should be able to edit an existing experiment
 +
|style="background-color:#9CF740"| Goal reached. Both participants were able to complete the tasks. One of the participants had a lower rating and took a longer time to complete the task.
 +
|-
 +
|style="text-align: center;"| 3
 +
|| Participants should be able to understand the experiment results
 +
|style="background-color:#9CF740"| Goal reached. Both participants were able to complete the tasks. However, both participants rated the usability as difficult as the labels were confusing.
 +
|-
 +
|style="text-align: center;"| 4
 +
|| Participants should able to find goals of from existing experiments
 +
|style="background-color:#9CF740"| Goal reached. Both participants were able to complete the tasks.
 +
|-
 +
|style="text-align: center;"| 5
 +
|| Participants should be able to start and stop the experiment easily
 +
|style="background-color:#9CF740"| Goal reached. Both participants were able to complete the tasks.
 +
|}
 +
 
 +
<h3>Key Findings</h3>
 +
{| class="wikitable" style="background-color:#FFFFFF; width: 800px;" align="center"
 
|-
 
|-
|style="text-align: center;"| 1
+
! colspan=2 style="background-color: #1d4c5d;color:#ecf0f1; background-color:#1d4c5d;" | Overall Findings Summary
|style="text-align: center;"| 15 Oct 2015 - 29 Oct 2015
+
|-
|style="text-align: center;"| [https://docs.google.com/spreadsheets/d/11OcMnDOxy760OerDdQBGbezKBGMYlc2wINMKm0Vq5bw/pubhtml?gid=1049369371&single=true Sprint Backlog 1]
+
| rowspan=4 style="text-align: center;"| Quantitative Analysis
 +
|| When adding a new experiment, one participant was unsure whether to click on ‘load’ or ‘next’ button to proceed to the next step.
 +
|-
 +
|| The participants were confused by the input for “jQuery code” and “CSS selector”. The participants does not understand what are these inputs for.
 +
|-
 +
|| Participants took a longer time to edit the existing experiment.
 +
|-
 +
|| The header of the result table was confusing. Participants find it hard to understand the result
 +
|-
 +
| rowspan=5 style="text-align: center;"| Possible changes before User Testing 2
 +
|| Revise the editing experiments page to reduce the number of clicks required.
 +
|-
 +
|| Revise the labels on the adding of a new experiment or adding more explanation text/information.
 +
|-
 +
|| Revise the labels on the result page.
 +
|-
 +
|| Revise the inputs placeholders to be more obvious.
 +
|-
 +
|| Revise the “Adding of new experiment” interface to be more obvious as it is a multi step process.
 +
|}
 +
 
 +
<h3>Detailed Findings</h3>
 +
{| class="wikitable" style="background-color:#FFFFFF; width: 800px;" align="center"
 +
|-
 +
|| Goal #1 - Participants should be able complete task #1 without any guidance from our test facilitator
 +
|style="background-color:#f7f759"|Goal #1 was partially reached
 +
|-
 +
|}
 +
 
 +
<ul>
 +
<li>All participants rated the usability of this function as very easy to use.</li>
 +
<li>However, from our observation, the participants seem to be <u> rather confused about the naming of the labels.</u></li>
 +
<li>One of the participant stated that the preview of the experiment is not obvious. The position of the preview was placed at the bottom of the page that the participant did not noticed that the experiment was loaded. <b>Unnecessary scrolling was required</b></li>
 +
<li>One of the participant stated that there is a <b>lack of information on the terminologies and parameters used</b>. If the user is using the function for the first time, the user might not understand what he/she was doing. The participant suggested that we should change the terminologies or provide more instruction/information to guide the user.</li>
 +
</ul>
 +
 
 +
{| class="wikitable" style="background-color:#FFFFFF; width: 800px;" align="center"
 +
|-
 +
|| Participants should be able to edit an existing experiment &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 +
|style="background-color:#9CF740"|Goal #2 was reached
 +
|-
 +
|}
 +
 
 +
<ul>
 +
<li>All participants rated the usability of this function no less than neutral.</li>
 +
</ul>
 +
 
 +
{| class="wikitable" style="background-color:#FFFFFF; width: 800px;" align="center"
 +
|-
 +
|| Participants should be able to understand the experiment results   
 +
|style="background-color:#9CF740"| Goal #3 was reached
 +
|-
 +
|}
 +
 
 +
<ul>
 +
<li>All participants rated the usability of this function as <b>difficult to use</b>.</li>
 +
<li>All participants seem to be confused by the labels used (Total completions and Total views). They took a longer time trying to understand what do the labels mean.</li>
 +
<li>One of the participant navigated between the home page and result page twice. The participant was unclear whether he was on the wrong page.</li>
 +
<li>However, all participants still managed to answer the question correctly and complete the task without any guidance from the test facilitator.</li>
 +
</ul>
 +
 
 +
{| class="wikitable" style="background-color:#FFFFFF; width: 800px;" align="center"
 +
|-
 +
|| Participants should able to find goals of from existing experiments
 +
|style="background-color:#9CF740"| Goal #4 was reached
 
|-
 
|-
|style="text-align: center;"| 2
+
|}
|style="text-align: center;"| 29 Oct 2015 - 11 Nov 2015
+
 
|style="text-align: center;"| [https://docs.google.com/spreadsheets/d/11OcMnDOxy760OerDdQBGbezKBGMYlc2wINMKm0Vq5bw/pubhtml?gid=1064693501&single=true Sprint Backlog 2]
+
<ul>
 +
<li>All participants completed the task quickly and rated the usability of the application no less than neutral.</li>
 +
</ul>
  
 +
{| class="wikitable" style="background-color:#FFFFFF; width: 800px;" align="center"
 +
|-
 +
|| Participants should be able to start and stop the experiment easily
 +
|style="background-color:#9CF740"| Goal #5 was reached
 +
|-
 
|}
 
|}
</center>
+
 
<center>
+
<ul>
[[Image: sirius_ut1.png|150px|UT1|link=https://docs.google.com/document/d/1RL0W3l1HpZa_1xbboz21UDh9lQRYIaYMPk3LbxZ_PHo/pub]]
+
<li>All participants rated the usability of this function as <b>very easy to use</b>.</li>
[[Image: sirius_ut2.png|150px|UT2|link=]]
+
<li>All participants used the start and stop buttons on the home page.</li>
[[Image: sirius_ut3.png|150px|UT3|ink=]]
+
<li>All participants did not used the stop button on the “view experiment” page.</li>
</center>
+
</ul>
 +
 
 
<!--Content End-->
 
<!--Content End-->

Latest revision as of 23:39, 9 April 2016

Sirius home icon.png   HOME

 

Sirius aboutus icon.png   ABOUT US

 

Sirius projectoverview icon.png   PROJECT OVERVIEW

 

Sirius projectmgmt icon.png   PROJECT MANAGEMENT

 

Sirius doc icon.png   DOCUMENTATION

 


User Test 3


User Test 3

Venue: Singapore Management University
Date: 16 and 17 Mar 2016, Wednesday and Thursday
Time: 1.00pm to 6.00pm
Duration: ~60 minutes
Number of Participant(s): 29
User Test: Instruction here

Objectives

  • To gather feedback on usability of the user interface from existing users
  • Identify usability issues user might face

Scope for User Testing 3

  • Login/Logout
  • Create an experiment
  • Add a new goal to an existing experiment
  • Interpret results from the experiment

Participants

We have recruited 29 students with programming background as our participants.

Goals and Results

S/N Goals Reached/Reached?
1 Participants should be able to login and identify the number of experiments running Goal reached. All participants were able to complete the tasks.
2 Participants should be able to create a new experiment Goal reached. All participants were able to complete the tasks. All participants took a longer time to complete the task.
3 Participants should be able add new goals to an existing experiment

Goal reached. All participants were able to complete the tasks. However, some participants felt that the layout and the number steps required was an issue.

4 Participants should be able to understand the experiment results

Goal reached. All participants were able to complete the tasks. However, some participants found that the labels used was an issue for them to understand the result.

Key Findings

Overall Findings Summary
Quantitative Analysis The experiment list page is full of experiments, no filter provided.
Participants felt there are no feedback given that an experiment should be stopped before making changes to it.
Participants were confused with the goal selectors' colours.
Participants were confused with the results tab. They were unable to find the statistical significance.
Some of the buttons that were purposely placed to guide the user to complete the task without any guidance were unused.
Possible changes before handover Revise the goal selector and "how it works"
Provide a prompt to user if the experiment is not stopped before making edits
Provide a filter to filter out the experiments of different status
Revise results tab, changed "Raw Data" to "Statistical Significance"

Detailed Findings

Participants should be able to login and identify the number of experiments running

Goal #1 was reached

Sirius ut3 task1.png

  • 82% of the participants rated the usability of this function as very easy.
  • Most of the participants were able to identify the number of experiments running.
  • However, 18% of the participants found the ‘total number of experiments running’ label was not obvious enough.
Participants should be able to create a new experiment Goal #2 was partially reached

Sirius ut3 task2.png

  • 42% of the participants rated the usability of this function as easy and above.
  • Some participants found the layout is simple that makes it easy to select and understand to create a experiment.
  • However, many participants found the CSS selector was hard to use, precision was required to select the elements.
  • Some of the participants were unable to locate the “How does it work?”. Hence, they were unable to complete the tasks without any guidance.

Sirius ut3 task2p2.png

  • 86% of the participants do not understand the difference between the ‘Original’ and ‘Variation’ tabs
  • They might not understand the concept of AB testing tools.
  • Most of the participants mentioned that the ‘Original’ is the previous layout of the page and ‘Variation’ is the edited layout after adding the JQuery code.
  • Some of the participants assumed that the ‘Original’ was verison A and ‘Variation’ was version B which can be technically correct.
Participants should be able add new goals to an existing experiment Goal #3 was partially reached

Sirius ut3 task3.png

  • 58% of the participants rated the usability of this function as neutral and below.
  • Some participants found that the instruction given was not comprehensive and intuitive.
  • Some participants commented that they does not know that they must a stop the experiment before editing. No prompt or instruction was given.
  • No confirmation message was given after adding a new goal on an existing experiment.
Participants should able to find goals of from existing experiments Goal #4 was partially reached
  • 71% of the participants rated the usability of this function as neutral and below.
  • Some of the participants found the terms used were confusing.
  • Some participants found there was lack of guidance to help them understand the terms and graph displayed.
  • Some participants commented that the inconsistency of terms used (rounding of figures) has confused them


User Test 2


User Test 2

Venue: IDA
Date: 4 Feb 2016, Thursday
Time: 9.30am
Duration: ~45 minutes
Number of Participant(s): 4
User Test: Instuction here

Objectives

  • To gather feedback on usability of the user interface from existing users
  • Identify usability issues user might face

Scope for User Testing 2

  • Login/Logout
  • Create an experiment
  • Add a new goal to an existing experiment
  • Interpret results from the experiment

Participants

We have recruited four developers from IDA as our participants.

Goals and Results

S/N Goals Reached/Reached?
1 Participants should be able to login and logout successfully without any form of guidance Goal reached. All participants were able to complete the tasks.
2 Participants should be able to create a new experiment Goal reached. All participants were able to complete the tasks. All participants took a longer time to complete the task.
3 Participants should be able add new goals to an existing experiment

Goal reached. All participants were able to complete the tasks. However, some participants felt that the layout and the number steps required was an issue.

4

Participants should be able to understand the experiment results

Goal reached. All participants were able to complete the tasks. However, some participants found that the labels used was an issue for them to understand the result.

Key Findings

Overall Findings Summary
Quantitative Analysis The labels used were inconsistent or confusing.
Participants felt the site was dull and could be more ‘lively’ instead.
Participants took a longer time to create, edit and view the result of an existing experiment.
Some of the buttons that were purposely placed to guide the user to complete the task without any guidance were unused.
Possible changes before User Testing 3 Revise the red box and its label on the login page
Revise the labels on landing page after the user has login
Unnatural spacing in goal form partial
Order of goal types wrt. help text
Awkward snippet code placement
Step help icon not noticed
Update weightage button
Modal dialog at last step to start experiment immediately
SG help trigger showing when goal type is not 'click'
Source free public domain icons
Implement step click navigation
Total visits showing up for every goal
Make create experiment buttons more concise
Remove 'click to select' text on snippet code dropdown
Update weightage modal too wide
Experiment list table header capitalisation consistency
Reviewer wanted: total visit over total conversion, pie-chart opportunity?
No feedback upon experiment creation/edit (state change in general)
Livelier colour scheme
Validation errors that occur further down the page are not apparent when they happen (user needs to scroll down to see)
Implement no-experiment view

Detailed Findings

Participants should be able to login and logout successfully without any form of guidance

Goal #1 was reached
  • All participants rated the usability of this function as easy.
  • However, participants were distracted by the red box (labelled ‘You need to sign in or sign up before continued’). Hence, they made more steps to explore what does the label means before trying to login.
Participants should be able to create a new experiment Goal #2 was partially reached
  • All participants rated the usability of this function as neutral.
  • All participants made numerous steps to complete the task.
  • All participants met problems to edit the weightage. They could not locate where to edit weightage and one commented that the popup can be improved, mainly on the layout and labels used.
  • Some participants did not realized the help button to complete their task. The button is not obvious. And the term used in the help menu is different from the selection
  • All participants could not locate the sidebar of the given web page for the experiment. This is because when the screen is narrow and the page is web responsive, the participants could see the sidebar. However, the slide button (purpose to help the participant to view the full page) is not obvious until the participants were told.
  • Some participants were unclear of the CSS selector. They seems to unsure of the colours of the selection and to understand the main function of it.
Participants should be able add new goals to an existing experiment Goal #3 was partially reached
  • All participants rated the usability of this function as neutral.
  • Some participants commented that there was too much vertical spacing for the non-click goal on the configuration variation page.
  • Some participants commented that there was too many steps to add a new goal.


Participants should able to find goals of from existing experiments Goal #4 was partially reached
  • All participants rated the usability of this function as neutral.
  • All participants took a longer time to interpret the result.
  • Some participants could not find the statistical significance figure.
  • Some participants felt that the term (e.g. tab labels, chart labels, Scroll goal and page goal) used were confusing. There could be improvement on the charts (e.g. single variable chart is meaningless, the layout of the charts)


User Test 1

User Test 1

Venue: IDA
Date: 30 Oct 2015, Friday
Time: 3.00pm
Duration: ~25 minutes
Number of Participant(s): 2
User Test: Instuction here

Objectives

  • To gather feedback on usability of the user interface from existing users</l1>
  • Identify usability issues user might face

Scope for User Testing 1

  • Create an experiment
  • Edit an existing experiment
  • Start/Stop an experiment
  • Preview an existing experiment

Participants

We have recruited our product owner and sponsor from IDA as our participants.

Goals and Results

S/N Goals Reached/Reached?
1 Participants should be able complete task #1 without any guidance from our test facilitator Goal partially reached. Both participants were confused about the process (navigating from page to page), unsure about what the inputs were for.
2 Participants should be able to edit an existing experiment Goal reached. Both participants were able to complete the tasks. One of the participants had a lower rating and took a longer time to complete the task.
3 Participants should be able to understand the experiment results Goal reached. Both participants were able to complete the tasks. However, both participants rated the usability as difficult as the labels were confusing.
4 Participants should able to find goals of from existing experiments Goal reached. Both participants were able to complete the tasks.
5 Participants should be able to start and stop the experiment easily Goal reached. Both participants were able to complete the tasks.

Key Findings

Overall Findings Summary
Quantitative Analysis When adding a new experiment, one participant was unsure whether to click on ‘load’ or ‘next’ button to proceed to the next step.
The participants were confused by the input for “jQuery code” and “CSS selector”. The participants does not understand what are these inputs for.
Participants took a longer time to edit the existing experiment.
The header of the result table was confusing. Participants find it hard to understand the result
Possible changes before User Testing 2 Revise the editing experiments page to reduce the number of clicks required.
Revise the labels on the adding of a new experiment or adding more explanation text/information.
Revise the labels on the result page.
Revise the inputs placeholders to be more obvious.
Revise the “Adding of new experiment” interface to be more obvious as it is a multi step process.

Detailed Findings

Goal #1 - Participants should be able complete task #1 without any guidance from our test facilitator Goal #1 was partially reached
  • All participants rated the usability of this function as very easy to use.
  • However, from our observation, the participants seem to be rather confused about the naming of the labels.
  • One of the participant stated that the preview of the experiment is not obvious. The position of the preview was placed at the bottom of the page that the participant did not noticed that the experiment was loaded. Unnecessary scrolling was required
  • One of the participant stated that there is a lack of information on the terminologies and parameters used. If the user is using the function for the first time, the user might not understand what he/she was doing. The participant suggested that we should change the terminologies or provide more instruction/information to guide the user.
Participants should be able to edit an existing experiment            Goal #2 was reached
  • All participants rated the usability of this function no less than neutral.
Participants should be able to understand the experiment results Goal #3 was reached
  • All participants rated the usability of this function as difficult to use.
  • All participants seem to be confused by the labels used (Total completions and Total views). They took a longer time trying to understand what do the labels mean.
  • One of the participant navigated between the home page and result page twice. The participant was unclear whether he was on the wrong page.
  • However, all participants still managed to answer the question correctly and complete the task without any guidance from the test facilitator.
Participants should able to find goals of from existing experiments Goal #4 was reached
  • All participants completed the task quickly and rated the usability of the application no less than neutral.
Participants should be able to start and stop the experiment easily Goal #5 was reached
  • All participants rated the usability of this function as very easy to use.
  • All participants used the start and stop buttons on the home page.
  • All participants did not used the stop button on the “view experiment” page.