HeaderSIS.jpg

Difference between revisions of "IS480 Team wiki: 2018T1 Five Stones User Testing UAT3"

From IS480
Jump to navigation Jump to search
(UAT3 main content update v1.0 - Results)
(UAT3 main content update v23- All Content)
 
(4 intermediate revisions by the same user not shown)
Line 183: Line 183:
 
|-
 
|-
 
|}
 
|}
 +
 +
==User Behavior==
 +
{| class="wikitable" style="font-family: Helvetica; background-color:#FFFFFF; width: 750px;" align="center"
 +
|-
 +
 +
! style="color: #FFFFFF; background-color:#334156;" width="10pt" | S/N
 +
! style="color: #FFFFFF; background-color:#334156;" width="120pt" | Module/Function
 +
! style="color: #FFFFFF; background-color:#334156;" width="160pt" | User Behavior
 +
! style="color: #FFFFFF; background-color:#334156;" width="160pt" | Changes to be made
 +
|-
 +
 +
|style="text-align: center;"| 1
 +
|style="text-align: left;"| Overall UI design
 +
|style="text-align: left;"| The “triple line” shown on the user interface is misleading. User clicked on the button thinking that it is like a menu drop down
 +
|style="text-align: left;"| Change the icon on the user interface main page to reflect that the button represents collapsible instead of menu drop down
 +
 +
(i.e. Change to +/- icon buttons instead)
 +
 +
|-
 +
 +
|style="text-align: center;"| 2
 +
|style="text-align: left;"| Developer / Creating a datasource
 +
|style="text-align: left;"| Developer users doing the test are able to understand what is required from them. All developer users do not require assistance to complete the testing
 +
|style="text-align: left;"| NIL
 +
 +
|-
 +
 +
|style="text-align: center;"| 3
 +
|style="text-align: left;"| Report Customization
 +
|style="text-align: left;"| Users were able to create a simple business report using the report template quickly without any assistance
 +
|style="text-align: left;"| NIL
 +
 +
|-
 +
 +
|style="text-align: center;"| 4
 +
|style="text-align: left;"| Exporting / Export as PPT
 +
|style="text-align: left;"| Users did not have issues navigating to the correct position of the web application to find the button required to export the report template as PPT
 +
|style="text-align: left;"| NIL
 +
 +
|-
 +
 +
|style="text-align: center;"| 5
 +
|style="text-align: left;"| Exporting / Export as PDF
 +
|style="text-align: left;"| Users did not have issues navigating to the correct position of the web application to find the button required to export the report template as PDF
 +
|style="text-align: left;"| NIL
 +
 +
|-
 +
 +
|style="text-align: center;"| 6
 +
|style="text-align: left;"| Slideshow
 +
|style="text-align: left;"| User did not notice the “Exit Slide Show” button and does not know that there the button exists to exit the slide show
 +
|style="text-align: left;"| Improve colour scheme and the button highlighting to make it obvious for users
 +
 +
|-
 +
 +
|style="text-align: center;"| 7
 +
|style="text-align: left;"| Communications / Email
 +
|style="text-align: left;"| User went on to click on the “Attached” button thinking that the report template needs to be attached
 +
 +
(i.e. User did not realize that the files are already attached)
 +
 +
|style="text-align: left;"| Increase the font for the pre-attached report templates to highlight to users that the chosen report template has been attached
 +
 +
|-
 +
|}
 +
 +
==User Comments==
 +
{| class="wikitable" style="font-family: Helvetica; background-color:#FFFFFF; width: 750px;" align="center"
 +
|-
 +
 +
! style="color: #FFFFFF; background-color:#334156;" width="10pt" | S/N
 +
! style="color: #FFFFFF; background-color:#334156;" width="120pt" | Module/Function
 +
! style="color: #FFFFFF; background-color:#334156;" width="160pt" | Comments
 +
! style="color: #FFFFFF; background-color:#334156;" width="160pt" | Changes to be made
 +
|-
 +
 +
|style="text-align: center;"| 1
 +
|style="text-align: left;"| Navigation bar
 +
|style="text-align: left;"| User got confused regarding the  wording “template” in the web application. User got confused and thought that “templates” meant the templates they could choose to create the report
 +
|style="text-align: left;"| Change the wording of “template” to “reports” instead to reduce confusion
 +
 +
|-
 +
 +
|style="text-align: center;"| 2
 +
|style="text-align: left;"| Account / Create a new User
 +
|style="text-align: left;"| User suggested that if the password check fails for the initial “Password” field, the system should not allow the user to carry on to “Confirm Password” field
 +
|style="text-align: left;"| To show a error pop-up and stop the user from writing on the “Confirm Password” if the initial password written does not pass the password checker
 +
 +
|-
 +
 +
|style="text-align: center;"| 3
 +
|style="text-align: left;"| Developer / Creating a datasource
 +
|style="text-align: left;"| Developer page is very straightforward. Easy to understand what is required to be configured
 +
|style="text-align: left;"| NIL
 +
 +
|-
 +
 +
|style="text-align: center;"| 4
 +
|style="text-align: left;"| Developer / developer page
 +
|style="text-align: left;"| The label and the textbox field are too closely aligned together.
 +
 +
(i.e. The labelling and their textbox fields are too closely aligned together)
 +
 +
|style="text-align: left;"| Increase the spacing between the field labels and their respective textboxes
 +
 +
|-
 +
 +
|style="text-align: center;"| 5
 +
|style="text-align: left;"| Report Components / Draggable
 +
|style="text-align: left;"| The textbox component was not fully draggable. Only the grey areas can be dragged
 +
|style="text-align: left;"| Change the cursor of the mouse from the arrow to the “hand” icon to signal area that is draggable
 +
 +
|-
 +
 +
|style="text-align: center;"| 6
 +
|style="text-align: left;"| Report Customization
 +
|style="text-align: left;"| User recommended for the implementation of a ‘Search’ feature for the report templates to allow the finding of the templates to be quicker
 +
|style="text-align: left;"| To implement a search feature to find the report template by name
 +
 +
|-
 +
 +
|style="text-align: center;"| 7
 +
|style="text-align: left;"| Exporting / Export PPT
 +
|style="text-align: left;"| User found that exporting report as PPT is very straightforward. No issues locating and exporting as local copy
 +
|style="text-align: left;"| NIL
 +
 +
|-
 +
 +
|style="text-align: center;"| 8
 +
|style="text-align: left;"| Exporting / Export PDF
 +
|style="text-align: left;"| User also found that exporting report as PDF is equally straightforward. No issues locating and exporting as local copy
 +
|style="text-align: left;"| NIL
 +
 +
|-
 +
 +
|style="text-align: center;"| 9
 +
|style="text-align: left;"| Communications / Email
 +
|style="text-align: left;"| The “send email” button at the email main page is confusing
 +
|style="text-align: left;"| Change naming convention on button to “Compose Email” instead
 +
 +
|-
 +
 +
|style="text-align: center;"| 10
 +
|style="text-align: left;"| Communications / Email
 +
|style="text-align: left;"| There is no address book included in the email and users have to type in the email addresses by hand. Very prone to errors due to manual typing
 +
|style="text-align: left;"| NIL
 +
 +
|-
 +
|}
 +
 +
==Quantitative Finding==
 +
{| class="wikitable" style="font-family: Helvetica; background-color:#FFFFFF; width: 750px;" align="center"
 +
|-
 +
 +
! style="color: #FFFFFF; background-color:#334156;" width="5pt" | S/N
 +
! style="color: #FFFFFF; background-color:#334156;" width="190pt" | Description of Task
 +
! style="color: #FFFFFF; background-color:#334156;" width="110pt" | No. of clicks measured (Tester 1)
 +
! style="color: #FFFFFF; background-color:#334156;" width="110pt" | No. of clicks measured (Tester 2)
 +
! style="color: #FFFFFF; background-color:#334156;" width="110pt" | No. of clicks measured (Tester 3)
 +
! style="color: #FFFFFF; background-color:#334156;" width="110pt" | No. of clicks measured (Tester 4)
 +
! style="color: #FFFFFF; background-color:#334156;" width="110pt" | Average No. of clicks
 +
! style="color: #FFFFFF; background-color:#334156;" width="110pt" | Benchmark
 +
! style="color: #FFFFFF; background-color:#334156;" width="110pt" | Factored Benchmark (by 1.5)
 +
! style="color: #FFFFFF; background-color:#334156;" width="100pt" | Target Met? 
 +
 +
|-
 +
 +
|style="text-align: center;"| 1
 +
|style="text-align: left;"| Participants should be able to create a data source as a Developer
 +
|style="text-align: center;"| 15
 +
|style="text-align: center;"| 17
 +
|style="text-align: center;"| 15
 +
|style="text-align: center;"| 18
 +
|style="text-align: center;"| <b><u>16.25</u></b>
 +
|style="text-align: center;"| <b>13</b>
 +
|style="text-align: center;"| <b><u>19.50</u></b>
 +
|style="text-align: center;"| Yes
 +
 +
|-
 +
 +
|style="text-align: center;"| 2
 +
|style="text-align: left;"| Participants should be able to create a basic report template
 +
|style="text-align: center;"| 30
 +
|style="text-align: center;"| 26
 +
|style="text-align: center;"| 23
 +
|style="text-align: center;"| 27
 +
|style="text-align: center;"| <b><u>26.50</u></b>
 +
|style="text-align: center;"| <b>23</b>
 +
|style="text-align: center;"| <b><u>34.50</u></b>
 +
|style="text-align: center;"| Yes
 +
 +
|-
 +
 +
|style="text-align: center;"| 3
 +
|style="text-align: left;"| Participants should be able to find and change the charts data on the dashboard
 +
|style="text-align: center;"| 20
 +
|style="text-align: center;"| 20
 +
|style="text-align: center;"| 15
 +
|style="text-align: center;"| 17
 +
|style="text-align: center;"| <b><u>18</u></b>
 +
|style="text-align: center;"| <b>15</b>
 +
|style="text-align: center;"| <b><u>22.5</u></b>
 +
|style="text-align: center;"| Yes
 +
 +
|-
 +
 +
|style="text-align: center;"| 4
 +
|style="text-align: left;"| Participants should be able to export the report template as PPT format
 +
|style="text-align: center;"| 3
 +
|style="text-align: center;"| 2
 +
|style="text-align: center;"| 2
 +
|style="text-align: center;"| 3
 +
|style="text-align: center;"| <b><u>2.50</u></b>
 +
|style="text-align: center;"| <b>2</b>
 +
|style="text-align: center;"| <b><u>4.5</u></b>
 +
|style="text-align: center;"| Yes
 +
 +
|-
 +
 +
|style="text-align: center;"| 5
 +
|style="text-align: left;"| Participants should be able to export the report template as PDF format
 +
|style="text-align: center;"| 2
 +
|style="text-align: center;"| 2
 +
|style="text-align: center;"| 3
 +
|style="text-align: center;"| 2
 +
|style="text-align: center;"| <b><u>2.25</u></b>
 +
|style="text-align: center;"| <b>2</b>
 +
|style="text-align: center;"| <b><u>4.5</u></b>
 +
|style="text-align: center;"| Yes
 +
 +
|-
 +
 +
|style="text-align: center;"| 6
 +
|style="text-align: left;"| Participants should be able to view the report template as a slideshow presentation from the browser
 +
|style="text-align: center;"| 3
 +
|style="text-align: center;"| 4
 +
|style="text-align: center;"| 4
 +
|style="text-align: center;"| 3
 +
|style="text-align: center;"| <b><u>3.50</u></b>
 +
|style="text-align: center;"| <b>4</b>
 +
|style="text-align: center;"| <b><u>6</u></b>
 +
|style="text-align: center;"| Yes
 +
 +
|-
 +
 +
|style="text-align: center;"| 7
 +
|style="text-align: left;"| Participants should be able to find and send the created report template as PPT & PDF via email
 +
|style="text-align: center;"| 7
 +
|style="text-align: center;"| 9
 +
|style="text-align: center;"| 8
 +
|style="text-align: center;"| 10
 +
|style="text-align: center;"| <b><u>8.50</u></b>
 +
|style="text-align: center;"| <b>7</b>
 +
|style="text-align: center;"| <b><u>10.5</u></b>
 +
|style="text-align: center;"| Yes
 +
 +
|-
 +
 +
|-
 +
|}
 +
 +
<div style="font-family: helvetica; font-size:15px">
 +
Overall Status:
 +
<ul>
 +
<li>All the task managed to meet the target below the factored benchmark</li>
 +
</ul>
 +
</div>
 +
 +
==Post-Test Questionnaire==
 +
[[File:Post-Test UAT3 pic1.PNG | 900px]]
 +
[[File:Post-Test UAT3 pic2.PNG | 900px]]
 +
[[File:Post-Test UAT3 pic3.PNG | 900px]]
 +
 +
==Overall Comments==
 +
<div style="font-family: helvetica; font-size:15px">
 +
<ul>
 +
<li>Web application is very straightforward and easy to use</li>
 +
<li>Users liked the password checker at the account page</li>
 +
<li>Users prefer if the web application has a “Search” function for the report templates to make searching easier</li>
 +
</ul>
 +
</div>

Latest revision as of 16:10, 16 November 2018

Logo 1.3.png


ClebsHome.png HOME

ClebsAboutUS.png ABOUT US

ClebsProjectOverview.png PROJECT OVERVIEW

ClebsProjectManagement.png PROJECT MANAGEMENT

ClebsDocumentation.png DOCUMENTATION

Persona & Scenarios Prototypes Meeting Minutes Diagrams User Testing Deliverables


5Stones UAT1.PNG 5Stones UAT2.PNG 5Stones UAT3.PNG


UAT 3 Test Plan

Venue: SingPost Centre, Coffee Bean
Date: 11 November 2018, Sunday
Time: 5 pm
Number of Participant(s): 4
Target Audience: Sponsor’s clients (Developers)

Objectives

  • To test core developed functionalities to gather feedback and gain deeper insights (i.e. Login, Creation of reports & exporting functions such as Powerpoint of PDF format)
  • To identify potential usability issues from users while using the web application
  • To gather comments following changes to UAT 2 for further improvements
  • To improve SCube web application’s overall User Interface (UI) and User Experience (UX)

Scope

  1. Account Module
    • Login/Logout
  2. Dashboard Module
    • Displaying business reports with basic graphs
  3. Report Components module
    • Draggable components (i.e. text, header, image, charts, video)
  4. Report Customization Module
    • User-customizable options for component properties (e.g. formatting of text and charts)
  5. Importing module
    • Process different data types from multiple data sources
  6. Chart Population Module
    • Loading data into charts
    • Integrate retrieved data sources to allow customization of report components and properties
  7. Report Descriptive Analytics Module
    • Display descriptive information of report (i.e. Summary Statistics on charts)
  8. Exporting Module
    • Exporting created report to PPT format
    • Exporting created report to PDF format
  9. Slideshow Module
    • View saved report as a slideshow in the browser
  10. Communications Module
    • Support emailing of exported report to client’s email
  11. Report Preview Module
    • Preview the report with component values according to URL parameter
    • Generate PDF based on preview

Procedure

  1. Participants will be provided with the link to the web application
  2. A set of task instructions will be will be handed out to participants
  3. Participants are encouraged to think aloud when they do the testing
  4. Participants will be given minimal assistance during the test so that user behavior can be observed by the team
  5. There will be a scribe assigned by the team to record down user behaviors, comments and feedback from the participants
  6. Participants will help the team fill in the post-UAT feedback form

Results

S/N Goals Achieved? % of participants able to complete the task
1 Participants should be able to login and logout Yes Pass. 100% successful
2 Participants should be able to create a data source and select the data configurations without any guidance from the facilitator Yes Pass. 100% successful
3 Participants should be able to create a report template based on the default template or using a new blank template Yes Pass. 100% successful
4 Participants should be able to create and customize charts (i.e. Bar and Line charts) on the dashboard Yes Pass. 100% successful
5 Participants should be able to see basic summary statistics when they create a chart (i.e. Both bar & line chart inclusive) Yes Pass. 100% successful
6 Participants should be able to export the created report template as PPT and PDF format respectively Yes Pass. 100% successful
7 Participants should be able to view the created report template as a slideshow on the browser Yes Pass. 100% successful
8 Participants should be able to be to send out the created report template through in-build email function Yes Pass. 100% successful
9 Participants should be able to open up a selected report template in PDF format by using a given URL link Yes Pass. 100% successful

User Behavior

S/N Module/Function User Behavior Changes to be made
1 Overall UI design The “triple line” shown on the user interface is misleading. User clicked on the button thinking that it is like a menu drop down Change the icon on the user interface main page to reflect that the button represents collapsible instead of menu drop down

(i.e. Change to +/- icon buttons instead)

2 Developer / Creating a datasource Developer users doing the test are able to understand what is required from them. All developer users do not require assistance to complete the testing NIL
3 Report Customization Users were able to create a simple business report using the report template quickly without any assistance NIL
4 Exporting / Export as PPT Users did not have issues navigating to the correct position of the web application to find the button required to export the report template as PPT NIL
5 Exporting / Export as PDF Users did not have issues navigating to the correct position of the web application to find the button required to export the report template as PDF NIL
6 Slideshow User did not notice the “Exit Slide Show” button and does not know that there the button exists to exit the slide show Improve colour scheme and the button highlighting to make it obvious for users
7 Communications / Email User went on to click on the “Attached” button thinking that the report template needs to be attached

(i.e. User did not realize that the files are already attached)

Increase the font for the pre-attached report templates to highlight to users that the chosen report template has been attached

User Comments

S/N Module/Function Comments Changes to be made
1 Navigation bar User got confused regarding the wording “template” in the web application. User got confused and thought that “templates” meant the templates they could choose to create the report Change the wording of “template” to “reports” instead to reduce confusion
2 Account / Create a new User User suggested that if the password check fails for the initial “Password” field, the system should not allow the user to carry on to “Confirm Password” field To show a error pop-up and stop the user from writing on the “Confirm Password” if the initial password written does not pass the password checker
3 Developer / Creating a datasource Developer page is very straightforward. Easy to understand what is required to be configured NIL
4 Developer / developer page The label and the textbox field are too closely aligned together.

(i.e. The labelling and their textbox fields are too closely aligned together)

Increase the spacing between the field labels and their respective textboxes
5 Report Components / Draggable The textbox component was not fully draggable. Only the grey areas can be dragged Change the cursor of the mouse from the arrow to the “hand” icon to signal area that is draggable
6 Report Customization User recommended for the implementation of a ‘Search’ feature for the report templates to allow the finding of the templates to be quicker To implement a search feature to find the report template by name
7 Exporting / Export PPT User found that exporting report as PPT is very straightforward. No issues locating and exporting as local copy NIL
8 Exporting / Export PDF User also found that exporting report as PDF is equally straightforward. No issues locating and exporting as local copy NIL
9 Communications / Email The “send email” button at the email main page is confusing Change naming convention on button to “Compose Email” instead
10 Communications / Email There is no address book included in the email and users have to type in the email addresses by hand. Very prone to errors due to manual typing NIL

Quantitative Finding

S/N Description of Task No. of clicks measured (Tester 1) No. of clicks measured (Tester 2) No. of clicks measured (Tester 3) No. of clicks measured (Tester 4) Average No. of clicks Benchmark Factored Benchmark (by 1.5) Target Met?
1 Participants should be able to create a data source as a Developer 15 17 15 18 16.25 13 19.50 Yes
2 Participants should be able to create a basic report template 30 26 23 27 26.50 23 34.50 Yes
3 Participants should be able to find and change the charts data on the dashboard 20 20 15 17 18 15 22.5 Yes
4 Participants should be able to export the report template as PPT format 3 2 2 3 2.50 2 4.5 Yes
5 Participants should be able to export the report template as PDF format 2 2 3 2 2.25 2 4.5 Yes
6 Participants should be able to view the report template as a slideshow presentation from the browser 3 4 4 3 3.50 4 6 Yes
7 Participants should be able to find and send the created report template as PPT & PDF via email 7 9 8 10 8.50 7 10.5 Yes

Overall Status:

  • All the task managed to meet the target below the factored benchmark

Post-Test Questionnaire

Post-Test UAT3 pic1.PNG Post-Test UAT3 pic2.PNG Post-Test UAT3 pic3.PNG

Overall Comments

  • Web application is very straightforward and easy to use
  • Users liked the password checker at the account page
  • Users prefer if the web application has a “Search” function for the report templates to make searching easier