HeaderSIS.jpg

2012T1 Fortune Cookies: User Testing

From IS480
Jump to navigation Jump to search


 Test Reports

Dashboard   Team & Project Overview   Project Design   Project Management   Progress Summary   Test Reports
Category   Heuristic Evaluation   Paper Prototype   Hallway Test   Lodestone End User Test   Usability Test 1   Usability & Field Test   UAT   Software Test  


Category

  • content -- give user what they expect to get
  • functionality - making sure specific function is working as it should be without giving any error.
  • navigation -- showing user the correct information after he / she chooses specific path
  • speed -- responding time or loading time needed
  • user interface :
    • layout -- where and how information is shown
    • design -- colors and shape displayed



We aim to reach our XFactor by doing these :

FortuneCookiesTestings.PNG



















User Acceptance Test

1. Objective
  • To obtain confirmation that a system meets mutually agreed-upon requirements.


2. Executive Summary
A final verification of the required business function and proper functioning of the system, emulating real-world usage conditions


3. Results
UAT1.1
UAT1.2

UAT2

UAT3

UAT4

UAT5.1
UAT5.2


Usability and Field Test - 9 November 2012

1. Objective
  • To determine design inconsistencies and usability problem areas within the user interface and content areas. Potential sources of error may include:
  • Navigation errors – failure to locate functions, excessive keystrokes to complete a function, failure to follow recommended screen flow.
  • Presentation errors – failure to locate and properly act upon desired information in screens, selection errors due to labelling ambiguities.
  • Control usage problems – improper toolbar or entry field usage.
  • Exercise the application under controlled test conditions with representative users. Data will be used to access whether usability goals regarding an effective, efficient, and well-received user interface have been achieved.
  • Establish functional performance of application under actual field usage.


2. Executive Summary
In order to measure the quality and level of intuitiveness of the new Delivery Excellence Dashboard, a series of usability tests are conducted with suitable volunteers of similar demographics and background, as well as staffs from Lodestone Management Consultants. The test will simulate the response, reaction and usage of users that are new to the new application and framework. Users will test the application under given scenarios that replicate their job responsibilities. Test researchers will note down usability points during the test through observation of users’ feedbacks and habits. A trace of navigation habits and time taken by the user is also logged and will be analysed to derive to quantify each usability goal. Users will also be directed to explore the application after the test base on their usage of the application in their job scope and feedback on their experience.


3. Participants
Participants will be members from Lodestone Management Consultant. The participants are a managing consultant taking a role of project manager in the application, an associate partner taking the role of country lead/board member and the administrator of the application herself.
The participants' responsibilities will be to attempt to complete a set of representative task scenarios presented to them in as efficient and timely a manner as possible, and to provide feedback regarding the usability and acceptability of the user interface. The participants will be directed to provide honest opinions regarding the usability of the application, and to participate in post-session subjective questionnaires and debriefing.


4. Method:
1. The participants will receive and overview of the usability test procedure, scenario and role taken, equipment and software.


Project Manager Background scenario: “You are a Project Manager at Lodestone Management Consultant based in Singapore, managing various project in the region. Your country lead (Stephen Wise) introduces a new quality assurance tool that requires you to complete a questionnaire for each of the project you are managing. Stephen also says that the tool will be helpful to give you an insight about the different aspects of your project.”


Administrator Background scenario: “You are an Administrator at Lodestone Management Consultant based in Singapore, tasked to support the running of different applications in the office. Your country lead (Stephen Wise) introduces a new quality assurance tool that requires you to support the various colleagues in your office and manage any system changes according to changes to Lodestone MC’s business rules. Project Managers in your office will also be contacting you to help them create entities of the projects they are managing into the system.”


Board Member Background scenario: “You are a Board Member at Lodestone Management Consultant based in Singapore, tasked to support the running of different applications in the office. Your fellow Board Member (Stephen Wise) introduces a new quality assurance and monitoring tool that will help you monitors the global progress of Lodestone MC’s projects. You have been wondering how projects in different regions are performing and this tool might prove to be a great help for you to analyse that.”


2. User is given his account credential and the will be allowed to try out the application for 3 minutes.


3. Based on the given scenario, user will attempt to complete their task within stipulated time.
4. After the test, user will fill up a survey regarding their use of the application.


5. Tasks Completed
  • Manage users (CRUD of users, including assigning role(s) to users)
  • Add and View offices
  • Manage projects
  • Manage regions
  • Modify questionnaires
  • View the graph for analysis of status report progression
  • Interactive map that shows how the project is going for specific country
  • Download report
  • Filter function


6. Observations
Test Observations



7. Details of Usability Test Result
User Test 1 & User Test 2 Overview


Survey Highlight
Survey Highlight


8. Field Test Conclusion
  • All functionalities performed as expected
  • Suggestion to include cost and possible integration with SAP system for more robust data in future development.
  • Apart from a few fix needed, users felt the user interface is good and easy to accomplish their task.

Usability Test 1 - 30 October 2012


1. Objective
  • To determine design inconsistencies and usability problem areas within the user interface and content areas. Potential sources of error may include:
  • Navigation errors – failure to locate functions, excessive keystrokes to complete a function, failure to follow recommended screen flow.
  • Presentation errors – failure to locate and properly act upon desired information in screens, selection errors due to labelling ambiguities.
  • Control usage problems – improper toolbar or entry field usage.
  • Exercise the application under controlled test conditions with representative users. Data will be used to access whether usability goals regarding an effective, efficient, and well-received user interface have been achieved.
  • Establish baseline user performance and user-satisfaction levels of the user interface for future usability evaluations.



2. Executive Summary
To simulate product launch and the response of target users to the new application, Delivery Excellence Dashboard. We will engage Professors with prior experience in project management to test the application under given scenarios. Each user will take up a specific role and test out the scope of functionality designed for that user role. Test researchers will note down usability points during the test. A trace of navigation habits and time taken by the user is also logged and will be analysed to derive to quantify each usability goal.


3. Participants
Subjects will be Professors with demographics of 28 – 40 years old with background in project management.
The participants' responsibilities will be to attempt to complete a set of representative task scenarios presented to them in as efficient and timely a manner as possible, and to provide feedback regarding the usability and acceptability of the user interface. The participants will be directed to provide honest opinions regarding the usability of the application, and to participate in post-session subjective questionnaires and debriefing.


4. Method:
1. The participants will receive and overview of the usability test procedure, scenario and role taken, equipment and software.


Project Manager Background scenario: “You are a Project Manager at Lodestone Management Consultant based in Singapore, managing various project in the region. Your country lead (Stephen Wise) introduces a new quality assurance tool that requires you to complete a questionnaire for each of the project you are managing. Stephen also says that the tool will be helpful to give you an insight about the different aspects of your project.”


Administrator Background scenario: “You are an Administrator at Lodestone Management Consultant based in Singapore, tasked to support the running of different applications in the office. Your country lead (Stephen Wise) introduces a new quality assurance tool that requires you to support the various colleagues in your office and manage any system changes according to changes to Lodestone MC’s business rules. Project Managers in your office will also be contacting you to help them create entities of the projects they are managing into the system.”


Board Member Background scenario: “You are a Board Member at Lodestone Management Consultant based in Singapore, tasked to support the running of different applications in the office. Your fellow Board Member (Stephen Wise) introduces a new quality assurance and monitoring tool that will help you monitors the global progress of Lodestone MC’s projects. You have been wondering how projects in different regions are performing and this tool might prove to be a great help for you to analyse that.”


2. User is given his account credential and the will be allowed to try out the application for 3 minutes.


3. Based on the given scenario, user will attempt to complete their task within stipulated time.


5. Tasks Completed
  • Manage users (CRUD of users, including assigning role(s) to users)
  • Add and View offices
  • Manage projects
  • Manage regions
  • Modify questionnaires
  • View the graph for analysis of status report progression
  • Interactive map that shows how the project is going for specific country
  • Download report
  • Filter function


6. Action Items
S/N Description Category Status Date Modified
1 To edit the date picker to be easily changed from year to year without much clicks Functionality done 7 November
2 To synchronize the naming used Content done 8 November
3 To change toolbox position and content Content done 8 November
4 To put the project description Functionalities ... ...


7. Evaluation from Testers / Professors
Observation.PNG



Usability test 1.PNG

Software Test

White Box Test

1. Description

The tester must be aware of how the output will be from the input he/she gives and the code used.
To simplified the white box test, the coder himself/herself test the specific part by himself/herself.


2. Test Date : 17 November

Black Box Test

1. Description

The tester is only aware of what the software is supposed to do, not how it does it.
To implement black box test, the person who is coding for certain part is not testing that specific part, and do the test for other member's part.


2. Test Date : 14 November

Regression Test

1. Objective

  • to make sure that old functions tested are still working well after the major changes in code
  • to ensure the old bugs are not coming back


2. Test Date

  • Regression Test 1 : 28 September
  • Regression Test 2 : 15 October
  • Regression Test 3 : 14 November


3. Functions Tested

Administrator
RT-admin-1.PNG
RT-admin-2.PNG
Project Manager
RT-PM.PNG
Higher Manager
RT-HigherManager.PNG



4. Regression Test Highlights

Regression Test 2 Highlight -1
Regression Test 2 Highlight -2



Regression Test 3 Highlight


5. Regression Test Details

Regression Test 1
Regression Test 2
Regression Test 3

Load Balancing Testing

1. Objective

  • to implement the scenario of daily activities in Lodestone when around 15 people will use the application at the same time


2. Description

Five of Fortune Cookies members are running the application in several browsers at the same time to measure the scalability of our application


3. Test Date : 9 November

Browser Compatibility Test

1. Objective

Every browser reacts differently with the codes that we have. Therefore, to ensure usability, we want our users to be comfortable in using our application in their preferred browser.
We have tested our application in :
  • Google Chrome
  • Internet Explorer
  • Mozila Firefox
  • Safari


2. Test Date

Browser Compatibility Tests are done on the same days with Regression Test.
  • Browser Compatibility Test 1 : 28 September
  • Browser Compatibility Test 2 : 15 October
  • Browser Compatibility Test 3 : 14 November

User Testing 4 (Lodestone End User Test) - 25 September 2012


1. Objective
  • To make sure that all agreeable functionalities between client and us are in the system
  • To verify that all functions work properly
  • Ensure that end users are able to perform their designated job functions with the new system
  • Better understand the alignment of objectives and goals for each category of stakeholders
  • Gather feedback on the quality and preference of data visualisation


2. Description
End User Test is used to test whether all functionalities in the system work as they planned. By testing within the boundaries of work or task flows, End User test will be able to check if the intended output is available for user based on specific input given.
All users except Admin, are given sample questions of what they will need to use the application for, such as :
  • How is the project in Singapore going?
  • Is there any improvement from the previous report?

This test will involve 5 stakeholders who have different roles, which are :
  • Stephen Wise - Engagement Partner
  • Rui Imamura - Project Manager
  • Ben Inglish - Project Manager
  • ParameshwaranKN - Project Manager
  • Rachel Tann - Admin


3. Methods
  • Users will be given the test case for specific role assigned.
  • At the start of the test, we gave users 10 minutes to explore the application.
  • Given the test case, users will follow the scenario, and put Pass / Fail in the column. Additional comments can also be added in by users.


4. Tasks Completed
  • Manage users (CRUD of users, including assigning role(s) to users)
  • Add and View offices
  • Manage projects
  • Modify questionnaires
  • View the graph for analysis of status report progression
  • Interactive map that shows how the project is going for specific country


Tasks that users can test based on his or her role :
Functionalities.PNG





















5. Action Items
S/N Description Category Status Date Modified
1 To change back map to 3 basic colors instead of 9 UI done 11 October
2 To create 2 separate views for map dashboard for board members (optional) Content done 27 October
3 To change regions to:

1. North America
2. South America
3. Europe
4. Asia Pacific (Includes whole of Asia and Australia)
5. Africa

Content done 10 October
4 To remove Unnecessary Countries: New Cadelona, New Renuion and tiny islands at the top of Europe except for Iceland, Falkland, and Hawaii. Content done 21 October
5 To add Countries:Serbia, Hongkong/ Macao, and Luxembourg. Content done 20 October
6 To cut Russia into half in Secondary View UI done 20 October
7 To change popup to smaller stating that there is "No Project Running Currently" for countries that do not have projects running Content done 12 October
8 To change font color in stacked chart to black instead of grey and font size to be larger. UI done 11 October
9 New Projects happening in Brazil is not shown in Secondary View Chart but it showed up in Primary View Chart (All the countries in South America do not reflect the project no properly) Functionality done 15 October
10 Link between World and Regional Map is not working on IPAD. Functionality done 16 October
11 Heading in PM’s Dashboard should highlight what the bar chart stands for – Missing title UI done 16 October
12 To add Search Filter/ Multi Criteria Bar to filter search results Functionality done 14 October
13 to sort Overall Status of Project correctly. (currently Yellow > Red > Green) Content done 13 October
14 To separate Trend from Overall Status column to enable sorting for trend as well. UI done 16 October
15 To remove the extra cell of Project List in Firefox browser. UI done 11 October
16 To remove IDEA Phase Content done 9 October
17 To change "Past Performance" in chart to "Performance to Date” UI done 15 October
18 To be able to save draft Functionality done 18 October
19 To have a pop up to inform user whether he/she wants to extend the project end date or to close the project if project has past the end date. Functionality ... ...
20 To give explanation of what strategic important is. Content done 15 October
21 To display add comment button for each pillar Functionality done 14 October
22 For Admin to be able to open back closed project Functionality done 15 October
23 To add a dropdown list to select currency. Functionality done 16 October
24 To change the chart to matrix for displaying of trend (optional) Functionality done 27 October
25 To include Read Receipt for Engagement Partner (optional) Functionality ... ...
26 To remove NA from the status Content done 16 October
27 To resolve 'Pillars in Report Page are not sorted properly.' Content done 13 October
28 To add a preview page before submitting of new report. Functionality done 15 October
29 To add “date of submission” for new report submitted. Functionality ... ...
30 To make text in comment box bigger. UI ... ...
31 To be able to download report Functionality done 20 October
32 To disable back button when editing product Functionality ... ...
33 To include email notification Engagement Partner after status report is submitted. Functionality ... ...
34 To remove Office Name field Content done 10 October
35 To change Country drop-down list Auto-complete for Countries Functionalities done 22 October
36 To make the button states "Add Project" instead of the tiny icon UI done 1 November


6. Evaluation of Lodestone End User Test from Testers
1. The color scheme is good.

EndUserTest-Q1.PNG

2. Font size and font type is easy to read.

EndUserTest-Q2.PNG

3. Easy to navigate around.

EndUserTest-Q3.PNG

4. Response time is good. The application is not laggy.

EndUserTest-Q4.PNG

5. All functions are performing as they should be.

EndUserTest-Q5.PNG

6. The outputs shown match the expected result.

EndUserTest-Q6.PNG


7. User Testing Repository
Click here to open Test Case for End User Test
A compiled report of tasks discussed with Lodestone after End User Test End User Test Summary


User Testing 3 (Hallway Test)- 18 September 2012


1. Objective
  • Explore if the display of data for current maps and chart is easily comprehensible/intuitive.
  • Discover unsolved bugs in system.
  • Gather feedbacks and ideas on improving current usability.


2. Description
  • Quick usability feedback on overview of the current prototype across a mix group of users.
  • “A hallway usability test is where you grab the next person that passes by in the hallway and force them to try to use the code you just wrote. If you do this to five people, you will learn 95% of what there is to learn about usability problems in your code.”
Testers
  • 15 students within SMU.
5 are from another FYP team - One Hit Wonder, while the rest are random students we found in concourse. We found out that the other 10 testers are non SIS students.



3. Methods
  • Approach random people with no background knowledge of the project.
  • With a simple description of the application and a few general tasks and scenarios, user will attempt to complete it while exploring the application on his/her own.
  • Bugs and difficulties are recorded by tester.
  • User give feedback base on his/her experience.


4. Tasks Completed
  • Manage users (CRUD of users, including assigning role(s) to users)
  • Add and View offices
  • Manage projects
  • Modify questionnaires
  • View the graph for analysis of status report progression
  • Interactive map that shows how the project is going for specific country


5. Action Items
S/N Description Category Status Date Modified
1 To put story line and detailed steps on how to do certain tasks because testers are not familiar with the projects N.A. - -
2 To provide more intuitive guidance by providing buttons in several pages
  • add back button in 'Add Office'
  • add button to view user in 'Edit User Page'
User Interface done 22 September
3 To debug project tab(error 404) Functional done 21 September
4 To change date to calendar view instead of manually typing User Interface done 22 September
5 To save questionnaire answer (currently if at least 1 questionnaire was not filled in, all the answers will be deleted and tester will need to fill in again the answer he/she previously keyed in) Functional done 23 September
6 To put clearly which role user is having when they first signed in, it is not really obvious currently. Content done 12 October
7 System give null when Australia is chosen for add office. Functional done 22 September


6. User Testing Repository
to check the test case , click on Hallway Test Case


User Testing 2 (Paper Prototype) - 31 August and 3 September 2012

click on Fortune Cookies: Paper Prototype Result

IMG 2656 - Copy.JPG


1. Objective
  • Propose new and fresh ideas by exploring more layout design and functionalities to be implemented
  • Get the preferable layout chosen by client



2. Description
Paper prototype of improvements to the system is used to clarify requirements and enable draft interaction designs and screen designs to be very rapidly simulated and tested. Users are given the prototype to test the functionalities and usability of the whole system without any guidance. Reactions and doubts are recorded through video to note down the areas of improvements to be made in regards to usability and ease of use. Potential usability problems can be detected at a very early stage in the design process before any code has been written and User will get the feel of the comprehensive system and revise the requirements at an early stage.


Testers :
  • Michiel Roosjen
  • Rachel Tann
Michiel (Director and PM) and Rachel (Admin) are the Person In Charge of the project that we are currently running. Therefore, two of them represented the Lodestone team to have a basic sense, especially the layout of our application.



3. Methods
  • Users are given a list of functionalities supported as discussed.
  • Presented with the paper prototype, users will attempt to test the functionality/complete the tasks without any guidance.
  • Tester will either voice out the interface the user should expect to see or add in prepared prototypes on other screenshots.
  • Mark on the prototype where a user attempted to “click” or otherwise interact with the interface



4. Action Items
S/N Description Category Status Date Modified
1 To delete the status (red, yellow, or green bubble) when PM is filling in questionnaire. Content done 9 September
2 to add more trigger to prompt user to answer questionnaire Content mitigated -
3 to give "warning" to user before submitting the report Content ... ...
4 to be able to generate report in Power Point Functionality done 20 October
5 to put the task list for project manager (the task only gets removed once the action has been completed), which will be put on top of the list of projects Functionality ... ...
6 to add on option for turn on and off notification via email Functionality ... ...
7 to implement the ascending sorting by 1st click and descending sorting by 2nd click Functionality done 20 September
8 to have filter function, based on traffic light / status they are looking for (e.g. red) Functionality done 21 September
9 to allow PM to edit project name Functionality done 8 September
10 to remove the world map from edit region (only table left) User Interface N.A. N.A.
11 to add back button from the individual project management page Functionality done 21 September
12 to change the display for engagement partner. It will start with clicking on continent to country (and filter based on the status they want) and list of projects Content done 14 September
13 to put the legend in the map to determine the severity and ease of viewing for partner Content done 15 September
14 to include archives for past projects for all roles Content mitigated -


5. Summary

For paper prototype testing, all the action items are either functionality or content.
The team now is able to have a clearer view of what functionalities and contents that Lodestone wants for its Delivery Excellence Dashboard.

Paper prototype summary.PNG



6. User Testing Repository
Paper Prototype Minutes 01 - 31 August
Paper Prototype Minutes 02 - 3 Sept (idem with external meeting minutes 08)


User Testing 1 (Heuristic Evaluation) - 1 August 2012

1. Objective
  • to allow user to use early prototype product with the brief initial design of the flow.
  • to match between client and Fortune Cookies' phrasing of words, contents, user interface, and functionalities in Delivery Excellence Dashboard.


2. Description
Heuristic Evaluation gives users chance to experience the initial design of the application to be able to learn and discover about its functionalities, contents, etc.
Users will then give feedback which will be very helpful in term of understanding the strengths and weaknesses that current system has.
Tester:
  • Michiel Roosjen



3. Methods
  • listing the metric that users should be able to get , such as consistency, flexibility, and simplicity of design
  • putting details on what users can expect on each metric
  • ask user to give a scale and feedback for each metric
  • create an action items list for us to work on


4. Action Items
S/N Description Category Status Date Modified
1 to be able to sort item Functionality done 20 September
2 to grey out past project (only show current fiscal year) User Interface mitigated -
3 to show history of previous years (according to manual close of project) Content mitigated -
4 to have revenue and strategic importance to measure the level(high, medium, low) of project Content done 26 August
5 for admin to be able to arrange functions according to frequency of use Functionality done 28 August
6 to have onsite checking of valid email and usernames Functionality done 7 September
7 to order messages from top-down Functionality done 20 September
8 to highlight error fields Content done 7 September


5. Summary
For Heuristic Evaluation, we need to work on functionalities and contents displayed by gathering more information and doing another user testing to find what client really wants.
Heuristic evaluation summary.PNG



6. User Testing Repository
Heuristic Report 01 - 1 August