2012T1 Fortune Cookies: User Testings

From IS480
Jump to navigation Jump to search

 User Testings

Click here to go back to Fortune Cookies Progress Summary.

User Testing 1 (Heuristic Evaluation) - 1 August 2012

1. Objective
  • to allow user to use early prototype product with the brief initial design of the flow.
  • to match between client and Fortune Cookies' phrasing of words, contents, user interface, and functionalities in Delivery Excellence Dashboard.

2. Description

Heuristic Evaluation gives users chance to experience the initial design of the application to be able to learn and discover about its functionalities, contents, etc. Users will then give feedback which will be very helpful in term of understanding the strengths and weaknesses that current system has.

  • Michiel Roosjen

3. Methods
  • listing the metric that users should be able to get , such as consistency, flexibility, and simplicity of design
  • putting details on what users can expect on each metric
  • ask user to give a scale and feedback for each metric
  • create an action items list for us to work on

4. Action Items
S/N Description Category Status Date Modified
1 to be able to sort item Functionality done 20 September
2 to grey out past project (only show current fiscal year) User Interface mitigated -
3 to show history of previous years (according to manual close of project) Content mitigated -
4 to have revenue and strategic importance to measure the level(high, medium, low) of project Content ... ...
5 for admin to be able to arrange functions according to frequency of use Functionality done 28 August
6 to have onsite checking of valid email and usernames Functionality done 7 September
7 to order messages from top-down Functionality done 20 September
8 to highlight error fields Content done 7 September

5. Summary

For Heuristic Evaluation, we need to work on functionalities and contents displayed by gathering more information and doing another user testing to find what client really wants. Heuristic evaluation summary.PNG

6. User Testing Repository

Heuristic Report 01 - 1 August

User Testing 2 (Paper Prototype) - 31 August and 3 September 2012

click on Fortune Cookies: Paper Prototype Result

IMG 2656 - Copy.JPG

1. Objective
  • Propose new and fresh ideas by exploring more layout design and functionalities to be implemented
  • Get the preferable layout chosen by client

2. Description

Paper prototype of improvements to the system is used to clarify requirements and enable draft interaction designs and screen designs to be very rapidly simulated and tested. Users are given the prototype to test the functionalities and usability of the whole system without any guidance. Reactions and doubts are recorded through video to note down the areas of improvements to be made in regards to usability and ease of use. Potential usability problems can be detected at a very early stage in the design process before any code has been written and User will get the feel of the comprehensive system and revise the requirements at an early stage.
Testers :

  • Michiel Roosjen
  • Rachel Tann

3. Methods
  • Users are given a list of functionalities supported as discussed.
  • Presented with the paper prototype, users will attempt to test the functionality/complete the tasks without any guidance.
  • Tester will either voice out the interface the user should expect to see or add in prepared prototypes on other screenshots.
  • Mark on the prototype where a user attempted to “click” or otherwise interact with the interface

4. Action Items
S/N Description Category Status Date Modified
1 To delete the status (red, yellow, or green bubble) when PM is filling in questionnaire. Content done 9 September
2 to add more trigger to prompt user to answer questionnaire Content mitigated -
3 to give "warning" to user before submitting the report Content ... ...
4 to be able to generate report in Power Point Functionality ... ...
5 to put the task list for project manager (the task only gets removed once the action has been completed), which will be put on top of the list of projects Functionality ... ...
6 to add on option for turn on and off notification via email Functionality ... ...
7 to implement the ascending sorting by 1st click and descending sorting by 2nd click Functionality done 20 September
8 to have filter function, based on traffic light / status they are looking for (e.g. red) Functionality done 21 September
9 to allow PM to edit project name Functionality done 8 September
10 to remove the world map from edit region (only table left) User Interface N.A. N.A.
11 to add back button from the individual project management page Functionality done 21 September
12 to change the display for engagement partner. It will start with clicking on continent to country (and filter based on the status they want) and list of projects Content done 14 September
13 to put the legend in the map to determine the severity and ease of viewing for partner Content done 15 September
14 to include archives for past projects for all roles Content mitigated -

5. Summary

For paper prototype testing, all the action items are either functionality or content.
The team now is able to have a clearer view of what functionalities and contents that Lodestone wants for its Delivery Excellence Dashboard.

Paper prototype summary.PNG

6. User Testing Repository

Paper Prototype Minutes 01 - 31 August
Paper Prototype Minutes 02 - 3 Sept (idem with external meeting minutes 08)

User Testing 3 (Hallway Test)- 18 September 2012

to check the test case , click on Hallway Test Case

1. Objective
  • Explore if the display of data for current maps and chart is easily comprehensible/intuitive.
  • Discover unsolved bugs in system.
  • Gather feedbacks and ideas on improving current usability.

2. Description
  • Quick usability feedback on overview of the current prototype across a mix group of users.
  • “A hallway usability test is where you grab the next person that passes by in the hallway and force them to try to use the code you just wrote. If you do this to five people, you will learn 95% of what there is to learn about usability problems in your code.”


  • 15 students within SMU

3. Methods
  • Approach random people with no background knowledge of the project.
  • With a simple description of the application and a few general tasks and scenarios, user will attempt to complete it while exploring the application on his/her own.
  • Bugs and difficulties are recorded by tester.
  • User give feedback base on his/her experience.

4. Action Items
S/N Description Category Status Date Modified
1 To put story line and detailed steps on how to do certain tasks because testers are not familiar with the projects N.A. - -
2 To provide more intuitive guidance by providing buttons in several pages
  • add back button in 'Add Office'
  • add button to view user in 'Edit User Page'
User Interface done 22 September
3 To debug project tab(error 404) Functional done 21 September
4 To change date to calendar view instead of manually typing User Interface done 22 September
5 To save questionnaire answer (currently if at least 1 questionnaire was not filled in, all the answers will be deleted and tester will need to fill in again the answer he/she previously keyed in) Functional done 23 September
6 To put clearly which role user is having when they first signed in, it is not really obvious currently. Content ... ...
7 System give null when Australia is chosen for add office. Functional done 22 September

5. Summary

Hallway test.PNG

User Testing 4 (Lodestone End User Test) - 25 September 2012

1. Objective
  • To make sure that all agreeable functionalities between client and us are in the system
  • To verify that all functions work properly
  • Ensure that end users are able to perform their designated job functions with the new system
  • Better understand the alignment of objectives and goals for each category of stakeholders
  • Gather feedback on the quality and preference of data visualisation

2. Description

End User Test is used to test whether all functionalities in the system work as they planned. By testing within the boundaries of work or task flows, End User test will be able to check if the intended output is available for user based on specific input given.
This test will involve 5 stakeholders who have different roles, which are :

  • Stephen Wise - Engagement Partner
  • Rui Imamura - Project Manager
  • Ben Inglish - Project Manager
  • ParameshwaranKN - Project Manager
  • Rachel Tann - Admin

3. Methods
  • Users will be given the test case for specific role assigned.
  • Given the test case, users will follow the scenario, and put Pass / Fail in the column. Additional comments can also be added in by users.

4. Action Items
S/N Description Category Status Date Modified
1 To change back map to 3 basic colors instead of 9 UI ... ...
2 To create 2 separate views for map dashboard for board members (optional) Content ... ...
3 To change regions to:

1. North America
2. South America
3. Europe
4. Asia Pacific (Includes whole of Asia and Australia)
5. Africa

Content ... ...
4 To remove Unnecessary Countries: New Cadelona, New Renuion and tiny islands at the top of Europe except for Iceland, Falkland, and Hawaii. Content ... ...
5 To add Countries:Serbia, Hongkong/ Macao, and Luxembourg. Content ... ...
6 To cut Russia into half in Secondary View UI ... ...
7 To change popup to smaller stating that there is "No Project Running Currently" for countries that do not have projects running Content ... ...
8 To change font color in stacked chart to black instead of grey and font size to be larger. UI ... ...
9 New Projects happening in Brazil is not shown in Secondary View Chart but it showed up in Primary View Chart (All the countries in South America do not reflect the project no properly) Functionality ... ...
10 Link between World and Regional Map is not working on IPAD. Functionality ... ...
11 Heading in PM’s Dashboard should highlight what the bar chart stands for – Missing title UI ... ...
12 To add Search Filter/ Multi Criteria Bar to filter search results Functionality ... ...
13 to sort Overall Status of Project correctly. (currently Yellow > Red > Green) Content ... ...
14 To separate Trend from Overall Status column to enable sorting for trend as well. UI ... ...
15 To remove the extra cell of Project List in Firefox browser. UI ... ...
16 To remove IDEA Phase Content ... ...
17 To change "Past Performance" in chart to "Performance to Date” UI ... ...
18 To be able to save draft Functionality ... ...
19 To have a pop up to inform user whether he/she wants to extend the project end date or to close the project if project has past the end date. Functionality ... ...
20 To give explanation of what strategic important is. Content ... ...
21 To display add comment button for each pillar Functionality ... ...
22 For Admin to be able to open back closed project Functionality ... ...
23 To add a dropdown list to select currency. Functionality ... ...
24 To change the chart to matrix for displaying of trend (optional) Functionality ... ...
25 To include Read Receipt for Engagement Partner (optional) Functionality ... ...
26 To remove NA from the status Content ... ...
27 To resolve 'Pillars in Report Page are not sorted properly.' Content ... ...
28 To add a preview page before submitting of new report. Functionality ... ...
29 To add “date of submission” for new report submitted. Functionality ... ...
30 To make text in comment box bigger. UI ... ...
31 To be able to download report Functionality ... ...
32 To disable back button when editing product Functionality ... ...
33 To include email notification Engagement Partner after status report is submitted. Functionality ... ...
34 To remove Office Name field Content ... ...
35 To change Country drop-down list Auto-complete for Countries Functionalities ... ...
36 To make the button states "Add Project" instead of the tiny icon UI ... ...

5. Summary

End user test summary.PNG

6. User Testing Repository

Click here to open Test Case for End User Test
A compiled report of tasks discussed with Lodestone after End User Test End User Test Summary

User Testing 5 (End To End Test) - TBC

1. Objective
  • to verify functional, performance, and reliability requirements of our application.
  • to make sure that all components within the application interact correctly.

2. Description

End To End Test is similar to integration test ; this occurs after unit testing. Users, based on their roles, will be given a test case with a specific input and expected output given. Our End To End Test will use a Big Bang approach which means that this test will cover most of the developed modules that linked together.

3. Methods
  • ...
  • ...

4. Action Items
S/N Description Category Status Date Modified
1 ... ... ... ...
2 ... ... ... ...

5. Summary


6. User Testing Repository