Difference between revisions of "2012T1 Fortune Cookies: User Testing"
Jump to navigation
Jump to search
Line 108: | Line 108: | ||
<br> | <br> | ||
+ | <div style="font-size:14px;">'''5. Details of Usability Test Result '''</div> | ||
+ | [[Image:UT2 ResultOverview.JPG|User Test 1 & User Test 2 Overview]] | ||
===<div style="font-family:'Century Gothic'; font-size:20px; background: #b20004; padding:14px; font-weight:bold; line-height:0.1em;"><font color = "white">Usability Test 1 - 30 October 2012</font></div>=== | ===<div style="font-family:'Century Gothic'; font-size:20px; background: #b20004; padding:14px; font-weight:bold; line-height:0.1em;"><font color = "white">Usability Test 1 - 30 October 2012</font></div>=== |
Revision as of 10:25, 19 November 2012
Test Reports
Dashboard | Team & Project Overview | Project Design | Project Management | Progress Summary | Test Reports |
Category | Heuristic Evaluation | Paper Prototype | Hallway Test | Lodestone End User Test | Usability Test 1 | Usability & Field Test | UAT | Regression Test |
Contents
- 1 Category
- 2 User Acceptance Test
- 3 Usability and Field Test - 9 November 2012
- 4 Usability Test 1 - 30 October 2012
- 5 Regression Test
- 6 User Testing 4 (Lodestone End User Test) - 25 September 2012
- 7 User Testing 3 (Hallway Test)- 18 September 2012
- 8 User Testing 2 (Paper Prototype) - 31 August and 3 September 2012
- 9 User Testing 1 (Heuristic Evaluation) - 1 August 2012
Category
- content -- give user what they expect to get
- functionality - making sure specific function is working as it should be without giving any error.
- navigation -- showing user the correct information after he / she chooses specific path
- speed -- responding time or loading time needed
- user interface :
- layout -- where and how information is shown
- design -- colors and shape displayed
We aim to reach our XFactor by doing these :
User Acceptance Test
1. Objective
- To obtain confirmation that a system meets mutually agreed-upon requirements.
2. Executive Summary
- A final verification of the required business function and proper functioning of the system, emulating real-world usage conditions
Usability and Field Test - 9 November 2012
1. Objective
- To determine design inconsistencies and usability problem areas within the user interface and content areas. Potential sources of error may include:
- Navigation errors – failure to locate functions, excessive keystrokes to complete a function, failure to follow recommended screen flow.
- Presentation errors – failure to locate and properly act upon desired information in screens, selection errors due to labelling ambiguities.
- Control usage problems – improper toolbar or entry field usage.
- Navigation errors – failure to locate functions, excessive keystrokes to complete a function, failure to follow recommended screen flow.
- Exercise the application under controlled test conditions with representative users. Data will be used to access whether usability goals regarding an effective, efficient, and well-received user interface have been achieved.
- Establish functional performance of application under actual field usage.
2. Executive Summary
- In order to measure the quality and level of intuitiveness of the new Delivery Excellence Dashboard, a series of usability tests are conducted with suitable volunteers of similar demographics and background, as well as staffs from Lodestone Management Consultants. The test will simulate the response, reaction and usage of users that are new to the new application and framework. Users will test the application under given scenarios that replicate their job responsibilities. Test researchers will note down usability points during the test through observation of users’ feedbacks and habits. A trace of navigation habits and time taken by the user is also logged and will be analysed to derive to quantify each usability goal. Users will also be directed to explore the application after the test base on their usage of the application in their job scope and feedback on their experience.
3. Participants
- Participants will be members from Lodestone Management Consultant. The participants are a managing consultant taking a role of project manager in the application, an associate partner taking the role of country lead/board member and the administrator of the application herself.
- The participants' responsibilities will be to attempt to complete a set of representative task scenarios presented to them in as efficient and timely a manner as possible, and to provide feedback regarding the usability and acceptability of the user interface. The participants will be directed to provide honest opinions regarding the usability of the application, and to participate in post-session subjective questionnaires and debriefing.
5. Details of Usability Test Result
Usability Test 1 - 30 October 2012
1. Objective
- To determine design inconsistencies and usability problem areas within the user interface and content areas. Potential sources of error may include:
- Navigation errors – failure to locate functions, excessive keystrokes to complete a function, failure to follow recommended screen flow.
- Presentation errors – failure to locate and properly act upon desired information in screens, selection errors due to labelling ambiguities.
- Control usage problems – improper toolbar or entry field usage.
- Navigation errors – failure to locate functions, excessive keystrokes to complete a function, failure to follow recommended screen flow.
- Exercise the application under controlled test conditions with representative users. Data will be used to access whether usability goals regarding an effective, efficient, and well-received user interface have been achieved.
- Establish baseline user performance and user-satisfaction levels of the user interface for future usability evaluations.
2. Executive Summary
- To simulate product launch and the response of target users to the new application, Delivery Excellence Dashboard. We will engage Professors with prior experience in project management to test the application under given scenarios. Each user will take up a specific role and test out the scope of functionality designed for that user role. Test researchers will note down usability points during the test. A trace of navigation habits and time taken by the user is also logged and will be analysed to derive to quantify each usability goal.
3. Participants
- Subjects will be Professors with demographics of 28 – 40 years old with background in project management.
- The participants' responsibilities will be to attempt to complete a set of representative task scenarios presented to them in as efficient and timely a manner as possible, and to provide feedback regarding the usability and acceptability of the user interface. The participants will be directed to provide honest opinions regarding the usability of the application, and to participate in post-session subjective questionnaires and debriefing.
4. Method:
- 1. The participants will receive and overview of the usability test procedure, scenario and role taken, equipment and software.
- Project Manager Background scenario: “You are a Project Manager at Lodestone Management Consultant based in Singapore, managing various project in the region. Your country lead (Stephen Wise) introduces a new quality assurance tool that requires you to complete a questionnaire for each of the project you are managing. Stephen also says that the tool will be helpful to give you an insight about the different aspects of your project.”
- Administrator Background scenario: “You are an Administrator at Lodestone Management Consultant based in Singapore, tasked to support the running of different applications in the office. Your country lead (Stephen Wise) introduces a new quality assurance tool that requires you to support the various colleagues in your office and manage any system changes according to changes to Lodestone MC’s business rules. Project Managers in your office will also be contacting you to help them create entities of the projects they are managing into the system.”
- Board Member Background scenario: “You are a Board Member at Lodestone Management Consultant based in Singapore, tasked to support the running of different applications in the office. Your fellow Board Member (Stephen Wise) introduces a new quality assurance and monitoring tool that will help you monitors the global progress of Lodestone MC’s projects. You have been wondering how projects in different regions are performing and this tool might prove to be a great help for you to analyse that.”
- 2. User is given his account credential and the will be allowed to try out the application for 3 minutes.
- 3. Based on the given scenario, user will attempt to complete their task within stipulated time.
5. Tasks Completed
- Manage users (CRUD of users, including assigning role(s) to users)
- Add and View offices
- Manage projects
- Manage regions
- Modify questionnaires
- View the graph for analysis of status report progression
- Interactive map that shows how the project is going for specific country
- Download report
- Filter function
6. Action Items
S/N | Description | Category | Status | Date Modified |
---|---|---|---|---|
1 | To edit the date picker to be easily changed from year to year without much clicks | Functionality | done | 7 November |
2 | To synchronize the naming used | Content | done | 8 November |
3 | To change toolbox position and content | Content | done | 8 November |
4 | To put the project description | Functionalities | ... | ... |
7. Evaluation from Testers / Professors
- Layout of Lodestone DxD
- What do you like about Lodestone DxD ?
- What do you NOT like about Lodestone DxD ?
8. Details of Usability Test Result
Regression Test
User Testing 4 (Lodestone End User Test) - 25 September 2012
1. Objective
- To make sure that all agreeable functionalities between client and us are in the system
- To verify that all functions work properly
- Ensure that end users are able to perform their designated job functions with the new system
- Better understand the alignment of objectives and goals for each category of stakeholders
- Gather feedback on the quality and preference of data visualisation
2. Description
- End User Test is used to test whether all functionalities in the system work as they planned. By testing within the boundaries of work or task flows, End User test will be able to check if the intended output is available for user based on specific input given.
- All users except Admin, are given sample questions of what they will need to use the application for, such as :
- How is the project in Singapore going?
- Is there any improvement from the previous report?
- This test will involve 5 stakeholders who have different roles, which are :
- Stephen Wise - Engagement Partner
- Rui Imamura - Project Manager
- Ben Inglish - Project Manager
- ParameshwaranKN - Project Manager
- Rachel Tann - Admin
3. Methods
- Users will be given the test case for specific role assigned.
- At the start of the test, we gave users 10 minutes to explore the application.
- Given the test case, users will follow the scenario, and put Pass / Fail in the column. Additional comments can also be added in by users.
4. Tasks Completed
- Manage users (CRUD of users, including assigning role(s) to users)
- Add and View offices
- Manage projects
- Modify questionnaires
- View the graph for analysis of status report progression
- Interactive map that shows how the project is going for specific country
- Tasks that users can test based on his or her role :
5. Action Items
S/N | Description | Category | Status | Date Modified |
---|---|---|---|---|
1 | To change back map to 3 basic colors instead of 9 | UI | done | 11 October |
2 | To create 2 separate views for map dashboard for board members (optional) | Content | done | 27 October |
3 | To change regions to: 1. North America |
Content | done | 10 October |
4 | To remove Unnecessary Countries: New Cadelona, New Renuion and tiny islands at the top of Europe except for Iceland, Falkland, and Hawaii. | Content | done | 21 October |
5 | To add Countries:Serbia, Hongkong/ Macao, and Luxembourg. | Content | done | 20 October |
6 | To cut Russia into half in Secondary View | UI | done | 20 October |
7 | To change popup to smaller stating that there is "No Project Running Currently" for countries that do not have projects running | Content | done | 12 October |
8 | To change font color in stacked chart to black instead of grey and font size to be larger. | UI | done | 11 October |
9 | New Projects happening in Brazil is not shown in Secondary View Chart but it showed up in Primary View Chart (All the countries in South America do not reflect the project no properly) | Functionality | done | 15 October |
10 | Link between World and Regional Map is not working on IPAD. | Functionality | done | 16 October |
11 | Heading in PM’s Dashboard should highlight what the bar chart stands for – Missing title | UI | done | 16 October |
12 | To add Search Filter/ Multi Criteria Bar to filter search results | Functionality | done | 14 October |
13 | to sort Overall Status of Project correctly. (currently Yellow > Red > Green) | Content | done | 13 October |
14 | To separate Trend from Overall Status column to enable sorting for trend as well. | UI | done | 16 October |
15 | To remove the extra cell of Project List in Firefox browser. | UI | done | 11 October |
16 | To remove IDEA Phase | Content | done | 9 October |
17 | To change "Past Performance" in chart to "Performance to Date” | UI | done | 15 October |
18 | To be able to save draft | Functionality | done | 18 October |
19 | To have a pop up to inform user whether he/she wants to extend the project end date or to close the project if project has past the end date. | Functionality | ... | ... |
20 | To give explanation of what strategic important is. | Content | done | 15 October |
21 | To display add comment button for each pillar | Functionality | done | 14 October |
22 | For Admin to be able to open back closed project | Functionality | done | 15 October |
23 | To add a dropdown list to select currency. | Functionality | done | 16 October |
24 | To change the chart to matrix for displaying of trend (optional) | Functionality | done | 27 October |
25 | To include Read Receipt for Engagement Partner (optional) | Functionality | ... | ... |
26 | To remove NA from the status | Content | done | 16 October |
27 | To resolve 'Pillars in Report Page are not sorted properly.' | Content | done | 13 October |
28 | To add a preview page before submitting of new report. | Functionality | done | 15 October |
29 | To add “date of submission” for new report submitted. | Functionality | ... | ... |
30 | To make text in comment box bigger. | UI | ... | ... |
31 | To be able to download report | Functionality | done | 20 October |
32 | To disable back button when editing product | Functionality | ... | ... |
33 | To include email notification Engagement Partner after status report is submitted. | Functionality | ... | ... |
34 | To remove Office Name field | Content | done | 10 October |
35 | To change Country drop-down list Auto-complete for Countries | Functionalities | done | 22 October |
36 | To make the button states "Add Project" instead of the tiny icon | UI | done | 1 November |
6. Evaluation of Lodestone End User Test from Testers
1. The color scheme is good. |
2. Font size and font type is easy to read. |
3. Easy to navigate around. |
4. Response time is good. The application is not laggy. |
5. All functions are performing as they should be. |
6. The outputs shown match the expected result. |
7. User Testing Repository
- Click here to open Test Case for End User Test
- A compiled report of tasks discussed with Lodestone after End User Test End User Test Summary
User Testing 3 (Hallway Test)- 18 September 2012
1. Objective
- Explore if the display of data for current maps and chart is easily comprehensible/intuitive.
- Discover unsolved bugs in system.
- Gather feedbacks and ideas on improving current usability.
2. Description
- Quick usability feedback on overview of the current prototype across a mix group of users.
- “A hallway usability test is where you grab the next person that passes by in the hallway and force them to try to use the code you just wrote. If you do this to five people, you will learn 95% of what there is to learn about usability problems in your code.”
- Testers
- 15 students within SMU.
- 5 are from another FYP team - One Hit Wonder, while the rest are random students we found in concourse. We found out that the other 10 testers are non SIS students.
3. Methods
- Approach random people with no background knowledge of the project.
- With a simple description of the application and a few general tasks and scenarios, user will attempt to complete it while exploring the application on his/her own.
- Bugs and difficulties are recorded by tester.
- User give feedback base on his/her experience.
4. Tasks Completed
- Manage users (CRUD of users, including assigning role(s) to users)
- Add and View offices
- Manage projects
- Modify questionnaires
- View the graph for analysis of status report progression
- Interactive map that shows how the project is going for specific country
5. Action Items
S/N | Description | Category | Status | Date Modified |
---|---|---|---|---|
1 | To put story line and detailed steps on how to do certain tasks because testers are not familiar with the projects | N.A. | - | - |
2 | To provide more intuitive guidance by providing buttons in several pages
|
User Interface | done | 22 September |
3 | To debug project tab(error 404) | Functional | done | 21 September |
4 | To change date to calendar view instead of manually typing | User Interface | done | 22 September |
5 | To save questionnaire answer (currently if at least 1 questionnaire was not filled in, all the answers will be deleted and tester will need to fill in again the answer he/she previously keyed in) | Functional | done | 23 September |
6 | To put clearly which role user is having when they first signed in, it is not really obvious currently. | Content | done | 12 October |
7 | System give null when Australia is chosen for add office. | Functional | done | 22 September |
6. User Testing Repository
- to check the test case , click on Hallway Test Case
User Testing 2 (Paper Prototype) - 31 August and 3 September 2012
click on Fortune Cookies: Paper Prototype Result
1. Objective
- Propose new and fresh ideas by exploring more layout design and functionalities to be implemented
- Get the preferable layout chosen by client
2. Description
- Paper prototype of improvements to the system is used to clarify requirements and enable draft interaction designs and screen designs to be very rapidly simulated and tested. Users are given the prototype to test the functionalities and usability of the whole system without any guidance. Reactions and doubts are recorded through video to note down the areas of improvements to be made in regards to usability and ease of use. Potential usability problems can be detected at a very early stage in the design process before any code has been written and User will get the feel of the comprehensive system and revise the requirements at an early stage.
- Testers :
- Michiel Roosjen
- Rachel Tann
- Michiel (Director and PM) and Rachel (Admin) are the Person In Charge of the project that we are currently running. Therefore, two of them represented the Lodestone team to have a basic sense, especially the layout of our application.
3. Methods
- Users are given a list of functionalities supported as discussed.
- Presented with the paper prototype, users will attempt to test the functionality/complete the tasks without any guidance.
- Tester will either voice out the interface the user should expect to see or add in prepared prototypes on other screenshots.
- Mark on the prototype where a user attempted to “click” or otherwise interact with the interface
4. Action Items
S/N | Description | Category | Status | Date Modified |
---|---|---|---|---|
1 | To delete the status (red, yellow, or green bubble) when PM is filling in questionnaire. | Content | done | 9 September |
2 | to add more trigger to prompt user to answer questionnaire | Content | mitigated | - |
3 | to give "warning" to user before submitting the report | Content | ... | ... |
4 | to be able to generate report in Power Point | Functionality | done | 20 October |
5 | to put the task list for project manager (the task only gets removed once the action has been completed), which will be put on top of the list of projects | Functionality | ... | ... |
6 | to add on option for turn on and off notification via email | Functionality | ... | ... |
7 | to implement the ascending sorting by 1st click and descending sorting by 2nd click | Functionality | done | 20 September |
8 | to have filter function, based on traffic light / status they are looking for (e.g. red) | Functionality | done | 21 September |
9 | to allow PM to edit project name | Functionality | done | 8 September |
10 | to remove the world map from edit region (only table left) | User Interface | N.A. | N.A. |
11 | to add back button from the individual project management page | Functionality | done | 21 September |
12 | to change the display for engagement partner. It will start with clicking on continent to country (and filter based on the status they want) and list of projects | Content | done | 14 September |
13 | to put the legend in the map to determine the severity and ease of viewing for partner | Content | done | 15 September |
14 | to include archives for past projects for all roles | Content | mitigated | - |
5. Summary
- For paper prototype testing, all the action items are either functionality or content.
- The team now is able to have a clearer view of what functionalities and contents that Lodestone wants for its Delivery Excellence Dashboard.
6. User Testing Repository
- Paper Prototype Minutes 01 - 31 August
- Paper Prototype Minutes 02 - 3 Sept (idem with external meeting minutes 08)
User Testing 1 (Heuristic Evaluation) - 1 August 2012
1. Objective
- to allow user to use early prototype product with the brief initial design of the flow.
- to match between client and Fortune Cookies' phrasing of words, contents, user interface, and functionalities in Delivery Excellence Dashboard.
2. Description
- Heuristic Evaluation gives users chance to experience the initial design of the application to be able to learn and discover about its functionalities, contents, etc.
- Users will then give feedback which will be very helpful in term of understanding the strengths and weaknesses that current system has.
- Tester:
- Michiel Roosjen
3. Methods
- listing the metric that users should be able to get , such as consistency, flexibility, and simplicity of design
- putting details on what users can expect on each metric
- ask user to give a scale and feedback for each metric
- create an action items list for us to work on
4. Action Items
S/N | Description | Category | Status | Date Modified |
---|---|---|---|---|
1 | to be able to sort item | Functionality | done | 20 September |
2 | to grey out past project (only show current fiscal year) | User Interface | mitigated | - |
3 | to show history of previous years (according to manual close of project) | Content | mitigated | - |
4 | to have revenue and strategic importance to measure the level(high, medium, low) of project | Content | done | 26 August |
5 | for admin to be able to arrange functions according to frequency of use | Functionality | done | 28 August |
6 | to have onsite checking of valid email and usernames | Functionality | done | 7 September |
7 | to order messages from top-down | Functionality | done | 20 September |
8 | to highlight error fields | Content | done | 7 September |
5. Summary
- For Heuristic Evaluation, we need to work on functionalities and contents displayed by gathering more information and doing another user testing to find what client really wants.
6. User Testing Repository