Difference between revisions of "IS480 Team wiki: 2013T2 D'PENZ User Testing"
(→Graph) |
|||
(12 intermediate revisions by the same user not shown) | |||
Line 553: | Line 553: | ||
|| [[Image:RowStyle.PNG|center|thumb|450px|Row style of selecting suppliers. (3 VOTES)]] | || [[Image:RowStyle.PNG|center|thumb|450px|Row style of selecting suppliers. (3 VOTES)]] | ||
|} | |} | ||
− | |||
====Horizontal-Vertical Tabs==== | ====Horizontal-Vertical Tabs==== | ||
Line 562: | Line 561: | ||
|| [[Image:VerticalTabs.PNG|center|thumb|450px|This was an experimental choice to determine if users are receptive to newer designs. (1 VOTE)]] | || [[Image:VerticalTabs.PNG|center|thumb|450px|This was an experimental choice to determine if users are receptive to newer designs. (1 VOTE)]] | ||
|} | |} | ||
+ | |||
+ | == <font face="Droid Sans" size="+3" color="#ff9a00">Overall Timings</font> == | ||
+ | |||
+ | ===Raw Data=== | ||
+ | <i>Timings are in seconds</i><br> | ||
+ | [[Image:RawData.PNG]] | ||
+ | <br> | ||
+ | <br> | ||
+ | |||
+ | ===Graph=== | ||
+ | <br> | ||
+ | [[Image:OverallTimingUT.PNG]] | ||
+ | <br> | ||
+ | <br> | ||
+ | |||
+ | ===% Improvement=== | ||
+ | [[Image:%Improvement.PNG]] | ||
+ | <br> | ||
== <font face="Droid Sans" size="+3" color="#ff9a00">Conclusion</font> == | == <font face="Droid Sans" size="+3" color="#ff9a00">Conclusion</font> == | ||
− | + | In conclusion, the overall timings have improved and hence we have been steadily viewing an improvement in the usability. The key limitation, however, is that we are limited to detecting large differences between designs or measures. The reason is that majority of our tests do not possess a significant number of participants. However we still believe that collecting data would aid us in providing guidance on which features are need improvement. Additionally we have tried to mitigate the lack of users by getting representative users (e.g. procurement staff) to provide guidance. |
Latest revision as of 23:17, 21 April 2014
Home | Our Team | Project Overview | Project Management | Project Documentation |
Use Cases | Architectural & Deployment Diagram | Prototypes | User Testing | Performance Testing | Presentation Slides | Meeting Minutes | Handover Documents |
Contents
Heuristic Evaluation
Note: This was done on our system coded in JSP.
Date: 22 October 2013
Purpose: To test if our system is intuitive enough to be used by people who have not used procurement systems before.
Results from HE
Round-up of Evaluation
- Overall, the app is easy to use and navigate around.
- More instructions could be provided so that users know what to do.
- The user interface is simple, which is well-liked by testers.
Changes to be implemented
- Instead of displaying search results under Other Projects section (where users still have to scroll down), we will place the Search Results in the same tab as the Search fields (Change implemented)
- We will include a help function to aid new users in navigating around the system (Change will be implemented after acceptance)
- "Save Edits" button in the project details page will be placed both at the top and bottom of the page (Change will be implemented after acceptance)
User Testing 1
Note: This is the first testing we have done with on our system coded with ASP.NET.
Date: 29 October 2013
Purpose: Evaluate usability of functions: Home page, Search project, View & Edit Project Details
Time: 5pm-6pm
User: Tam Chee Hong
Location: Skype (Screen sharing with our client as we are still using internal IP)
Browser: Google Chrome
Role: Procurement staff
Comments from UT1
S/N | Function | Description | Suggestion |
---|---|---|---|
1 | Search Projects | There are too many functions in one page (Home page) which leads to a lot of scrolling. Also, search function may not be used often since I can already see the projects that I'm working on. | Remove the search section and replace it with a search bar at the top. Search results could be displayed on a different page. |
2 | View Project Details | It is not clear which fields can or cannot be edited. | Fields that cannot be edited should be greyed out. |
3 | View Project Details | The start date is in a format like 01/03/2013, and it is confusing as it is not clear which is the month and which is the date. | Spell out the month in full. |
4 | View Project Details | Projects are submitted from various countries, hence it is important to know the currency of the project budget. | Indicate the currency of the project budget. |
5 | Home Page | There should be a toggle button to see all projects. | Add a toggle button. |
6 | Audit Log | It's funny that the audit log has it's own page when it should be for specific projects | Shift audit log button to view project details page. |
7 | Add New User | It will be good to have a function to add new users. | See previous. Comments: This function will be considered as an additional request and we will further evaluate if we have enough resources to add it to our scope. |
Changes to be implemented
(All changes to be implemented after acceptance)
- Remove the search section and replace it with a search bar at the top, with results displayed on a different page.
- View project details: grey out fields that cannot be edited.
- Project dates: spell out the month in full.
- Indicate the currency of the project budget.
- Add a toggle button for procurement staff to see all projects.
- Shift audit log button to view project details page.
User Testing 2
Date: 30 December 2013
Purpose: Evaluate usability of system including new functions: Edit PAF, Submit Supplier Information, Audit Log, changes since previous user test
Time: 2.15pm-2.45pm
User: Tam Chee Hong
Location: Skype (Screen sharing with our client as we are still using internal IP)
Browser: Internet Explorer
Role: Procurement staff
Comments from UT2
S/N | Function | Comments | Suggestion |
---|---|---|---|
1 | Search bar | What fields will the search bar search for? -Searches under all fields (including project description) |
Good. |
2 | Audit Log | Project log is good. We might need other kinds of logs too. |
Have logs for change of template, log in/log out (system-wide log) |
3 | Edit PAF Template | For basic information, the field type should be a drop-down list. | Change field type to dropdown list instead of getting users to enter the type (e.g. Integer, Date) |
4 | Edit PAF Template | Field types: most people won’t understand what does integer or text stand for, it’s confusing | Use simpler words like “numbers” or “words”, “paragraph”, “attachment” |
5 | Edit PAF Template | How many roles can I create under privilege? Can I create new privileges in future from the admin panel? How do I identify if someone is under procurement or not? | Allow changing of privileges from the admin panel. |
6 | Edit PAF Template | Customizable field sequence is nice | -- |
7 | Edit PAF Template | Assigned to & functional owner should be a dropdown list. | Change assigned to & functional owner to dropdown list. |
8 | Edit PAF Template | Project status should not be hardcoded. | Allow admin to add project status from admin panel. |
7 | Edit PAF Template | Assigned to & functional owner should be a dropdown list. | Change assigned to & functional owner to dropdown list. |
8 | Submit Supplier Information | We should cater for suppliers who sell products across multiple domains | Use tags to list out the products the suppliers provide. |
9 | Submit Supplier Information | Rating be tagged to the project instead of the supplier itself. | Don’t allow input of rating when a new supplier is created. Instead, when the project closes, procurement staff should do a rating of the supplier. |
Changes to be implemented
Function | Changes |
---|---|
Audit Log |
|
Edit PAF Template |
|
Submit Supplier Information |
|
Admin Panel (Future Implementation) |
|
User Testing 3
Date: 27 January 2014
Purpose: Evaluate usability of system: Home page, Create + Edit + Search Projects, Create + Edit + Search Suppliers, Edit PAF Template, View Audit Log, Approval
Time: 2pm-3pm
User: 2 Procurement team members
Location: Client's premises
Browser: IE11
Role: Procurement, Functional Owner
Information about Participants
Data | Participant 1 | Participant 2 |
---|---|---|
Photos | ||
Name | M. Bertl | C.H. Tam |
Occupation | Head of Procurement | Service Level Manager |
Experience with Procurement Systems | Yes | Yes |
Web Browser | IE11 | IE11 |
Goals
We would like to find out what are the thoughts that actual users of our Procurement Workflow Management System would have, be it in terms of functionality, intuitiveness or user interface. As M. Bertl is the Head of the department, we will consider his feedback as HiPPO (Highest Paid Person's Opinion).
Documents
Participants will start with part 1 first, before moving on to part 2.
Feedback from UT3
We have compiled all the suggestions given to us from the feedback gathered during the UT. They are grouped according to functions of our system. We discussed the various comments before deciding on whether to implement the changes or not.
Examples of UT3 Changes
Buttons-Tabs
Before Test | After Test |
---|---|
Middle-Bottom Approval Fields
Before Test | After Test |
---|---|
User Testing 4
Date: 7 February 2014
Purpose: Evaluate usability of system: Home page, Create + Edit + Search Projects, Create + Edit + Search Suppliers, Edit PAF Template, View Audit Log, Approval
Time: 12pm-7pm
User: 30 SMU students
Location: SIS CR 3-2 & SIS SR 3-2
Browser: IE8 - IE11
Role: Procurement, Functional Owner, Approver
Our Participants
Briefing participants before the start of the test: |
Bringing one of our participants through the task instructions: |
Participant enjoying the testing process: |
Goals
This is the first testing we have done with people who are not from our client's company. Our testers will be similar to new hires of the company who do not have any experience with the Procurement workflow management system before. This allows us to see whether our system is intuitive enough for users.
Documents
As there will be 3 participants within each 30min time slot, we have prepared 3 versions of the task instructions. Each of them will carry out the same tasks, but with different user ID.
Analysis of Data
We did an analysis based on the results collected from UT4 and divided them based on the functions tested.
Feedback from UT4
We have compiled all the suggestions given to us from the feedback gathered during the UT. They are grouped according to functions of our system. We discussed the various comments before deciding on whether to implement the changes or not.
Examples of UT4 Changes
Shortcut Buttons
Before Test | After Test |
---|---|
Chronological Order of Audit Log
Before Test | After Test |
---|---|
User Testing 5
Date: 25 March 2014
Time: 2pm-3pm
User: 5 LGB staff, with all user roles represented
Location: Client's office
Browser: IE8 - IE11
Role: Procurement, Head of Procurement, Functional Owner, Approvers (who are also functional owners)
Our Participants
Pre-test briefing of the participants regarding changes since last update. |
Participant during the testing process |
Goals
Functional testing of our system with all required features implemented. Testing was done with real data (real XPIDs, dummy project data) to test the usefulness and content validation of our system.
Documents
Feedback from UT5
We have compiled all the suggestions given to us from the feedback gathered during the UT. They are grouped according to functions of our system. We discussed the various comments before deciding on whether to implement the changes or not.
Examples of UT5 Changes
Links-Buttons
Before Test | After Test |
---|---|
Remarks Column
Before Test | After Test |
---|---|
Final Quotation & Cost Avoidance
Before Test | After Test |
---|---|
A/B Testing
Horizontal-Vertical Shortcut
Choice A | Choice B (Chosen) |
---|---|
Supplier Selection
Choice A | Choice B (Chosen) |
---|---|
Horizontal-Vertical Tabs
Choice A (Chosen) | Choice B |
---|---|
Overall Timings
Raw Data
Graph
% Improvement
Conclusion
In conclusion, the overall timings have improved and hence we have been steadily viewing an improvement in the usability. The key limitation, however, is that we are limited to detecting large differences between designs or measures. The reason is that majority of our tests do not possess a significant number of participants. However we still believe that collecting data would aid us in providing guidance on which features are need improvement. Additionally we have tried to mitigate the lack of users by getting representative users (e.g. procurement staff) to provide guidance.