Difference between revisions of "IS480 Team wiki: 2013T2 D'PENZ User Testing"
(→Graph) |
|||
(23 intermediate revisions by the same user not shown) | |||
Line 27: | Line 27: | ||
|style="padding:0.4em; border-left:1px solid #ffffff; border-right:1px solid #ffffff; text-align:center; background-color:#ff9a00; " width="10%" | [[IS480_Team_wiki:_2013T2 D'PENZ_User_Testing |<font color="#ffffff"><b>User Testing</b></font>]] | |style="padding:0.4em; border-left:1px solid #ffffff; border-right:1px solid #ffffff; text-align:center; background-color:#ff9a00; " width="10%" | [[IS480_Team_wiki:_2013T2 D'PENZ_User_Testing |<font color="#ffffff"><b>User Testing</b></font>]] | ||
+ | |||
+ | |style="padding:0.4em; border-left:1px solid #ffffff; border-right:1px solid #ffffff; text-align:center; background-color:#0056a6; " width="10%" | [[IS480_Team_wiki:_2013T2 D'PENZ_Performance_Testing |<font color="#ffffff"><b>Performance Testing</b></font>]] | ||
|style="padding:0.4em; border-left:1px solid #ffffff; border-right:1px solid #ffffff; text-align:center; background-color:#0056a6; " width="10%" | [[IS480_Team_wiki:_2013T2 D'PENZ_Presentation_Slides |<font color="#ffffff"><b>Presentation Slides</b></font>]] | |style="padding:0.4em; border-left:1px solid #ffffff; border-right:1px solid #ffffff; text-align:center; background-color:#0056a6; " width="10%" | [[IS480_Team_wiki:_2013T2 D'PENZ_Presentation_Slides |<font color="#ffffff"><b>Presentation Slides</b></font>]] | ||
Line 393: | Line 395: | ||
* [[Media:UT3_Feedback_compiled.docx|Compiled Feedback for Changes]] | * [[Media:UT3_Feedback_compiled.docx|Compiled Feedback for Changes]] | ||
+ | ===Examples of UT3 Changes=== | ||
− | + | ====Buttons-Tabs==== | |
− | == | ||
− | |||
− | == Buttons-Tabs == | ||
{| class="wikitable" border="1" | {| class="wikitable" border="1" | ||
! Before Test!! After Test | ! Before Test!! After Test | ||
Line 405: | Line 405: | ||
|} | |} | ||
− | == Middle-Bottom Approval Fields == | + | ====Middle-Bottom Approval Fields==== |
{| class="wikitable" border="1" | {| class="wikitable" border="1" | ||
! Before Test!! After Test | ! Before Test!! After Test | ||
|- | |- | ||
− | | [[Image: | + | | [[Image:Middle.PNG|center|thumb|450px|Previously we used to put in the details of the approval in the middle of the form]] |
− | || [[Image: | + | || [[Image:Bottom.PNG|center|thumb|450px|We currently place it at the end of form. This flow makes more sense as the approver will peruse the details of the form before approving/rejecting the project.]] |
|} | |} | ||
Line 458: | Line 458: | ||
We have compiled all the suggestions given to us from the feedback gathered during the UT. They are grouped according to functions of our system. We discussed the various comments before deciding on whether to implement the changes or not. | We have compiled all the suggestions given to us from the feedback gathered during the UT. They are grouped according to functions of our system. We discussed the various comments before deciding on whether to implement the changes or not. | ||
*[[Media:DPENZ_UT4_Feedback.xlsx|Compiled Feedback for Changes]]<br><br> | *[[Media:DPENZ_UT4_Feedback.xlsx|Compiled Feedback for Changes]]<br><br> | ||
+ | |||
+ | ===Examples of UT4 Changes=== | ||
+ | |||
+ | ====Shortcut Buttons==== | ||
+ | {| class="wikitable" border="1" | ||
+ | ! Before Test!! After Test | ||
+ | |- | ||
+ | | [[Image:No Shortcut.PNG|center|thumb|450px|Previously there were no shortcut buttons to access the key features of IPS.]] | ||
+ | || [[Image:Shortcut.PNG|center|thumb|450px|We currently have them for better accessibility.]] | ||
+ | |} | ||
+ | |||
+ | ====Chronological Order of Audit Log==== | ||
+ | {| class="wikitable" border="1" | ||
+ | ! Before Test!! After Test | ||
+ | |- | ||
+ | | [[Image:OldToNew.PNG|center|thumb|450px|Previously we used to sort the Audit Log from Old to New audits.]] | ||
+ | || [[Image:NewToOld.PNG|center|thumb|450px|We currently do it reverse chronologically so that they can see the latest edits for convenience.]] | ||
+ | |} | ||
== <font face="Droid Sans" size="+3" color="#ff9a00">User Testing 5</font> == | == <font face="Droid Sans" size="+3" color="#ff9a00">User Testing 5</font> == | ||
Line 490: | Line 508: | ||
We have compiled all the suggestions given to us from the feedback gathered during the UT. They are grouped according to functions of our system. We discussed the various comments before deciding on whether to implement the changes or not. | We have compiled all the suggestions given to us from the feedback gathered during the UT. They are grouped according to functions of our system. We discussed the various comments before deciding on whether to implement the changes or not. | ||
*[[Media:Dpenz_Compiled_UT5_Feedback.pdf|Compiled Feedback for Changes]] | *[[Media:Dpenz_Compiled_UT5_Feedback.pdf|Compiled Feedback for Changes]] | ||
+ | |||
+ | ===Examples of UT5 Changes=== | ||
+ | |||
+ | ====Links-Buttons==== | ||
+ | {| class="wikitable" border="1" | ||
+ | ! Before Test!! After Test | ||
+ | |- | ||
+ | | [[Image:LinksButtons.PNG|center|thumb|450px|Previously we used links for some of our features.]] | ||
+ | || [[Image:ButtonButton.PNG|center|thumb|450px|We have standardized it all to be buttons for greater consistency.]] | ||
+ | |} | ||
+ | |||
+ | ====Remarks Column==== | ||
+ | {| class="wikitable" border="1" | ||
+ | ! Before Test!! After Test | ||
+ | |- | ||
+ | | [[Image:NoRemarks.PNG|center|thumb|450px|Previously we had no remarks column.]] | ||
+ | || [[Image:Remarks.PNG|center|thumb|450px|We currently included one for the purposes of making miscellaneous comments. Examples of comments include reasons for transferring ownership for the project.]] | ||
+ | |} | ||
+ | |||
+ | ====Final Quotation & Cost Avoidance==== | ||
+ | {| class="wikitable" border="1" | ||
+ | ! Before Test!! After Test | ||
+ | |- | ||
+ | | [[Image:NoFinalQuotes.PNG|center|thumb|450px|Previously we had no final quotations and cost avoidance field.]] | ||
+ | || [[Image:FinalQuotes.PNG|center|thumb|450px|We currently included it to automate calculation of cost savings/avoidance along with identifying how much the respective staff member has saved.]] | ||
+ | |} | ||
+ | |||
+ | ===A/B Testing=== | ||
+ | |||
+ | ====Horizontal-Vertical Shortcut==== | ||
+ | {| class="wikitable" border="1" | ||
+ | ! Choice A!! Choice B (Chosen) | ||
+ | |- | ||
+ | | [[Image:HorizontalShortcut.PNG|center|thumb|450px|Horizontal shortcut button would be in alignment with the tabs. (1 VOTE)]] | ||
+ | || [[Image:VerticalShortcut.PNG|center|thumb|450px|Vertical shortcut buttons would be useful in not pushing the items of the page down. For example the projects that they own. (4 VOTES)]] | ||
+ | |} | ||
+ | |||
+ | |||
+ | ====Supplier Selection==== | ||
+ | {| class="wikitable" border="1" | ||
+ | ! Choice A!! Choice B (Chosen) | ||
+ | |- | ||
+ | | [[Image:TableStyle.PNG|center|thumb|450px|Table style of selecting suppliers. (2 VOTES)]] | ||
+ | || [[Image:RowStyle.PNG|center|thumb|450px|Row style of selecting suppliers. (3 VOTES)]] | ||
+ | |} | ||
+ | |||
+ | ====Horizontal-Vertical Tabs==== | ||
+ | {| class="wikitable" border="1" | ||
+ | ! Choice A (Chosen) !! Choice B | ||
+ | |- | ||
+ | | [[Image:HorizontalTabs.PNG|center|thumb|450px|Horizontal tabs are the prevalent design in websites. (4 VOTES)]] | ||
+ | || [[Image:VerticalTabs.PNG|center|thumb|450px|This was an experimental choice to determine if users are receptive to newer designs. (1 VOTE)]] | ||
+ | |} | ||
+ | |||
+ | == <font face="Droid Sans" size="+3" color="#ff9a00">Overall Timings</font> == | ||
+ | |||
+ | ===Raw Data=== | ||
+ | <i>Timings are in seconds</i><br> | ||
+ | [[Image:RawData.PNG]] | ||
+ | <br> | ||
+ | <br> | ||
+ | |||
+ | ===Graph=== | ||
+ | <br> | ||
+ | [[Image:OverallTimingUT.PNG]] | ||
+ | <br> | ||
+ | <br> | ||
+ | |||
+ | ===% Improvement=== | ||
+ | [[Image:%Improvement.PNG]] | ||
+ | <br> | ||
+ | |||
+ | == <font face="Droid Sans" size="+3" color="#ff9a00">Conclusion</font> == | ||
+ | |||
+ | In conclusion, the overall timings have improved and hence we have been steadily viewing an improvement in the usability. The key limitation, however, is that we are limited to detecting large differences between designs or measures. The reason is that majority of our tests do not possess a significant number of participants. However we still believe that collecting data would aid us in providing guidance on which features are need improvement. Additionally we have tried to mitigate the lack of users by getting representative users (e.g. procurement staff) to provide guidance. |
Latest revision as of 23:17, 21 April 2014
Home | Our Team | Project Overview | Project Management | Project Documentation |
Use Cases | Architectural & Deployment Diagram | Prototypes | User Testing | Performance Testing | Presentation Slides | Meeting Minutes | Handover Documents |
Contents
Heuristic Evaluation
Note: This was done on our system coded in JSP.
Date: 22 October 2013
Purpose: To test if our system is intuitive enough to be used by people who have not used procurement systems before.
Results from HE
Round-up of Evaluation
- Overall, the app is easy to use and navigate around.
- More instructions could be provided so that users know what to do.
- The user interface is simple, which is well-liked by testers.
Changes to be implemented
- Instead of displaying search results under Other Projects section (where users still have to scroll down), we will place the Search Results in the same tab as the Search fields (Change implemented)
- We will include a help function to aid new users in navigating around the system (Change will be implemented after acceptance)
- "Save Edits" button in the project details page will be placed both at the top and bottom of the page (Change will be implemented after acceptance)
User Testing 1
Note: This is the first testing we have done with on our system coded with ASP.NET.
Date: 29 October 2013
Purpose: Evaluate usability of functions: Home page, Search project, View & Edit Project Details
Time: 5pm-6pm
User: Tam Chee Hong
Location: Skype (Screen sharing with our client as we are still using internal IP)
Browser: Google Chrome
Role: Procurement staff
Comments from UT1
S/N | Function | Description | Suggestion |
---|---|---|---|
1 | Search Projects | There are too many functions in one page (Home page) which leads to a lot of scrolling. Also, search function may not be used often since I can already see the projects that I'm working on. | Remove the search section and replace it with a search bar at the top. Search results could be displayed on a different page. |
2 | View Project Details | It is not clear which fields can or cannot be edited. | Fields that cannot be edited should be greyed out. |
3 | View Project Details | The start date is in a format like 01/03/2013, and it is confusing as it is not clear which is the month and which is the date. | Spell out the month in full. |
4 | View Project Details | Projects are submitted from various countries, hence it is important to know the currency of the project budget. | Indicate the currency of the project budget. |
5 | Home Page | There should be a toggle button to see all projects. | Add a toggle button. |
6 | Audit Log | It's funny that the audit log has it's own page when it should be for specific projects | Shift audit log button to view project details page. |
7 | Add New User | It will be good to have a function to add new users. | See previous. Comments: This function will be considered as an additional request and we will further evaluate if we have enough resources to add it to our scope. |
Changes to be implemented
(All changes to be implemented after acceptance)
- Remove the search section and replace it with a search bar at the top, with results displayed on a different page.
- View project details: grey out fields that cannot be edited.
- Project dates: spell out the month in full.
- Indicate the currency of the project budget.
- Add a toggle button for procurement staff to see all projects.
- Shift audit log button to view project details page.
User Testing 2
Date: 30 December 2013
Purpose: Evaluate usability of system including new functions: Edit PAF, Submit Supplier Information, Audit Log, changes since previous user test
Time: 2.15pm-2.45pm
User: Tam Chee Hong
Location: Skype (Screen sharing with our client as we are still using internal IP)
Browser: Internet Explorer
Role: Procurement staff
Comments from UT2
S/N | Function | Comments | Suggestion |
---|---|---|---|
1 | Search bar | What fields will the search bar search for? -Searches under all fields (including project description) |
Good. |
2 | Audit Log | Project log is good. We might need other kinds of logs too. |
Have logs for change of template, log in/log out (system-wide log) |
3 | Edit PAF Template | For basic information, the field type should be a drop-down list. | Change field type to dropdown list instead of getting users to enter the type (e.g. Integer, Date) |
4 | Edit PAF Template | Field types: most people won’t understand what does integer or text stand for, it’s confusing | Use simpler words like “numbers” or “words”, “paragraph”, “attachment” |
5 | Edit PAF Template | How many roles can I create under privilege? Can I create new privileges in future from the admin panel? How do I identify if someone is under procurement or not? | Allow changing of privileges from the admin panel. |
6 | Edit PAF Template | Customizable field sequence is nice | -- |
7 | Edit PAF Template | Assigned to & functional owner should be a dropdown list. | Change assigned to & functional owner to dropdown list. |
8 | Edit PAF Template | Project status should not be hardcoded. | Allow admin to add project status from admin panel. |
7 | Edit PAF Template | Assigned to & functional owner should be a dropdown list. | Change assigned to & functional owner to dropdown list. |
8 | Submit Supplier Information | We should cater for suppliers who sell products across multiple domains | Use tags to list out the products the suppliers provide. |
9 | Submit Supplier Information | Rating be tagged to the project instead of the supplier itself. | Don’t allow input of rating when a new supplier is created. Instead, when the project closes, procurement staff should do a rating of the supplier. |
Changes to be implemented
Function | Changes |
---|---|
Audit Log |
|
Edit PAF Template |
|
Submit Supplier Information |
|
Admin Panel (Future Implementation) |
|
User Testing 3
Date: 27 January 2014
Purpose: Evaluate usability of system: Home page, Create + Edit + Search Projects, Create + Edit + Search Suppliers, Edit PAF Template, View Audit Log, Approval
Time: 2pm-3pm
User: 2 Procurement team members
Location: Client's premises
Browser: IE11
Role: Procurement, Functional Owner
Information about Participants
Data | Participant 1 | Participant 2 |
---|---|---|
Photos | ||
Name | M. Bertl | C.H. Tam |
Occupation | Head of Procurement | Service Level Manager |
Experience with Procurement Systems | Yes | Yes |
Web Browser | IE11 | IE11 |
Goals
We would like to find out what are the thoughts that actual users of our Procurement Workflow Management System would have, be it in terms of functionality, intuitiveness or user interface. As M. Bertl is the Head of the department, we will consider his feedback as HiPPO (Highest Paid Person's Opinion).
Documents
Participants will start with part 1 first, before moving on to part 2.
Feedback from UT3
We have compiled all the suggestions given to us from the feedback gathered during the UT. They are grouped according to functions of our system. We discussed the various comments before deciding on whether to implement the changes or not.
Examples of UT3 Changes
Buttons-Tabs
Before Test | After Test |
---|---|
Middle-Bottom Approval Fields
Before Test | After Test |
---|---|
User Testing 4
Date: 7 February 2014
Purpose: Evaluate usability of system: Home page, Create + Edit + Search Projects, Create + Edit + Search Suppliers, Edit PAF Template, View Audit Log, Approval
Time: 12pm-7pm
User: 30 SMU students
Location: SIS CR 3-2 & SIS SR 3-2
Browser: IE8 - IE11
Role: Procurement, Functional Owner, Approver
Our Participants
Briefing participants before the start of the test: |
Bringing one of our participants through the task instructions: |
Participant enjoying the testing process: |
Goals
This is the first testing we have done with people who are not from our client's company. Our testers will be similar to new hires of the company who do not have any experience with the Procurement workflow management system before. This allows us to see whether our system is intuitive enough for users.
Documents
As there will be 3 participants within each 30min time slot, we have prepared 3 versions of the task instructions. Each of them will carry out the same tasks, but with different user ID.
Analysis of Data
We did an analysis based on the results collected from UT4 and divided them based on the functions tested.
Feedback from UT4
We have compiled all the suggestions given to us from the feedback gathered during the UT. They are grouped according to functions of our system. We discussed the various comments before deciding on whether to implement the changes or not.
Examples of UT4 Changes
Shortcut Buttons
Before Test | After Test |
---|---|
Chronological Order of Audit Log
Before Test | After Test |
---|---|
User Testing 5
Date: 25 March 2014
Time: 2pm-3pm
User: 5 LGB staff, with all user roles represented
Location: Client's office
Browser: IE8 - IE11
Role: Procurement, Head of Procurement, Functional Owner, Approvers (who are also functional owners)
Our Participants
Pre-test briefing of the participants regarding changes since last update. |
Participant during the testing process |
Goals
Functional testing of our system with all required features implemented. Testing was done with real data (real XPIDs, dummy project data) to test the usefulness and content validation of our system.
Documents
Feedback from UT5
We have compiled all the suggestions given to us from the feedback gathered during the UT. They are grouped according to functions of our system. We discussed the various comments before deciding on whether to implement the changes or not.
Examples of UT5 Changes
Links-Buttons
Before Test | After Test |
---|---|
Remarks Column
Before Test | After Test |
---|---|
Final Quotation & Cost Avoidance
Before Test | After Test |
---|---|
A/B Testing
Horizontal-Vertical Shortcut
Choice A | Choice B (Chosen) |
---|---|
Supplier Selection
Choice A | Choice B (Chosen) |
---|---|
Horizontal-Vertical Tabs
Choice A (Chosen) | Choice B |
---|---|
Overall Timings
Raw Data
Graph
% Improvement
Conclusion
In conclusion, the overall timings have improved and hence we have been steadily viewing an improvement in the usability. The key limitation, however, is that we are limited to detecting large differences between designs or measures. The reason is that majority of our tests do not possess a significant number of participants. However we still believe that collecting data would aid us in providing guidance on which features are need improvement. Additionally we have tried to mitigate the lack of users by getting representative users (e.g. procurement staff) to provide guidance.