HeaderSIS.jpg

IS480 Team wiki: 2013T2 D'PENZ User Testing

From IS480
Jump to navigation Jump to search

Dpenz logo.jpg

Neophase Home.png Home DPENZ Team Icon 2.png Our Team DPENZ Overview Icon.png Project Overview DPENZ Project Management Icon.png Project Management DPENZ Project Documentation Icon.png Project Documentation


Use Cases Architectural & Deployment Diagram Prototypes User Testing Performance Testing Presentation Slides Meeting Minutes Handover Documents



Heuristic Evaluation

Note: This was done on our system coded in JSP.
Date: 22 October 2013
Purpose: To test if our system is intuitive enough to be used by people who have not used procurement systems before.

Results from HE

Data Pilot Participant Participant 1 Participant 2 Participant 3
Participants Dpenz HE Shanaaz1.jpg Dpenz HE Derrick.jpg Dpenz HE Meizhen.jpg Dpenz HE Joel.jpg
Name Shanaaz Derrick Meizhen Joel
Age 23 22 22 27
Location SIS Level 4 SIS Level 4 SIS Level 4 SIS Level 4
Web Browser Google Chrome Google Chrome Google Chrome Google Chrome
Quantitative Data
Prior experience with Procurement systems) No No No No
Ease of editing a project
(Scale of 1 - 5 [highest])
5 4 5 5
Ease of searching for a project
(Scale of 1 - 5 [highest])
5 3 4 5
Ease of learning & using the interface
(Scale of 1 - 5 [highest])
5 4 5 4
Ease of navigating around the system
(Scale of 1 - 5 [highest])
5 3 4 4
Able to quickly find information needed
(Scale of 1 - 5 [highest])
5 4 3 3
Pathway for Manage Projects
(Scale of [not completed] 0 - 2 [easily completed])
2 2 1 2
Pathway for Search Projects
(Scale of [not completed] 0 - 2 [easily completed])
2 1 2 2
Pathway for Edit Project - Procurement Staff
(Scale of [not completed] 0 - 2 [easily completed])
2 1 2 2
Pathway for Edit Project - Functional Owner
(Scale of [not completed] 0 - 2 [easily completed])
2 2 2 2
Time taken to complete Manage Projects 2 min 25 sec 1 min 1 min
Time taken to complete Search Projects 1 min 3 min 30 sec 1 min 2 min
Time taken to complete Edit Project (Procurement Staff) 1 min 2 min 1 min 1 min
Time taken to complete Edit Project (Functional Owner) 2 min 3 min 1 min 1 min
Number of times participants asked for help 1 2 2 2
Number of bugs found in system 0 0 1 0
Qualitative Data
What does the participant think is required for a procurement workflow management system? Good user interface and experience that is understood by all the departments involved. Probably a centralised platform so that whoever is in-charge can manage the procurements I am not too sure about how a procurement workflow management system is like. Knowing the total number of projects on hand. Knowing important deadlines. Knowing who are the people involved.
Anything the participant like about our system? I like the simple UI and easy navigation through the pages. The UI looks rather professional. Simple and easy to navigate. Not too much of information on one page. The layout.
Anything the participant dislike about our system? Not that I can think of at the moment. It was pretty easy and sleek. Personal opinion is that there's too many information loaded on one page. - No help function.
Anything to be changed to provide an improved user experience? 1. It would be better if the search results appear on top of the User's current projects.
2. When I change the assigned user and save, I would like to see a confirmation message.
Maybe to provide those "changes have been saved" notifications. 1. If there is more than one table to be shown in a page, probably could use a different color for each table so that it is easier to identify and would not look as if there are too much of information in a page.
2. Color contrast can be improved.
Add a help function.


Round-up of Evaluation

  • Overall, the app is easy to use and navigate around.
  • More instructions could be provided so that users know what to do.
  • The user interface is simple, which is well-liked by testers.


Changes to be implemented

  1. Instead of displaying search results under Other Projects section (where users still have to scroll down), we will place the Search Results in the same tab as the Search fields (Change implemented)
  2. We will include a help function to aid new users in navigating around the system (Change will be implemented after acceptance)
  3. "Save Edits" button in the project details page will be placed both at the top and bottom of the page (Change will be implemented after acceptance)



User Testing 1

Note: This is the first testing we have done with on our system coded with ASP.NET.

Date: 29 October 2013
Purpose: Evaluate usability of functions: Home page, Search project, View & Edit Project Details

Time: 5pm-6pm
User: Tam Chee Hong
Location: Skype (Screen sharing with our client as we are still using internal IP)
Browser: Google Chrome
Role: Procurement staff


Comments from UT1

S/N Function Description Suggestion
1 Search Projects There are too many functions in one page (Home page) which leads to a lot of scrolling. Also, search function may not be used often since I can already see the projects that I'm working on. Remove the search section and replace it with a search bar at the top. Search results could be displayed on a different page.
2 View Project Details It is not clear which fields can or cannot be edited. Fields that cannot be edited should be greyed out.
3 View Project Details The start date is in a format like 01/03/2013, and it is confusing as it is not clear which is the month and which is the date. Spell out the month in full.
4 View Project Details Projects are submitted from various countries, hence it is important to know the currency of the project budget. Indicate the currency of the project budget.
5 Home Page There should be a toggle button to see all projects. Add a toggle button.
6 Audit Log It's funny that the audit log has it's own page when it should be for specific projects Shift audit log button to view project details page.
7 Add New User It will be good to have a function to add new users. See previous.
Comments: This function will be considered as an additional request and we will further evaluate if we have enough resources to add it to our scope.

Changes to be implemented

(All changes to be implemented after acceptance)

  1. Remove the search section and replace it with a search bar at the top, with results displayed on a different page.
  2. View project details: grey out fields that cannot be edited.
  3. Project dates: spell out the month in full.
  4. Indicate the currency of the project budget.
  5. Add a toggle button for procurement staff to see all projects.
  6. Shift audit log button to view project details page.



User Testing 2

Date: 30 December 2013
Purpose: Evaluate usability of system including new functions: Edit PAF, Submit Supplier Information, Audit Log, changes since previous user test
Time: 2.15pm-2.45pm
User: Tam Chee Hong
Location: Skype (Screen sharing with our client as we are still using internal IP)
Browser: Internet Explorer
Role: Procurement staff

Comments from UT2

S/N Function Comments Suggestion
1 Search bar What fields will the search bar search for?
-Searches under all fields (including project description)
Good.
2 Audit Log Project log is good.

We might need other kinds of logs too.
Have logs for change of template, log in/log out (system-wide log)
3 Edit PAF Template For basic information, the field type should be a drop-down list. Change field type to dropdown list instead of getting users to enter the type (e.g. Integer, Date)
4 Edit PAF Template Field types: most people won’t understand what does integer or text stand for, it’s confusing Use simpler words like “numbers” or “words”, “paragraph”, “attachment”
5 Edit PAF Template How many roles can I create under privilege? Can I create new privileges in future from the admin panel? How do I identify if someone is under procurement or not? Allow changing of privileges from the admin panel.
6 Edit PAF Template Customizable field sequence is nice --
7 Edit PAF Template Assigned to & functional owner should be a dropdown list. Change assigned to & functional owner to dropdown list.
8 Edit PAF Template Project status should not be hardcoded. Allow admin to add project status from admin panel.
7 Edit PAF Template Assigned to & functional owner should be a dropdown list. Change assigned to & functional owner to dropdown list.
8 Submit Supplier Information We should cater for suppliers who sell products across multiple domains Use tags to list out the products the suppliers provide.
9 Submit Supplier Information Rating be tagged to the project instead of the supplier itself. Don’t allow input of rating when a new supplier is created. Instead, when the project closes, procurement staff should do a rating of the supplier.

Changes to be implemented

Function Changes
Audit Log
  1. Logs for change of template, log in/log out (system-wide log)
    we will evaluate this according to potential impacts on page response times, will consider it as tertiary function
Edit PAF Template
  1. For basic project information, change field type to dropdown list instead of getting users to enter the type (e.g. Integer, Date)
  2. Use simpler words like “numbers” or “words”, “paragraph”, “attachment” for field types
  3. Change assigned to & functional owner to dropdown list.
Submit Supplier Information
  1. Use checkboxes instead for procurement staff to list out the products the suppliers provide.
  2. Put rating of suppliers under the last phase of the project details. This rating will be linked to the supplier.
  3. Rating should be reflected as an average from various projects.
Admin Panel (Future Implementation)
  1. Allow changing of privileges from the admin panel
  2. Allow admin to add project status from admin panel
  3. Allow a section in the admin panel to edit/add more domains


User Testing 3

Date: 27 January 2014
Purpose: Evaluate usability of system: Home page, Create + Edit + Search Projects, Create + Edit + Search Suppliers, Edit PAF Template, View Audit Log, Approval
Time: 2pm-3pm
User: 2 Procurement team members
Location: Client's premises
Browser: IE11
Role: Procurement, Functional Owner

Information about Participants

Data Participant 1 Participant 2
Photos
DPENZ UT3 MB.jpg
DPENZ UT3 TCH.jpg
Name M. Bertl C.H. Tam
Occupation Head of Procurement Service Level Manager
Experience with Procurement Systems Yes Yes
Web Browser IE11 IE11

Goals

We would like to find out what are the thoughts that actual users of our Procurement Workflow Management System would have, be it in terms of functionality, intuitiveness or user interface. As M. Bertl is the Head of the department, we will consider his feedback as HiPPO (Highest Paid Person's Opinion).

Documents

Participants will start with part 1 first, before moving on to part 2.

Feedback from UT3

We have compiled all the suggestions given to us from the feedback gathered during the UT. They are grouped according to functions of our system. We discussed the various comments before deciding on whether to implement the changes or not.

Examples of UT3 Changes

Buttons-Tabs

Before Test After Test
Previously we used buttons for the staff to access key buttons.
We used tabs to keep the page cleaner.

Middle-Bottom Approval Fields

Before Test After Test
Previously we used to put in the details of the approval in the middle of the form
We currently place it at the end of form. This flow makes more sense as the approver will peruse the details of the form before approving/rejecting the project.


User Testing 4

Date: 7 February 2014
Purpose: Evaluate usability of system: Home page, Create + Edit + Search Projects, Create + Edit + Search Suppliers, Edit PAF Template, View Audit Log, Approval
Time: 12pm-7pm
User: 30 SMU students
Location: SIS CR 3-2 & SIS SR 3-2
Browser: IE8 - IE11
Role: Procurement, Functional Owner, Approver

Our Participants

Briefing participants before the start of the test:
Briefing participants before the start of the test.
Bringing one of our participants through the task instructions:
Bringing one of our participant through the task instructions.
Participant enjoying the testing process:
Participant enjoying the testing process.

Goals

This is the first testing we have done with people who are not from our client's company. Our testers will be similar to new hires of the company who do not have any experience with the Procurement workflow management system before. This allows us to see whether our system is intuitive enough for users.

Documents

As there will be 3 participants within each 30min time slot, we have prepared 3 versions of the task instructions. Each of them will carry out the same tasks, but with different user ID.

Analysis of Data

We did an analysis based on the results collected from UT4 and divided them based on the functions tested.

Feedback from UT4

We have compiled all the suggestions given to us from the feedback gathered during the UT. They are grouped according to functions of our system. We discussed the various comments before deciding on whether to implement the changes or not.

Examples of UT4 Changes

Shortcut Buttons

Before Test After Test
Previously there were no shortcut buttons to access the key features of IPS.
We currently have them for better accessibility.

Chronological Order of Audit Log

Before Test After Test
Previously we used to sort the Audit Log from Old to New audits.
We currently do it reverse chronologically so that they can see the latest edits for convenience.

User Testing 5

Date: 25 March 2014
Time: 2pm-3pm
User: 5 LGB staff, with all user roles represented
Location: Client's office
Browser: IE8 - IE11
Role: Procurement, Head of Procurement, Functional Owner, Approvers (who are also functional owners)

Our Participants

Pre-test briefing of the participants regarding changes since last update.
Dpenz UT5 Photo1.jpg
Participant during the testing process
Dpenz UT5 Photo2.jpg

Goals

Functional testing of our system with all required features implemented. Testing was done with real data (real XPIDs, dummy project data) to test the usefulness and content validation of our system.

Documents

Feedback from UT5

We have compiled all the suggestions given to us from the feedback gathered during the UT. They are grouped according to functions of our system. We discussed the various comments before deciding on whether to implement the changes or not.

Examples of UT5 Changes

Links-Buttons

Before Test After Test
Previously we used links for some of our features.
We have standardized it all to be buttons for greater consistency.

Remarks Column

Before Test After Test
Previously we had no remarks column.
We currently included one for the purposes of making miscellaneous comments. Examples of comments include reasons for transferring ownership for the project.

Final Quotation & Cost Avoidance

Before Test After Test
Previously we had no final quotations and cost avoidance field.
We currently included it to automate calculation of cost savings/avoidance along with identifying how much the respective staff member has saved.

A/B Testing

Horizontal-Vertical Shortcut

Choice A Choice B (Chosen)
Horizontal shortcut button would be in alignment with the tabs. (1 VOTE)
Vertical shortcut buttons would be useful in not pushing the items of the page down. For example the projects that they own. (4 VOTES)


Supplier Selection

Choice A Choice B (Chosen)
Table style of selecting suppliers. (2 VOTES)
Row style of selecting suppliers. (3 VOTES)

Horizontal-Vertical Tabs

Choice A (Chosen) Choice B
Horizontal tabs are the prevalent design in websites. (4 VOTES)
This was an experimental choice to determine if users are receptive to newer designs. (1 VOTE)

Overall Timings


OverallTimingUT.PNG

Conclusion

In conclusion, the overall timings have improved and hence we have been steadily viewing an improvement in the usability. The key limitation, however, is that we are limited to detecting large differences between designs or measures. The reason is that majority of our tests do not possess a significant number of participants. However we still believe that collecting data would aid us in providing guidance on which features are need improvement. Additionally we have tried to mitigate the lack of users by getting representative users (e.g. procurement staff) to provide guidance.