HeaderSIS.jpg

IS480 Team wiki: 2013T2 D'PENZ User Testing

From IS480
Jump to navigation Jump to search

Dpenz logo.jpg

Neophase Home.png Home DPENZ Team Icon 2.png Our Team DPENZ Overview Icon.png Project Overview DPENZ Project Management Icon.png Project Management DPENZ Project Documentation Icon.png Project Documentation


Use Cases Architectural & Deployment Diagram Prototypes User Testing Presentation Slides Meeting Minutes



Heuristic Evaluation

Note: This was done on our system coded in JSP.
Date: 22 October 2013
Purpose: To test if our system is intuitive enough to be used by people who have not used procurement systems before.

Results from HE

Data Pilot Participant Participant 1 Participant 2 Participant 3
Participants Dpenz HE Shanaaz1.jpg Dpenz HE Derrick.jpg Dpenz HE Meizhen.jpg Dpenz HE Joel.jpg
Name Shanaaz Derrick Meizhen Joel
Age 23 22 22 27
Location SIS Level 4 SIS Level 4 SIS Level 4 SIS Level 4
Web Browser Google Chrome Google Chrome Google Chrome Google Chrome
Quantitative Data
Prior experience with Procurement systems) No No No No
Ease of editing a project
(Scale of 1 - 5 [highest])
5 4 5 5
Ease of searching for a project
(Scale of 1 - 5 [highest])
5 3 4 5
Ease of learning & using the interface
(Scale of 1 - 5 [highest])
5 4 5 4
Ease of navigating around the system
(Scale of 1 - 5 [highest])
5 3 4 4
Able to quickly find information needed
(Scale of 1 - 5 [highest])
5 4 3 3
Pathway for Manage Projects
(Scale of [not completed] 0 - 2 [easily completed])
2 2 1 2
Pathway for Search Projects
(Scale of [not completed] 0 - 2 [easily completed])
2 1 2 2
Pathway for Edit Project - Procurement Staff
(Scale of [not completed] 0 - 2 [easily completed])
2 1 2 2
Pathway for Edit Project - Functional Owner
(Scale of [not completed] 0 - 2 [easily completed])
2 2 2 2
Time taken to complete Manage Projects 2 min 25 sec 1 min 1 min
Time taken to complete Search Projects 1 min 3 min 30 sec 1 min 2 min
Time taken to complete Edit Project (Procurement Staff) 1 min 2 min 1 min 1 min
Time taken to complete Edit Project (Functional Owner) 2 min 3 min 1 min 1 min
Number of times participants asked for help 1 2 2 2
Number of bugs found in system 0 0 1 0
Qualitative Data
What does the participant think is required for a procurement workflow management system? Good user interface and experience that is understood by all the departments involved. Probably a centralised platform so that whoever is in-charge can manage the procurements I am not too sure about how a procurement workflow management system is like. Knowing the total number of projects on hand. Knowing important deadlines. Knowing who are the people involved.
Anything the participant like about our system? I like the simple UI and easy navigation through the pages. The UI looks rather professional. Simple and easy to navigate. Not too much of information on one page. The layout.
Anything the participant dislike about our system? Not that I can think of at the moment. It was pretty easy and sleek. Personal opinion is that there's too many information loaded on one page. - No help function.
Anything to be changed to provide an improved user experience? 1. It would be better if the search results appear on top of the User's current projects.
2. When I change the assigned user and save, I would like to see a confirmation message.
Maybe to provide those "changes have been saved" notifications. 1. If there is more than one table to be shown in a page, probably could use a different color for each table so that it is easier to identify and would not look as if there are too much of information in a page.
2. Color contrast can be improved.
Add a help function.


Round-up of Evaluation

  • Overall, the app is easy to use and navigate around.
  • More instructions could be provided so that users know what to do.
  • The user interface is simple, which is well-liked by testers.


Changes to be implemented

  1. Instead of displaying search results under Other Projects section (where users still have to scroll down), we will place the Search Results in the same tab as the Search fields (Change implemented)
  2. We will include a help function to aid new users in navigating around the system (Change will be implemented after acceptance)
  3. "Save Edits" button in the project details page will be placed both at the top and bottom of the page (Change will be implemented after acceptance)



User Testing 1

Note: This is the first testing we have done with on our system coded with ASP.NET.

Date: 29 November 2013
Purpose: Evaluate usability of functions: Home page, Search project, View & Edit Project Details

Time: 5pm-6pm
User: Tam Chee Hong
Location: Skype (Screen sharing with our client as we are still using internal IP)
Browser: Google Chrome
Role: Procurement staff


Comments from UT1

S/N Function Description Suggestion
1 Search Projects There are too many functions in one page (Home page) which leads to a lot of scrolling. Also, search function may not be used often since I can already see the projects that I'm working on. Remove the search section and replace it with a search bar at the top. Search results could be displayed on a different page.
2 View Project Details It is not clear which fields can or cannot be edited. Fields that cannot be edited should be greyed out.
3 View Project Details The start date is in a format like 01/03/2013, and it is confusing as it is not clear which is the month and which is the date. Spell out the month in full.
4 View Project Details Projects are submitted from various countries, hence it is important to know the currency of the project budget. Indicate the currency of the project budget.
5 Home Page There should be a toggle button to see all projects. Add a toggle button.
6 Audit Log It's funny that the audit log has it's own page when it should be for specific projects Shift audit log button to view project details page.
7 Add New User It will be good to have a function to add new users. See previous.
Comments: This function will be considered as an additional request and we will further evaluate if we have enough resources to add it to our scope.

Changes to be implemented

(All changes to be implemented after acceptance)

  1. Remove the search section and replace it with a search bar at the top, with results displayed on a different page.
  2. View project details: grey out fields that cannot be edited.
  3. Project dates: spell out the month in full.
  4. Indicate the currency of the project budget.
  5. Add a toggle button for procurement staff to see all projects.
  6. Shift audit log button to view project details page.



User Testing 2

Date: 30 December 2013
Purpose: Evaluate usability of system including new functions: Edit PAF, Submit Supplier Information, Audit Log, changes since previous user test
Time: 2.15pm-2.45pm
User: Tam Chee Hong
Location: Skype (Screen sharing with our client as we are still using internal IP)
Browser: Internet Explorer
Role: Procurement staff

Comments from UT2

S/N Function Comments Suggestion
1 Search bar What fields will the search bar search for?
-Searches under all fields (including project description)
Good.
2 Audit Log Project log is good.

We might need other kinds of logs too.
Have logs for change of template, log in/log out (system-wide log)
3 Edit PAF Template For basic information, the field type should be a drop-down list. Change field type to dropdown list instead of getting users to enter the type (e.g. Integer, Date)
4 Edit PAF Template Field types: most people won’t understand what does integer or text stand for, it’s confusing Use simpler words like “numbers” or “words”, “paragraph”, “attachment”
5 Edit PAF Template How many roles can I create under privilege? Can I create new privileges in future from the admin panel? How do I identify if someone is under procurement or not? Allow changing of privileges from the admin panel.
6 Edit PAF Template Customizable field sequence is nice --
7 Edit PAF Template Assigned to & functional owner should be a dropdown list. Change assigned to & functional owner to dropdown list.
8 Edit PAF Template Project status should not be hardcoded. Allow admin to add project status from admin panel.
7 Edit PAF Template Assigned to & functional owner should be a dropdown list. Change assigned to & functional owner to dropdown list.
8 Submit Supplier Information We should cater for suppliers who sell products across multiple domains Use tags to list out the products the suppliers provide.
9 Submit Supplier Information Rating be tagged to the project instead of the supplier itself. Don’t allow input of rating when a new supplier is created. Instead, when the project closes, procurement staff should do a rating of the supplier.

Changes to be implemented

Function Changes
Audit Log
  1. Logs for change of template, log in/log out (system-wide log)
    we will evaluate this according to potential impacts on page response times, will consider it as tertiary function
Edit PAF Template
  1. For basic project information, change field type to dropdown list instead of getting users to enter the type (e.g. Integer, Date)
  2. Use simpler words like “numbers” or “words”, “paragraph”, “attachment” for field types
  3. Change assigned to & functional owner to dropdown list.
Submit Supplier Information
  1. Use checkboxes instead for procurement staff to list out the products the suppliers provide.
  2. Put rating of suppliers under the last phase of the project details. This rating will be linked to the supplier.
  3. Rating should be reflected as an average from various projects.
Admin Panel (Future Implementation)
  1. Allow changing of privileges from the admin panel
  2. Allow admin to add project status from admin panel
  3. Allow a section in the admin panel to edit/add more domains


User Testing 3

Date: 27 January 2014
Purpose: Evaluate usability of system: Home page, Create + Edit + Search Projects, Create + Edit + Search Suppliers, Edit PAF Template, View Audit Log, Approval
Time: 2pm-3pm
User: 2 Procurement team members
Location: Client's premises
Browser: IE11
Role: Procurement, Functional Owner

Information about Participants

Data Participant 1 Participant 2
Photos
DPENZ UT3 MB.jpg
DPENZ UT3 TCH.jpg
Name M. Bertl C.H. Tam
Occupation Head of Procurement Service Level Manager
Experience with Procurement Systems Yes Yes
Web Browser IE11 IE11

Goals

We would like to find out what are the thoughts that actual users of our Procurement Workflow Management System would have, be it in terms of functionality, intuitiveness or user interface. As M. Bertl is the Head of the department, we will consider his feedback as HiPPO (Highest Paid Person's Opinion).

Documents

Participants will start with part 1 first, before moving on to part 2.

Changes based on UT3

We have compiled all the suggestions given to us from the feedback gathered during the UT. They are grouped according to functions of our system. We discussed the various comments before deciding on whether to implement the changes or not.


User Testing 4

Date: 7 February 2014
Purpose: Evaluate usability of system: Home page, Create + Edit + Search Projects, Create + Edit + Search Suppliers, Edit PAF Template, View Audit Log, Approval
Time: 12pm-7pm
User: 30 SMU students
Location: SIS CR 3-2 & SIS SR 3-2
Browser: IE8 - IE11
Role: Procurement, Functional Owner, Approver

Our Participants

Briefing participants before the start of the test:
Briefing participants before the start of the test.
Bringing one of our participants through the task instructions:
Bringing one of our participant through the task instructions.
Participant enjoying the testing process:
Participant enjoying the testing process.


Goals

This is the first testing we have done with people who are not from our client's company. Our testers will be similar to new hires of the company who do not have any experience with the Procurement workflow management system before. This allows us to see whether our system is intuitive enough for users.

Documents

As there will be 3 participants within each 30min time slot, we have prepared 3 versions of the task instructions. Each of them will carry out the same tasks, but with different user ID.