HeaderSIS.jpg

IS480 Team wiki:2018T1 FORD USER TESTING

From IS480
Jump to navigation Jump to search
FordLogo.png

Ford home logo.png   HOME

 

Ford about us.png   ABOUT US

 

Ford project overview.png   PROJECT OVERVIEW

 

Ford project management.png   PROJECT MANAGEMENT

 

Ford documentation.png   DOCUMENTATION

 



UAT 1

Testing Plan

  • Date: 6 Aug 2018
  • Iteration: 5
  • Location: PSA Help Desk Office
  • Participants: 6 PSA help desk staff
  • Testing system: Windows
  • Testing browser: Google Chrome

Objectives

  • To test all the developed functionalities for both chatbot and dashboard
  • To test the accuracy rate of the chatbot
  • To test whether the functionalities and user experience meet expectations of the clients

Scope

Chatbot Admin Portal
Portnet Related Authentication
Enquiries Dashboard - Hits Analysis
API Module Dashboard - Category Analysis
Account Management - Change Password
Account Management - Add New User
Data Management - Bootstrap

Test Instruction Form

  • Admin Portal Testing: Instructions can be found here
  • Chatbot Testing: Instructions can be found here

Key Comments Raised by the Help Desk Staff

Chatbot Chatbot logic need to be improved to handle some corner questions, such as specific error code.
Admin Portal 1. Dashboard should be more informative, eg dual axis for hit analysis graph

2. Time frame filtering function should be added, which allows users to filter by day, week, year

Overall Result

  • The admin portal is fully functioning with 0 bug detected.
  • There are minor logic errors found for chatbot, which will be fixed during next iteration.

UAT 2

Testing Plan

  • Date:27 September 2018 - 1 October
  • Iteration:9
  • Location: PSA Help Desk Office
  • Participants: 6 PSA help desk staff
  • Testing system: Windows
  • Testing browser: Google Chrome

Objectives

  • To test all the developed functionalities for both Chatbot and dashboard
  • To test the accuracy rate of the Chatbot
  • To test whether the functionalities and user experience meet the expectations of the clients

Scope

Chatbot Admin Portal
Portnet Related Enquiries Authentication
Provide User Guide Dashboard (Hits Analysis, Category Analysis, Top Question Analysis)
Check Vessel/Container Status FAQ Database Modification on UI
Feedback FAQ Database Filter by Category
Feedback Management

Test Instruction Form

  • Admin Portal Testing: Instructions can be found here
  • Chatbot Testing: Instructions can be found here

Key Comments Raised by the Help Desk Staff

Chatbot 1. User guide should be opened in a new tab instead of the original page, since the original user guide page contains the chat history
2. Submission of the feedback should not refresh the chatbot page, which will clear the chat history

3. The Chatbot did not provide the answer for some of the testers’ queries

Overall Result

  • The admin portal is fully functioning with 0 bug detected. Its performance meets the expectation of the clients.
  • Minor user experience issues are identified for chatbot, and the accuracy rate can be further improved. These changes will be discussed and implemented during the following iterations.

UAT 3

Testing Plan

  • Date:1 November - 15 November 2018
  • Iteration:12
  • Location: PSA Help Desk Office
  • Participants: 6 PSA help desk staff
  • Testing system: Windows
  • Testing browser: Google Chrome

Objectives

  • To test all the developed functionalities for both chatbot and dashboard
  • To test the accuracy rate of the chatbot
  • To test whether the functionalities and user experience meet the expectations of the clients

Scope

Chatbot Admin Portal
HTTPS Connection Authentication
User Instruction Button Dashboard (Hits Analysis, Category Analysis, Top Question Analysis)
Portnet Related Enquiries FAQ Database Modification on UI
Provide User Guide FAQ Database Filter by Category
Check Vessel/Container Status Announcement Management
Feedback Unresolved Question Analysis

Test Instruction Form

  • Admin Portal Testing: Instructions can be found here
  • Chatbot Testing: Instructions can be found here

Key Comments Raised by the Help Desk Staff

Chatbot 1. It will be more useful if the Chatbot can interpret the users’ query better.
Admin Portal 1. Auto-cleaning of expired announcements will be useful, as the help desk staff will not need to manually clear expired announcements.
2. The dashboard is useful to analyse the chatbot performance and user behaviour patterns.
3. The dashboard is useful to manage the FAQ database.

Overall Result

  • The helpdesk staff are satisfied with the dashboard and data management functions provided by the admin portal. However, they feel the accuracy rate of the chatbot can be further improved.