HeaderSIS.jpg

Difference between revisions of "IS480 Team wiki: 2018T1 Kaki&Kin/Documentations/Testing"

From IS480
Jump to navigation Jump to search
Line 148: Line 148:
 
<br>
 
<br>
 
<p><h1>Finals User Acceptance Testing</h1></p>
 
<p><h1>Finals User Acceptance Testing</h1></p>
'''Finals Testing will be conducted with the client on 16 November 2018. Results of testing will be updated here accordingly.'''
+
{| class="wikitable"
 +
! colspan="4" style="text-align: center; font-weight:bold; font-family:'Lucida Sans Unicode', 'Lucida Grande', sans-serif !important;; background-color:#5086d2; color:#ffffff;" | Overview
 +
|-
 +
| colspan="4" style="font-family:'Lucida Sans Unicode', 'Lucida Grande', sans-serif !important;; background-color:#ffffff;" | 1. Conducted at KeyReply’s office on 19 November 2018
 +
|-
 +
| colspan="4" style="font-family:'Lucida Sans Unicode', 'Lucida Grande', sans-serif !important;; background-color:#ffffff;" | 2. Data set used was client's own private data
 +
|-
 +
| colspan="4" style="font-family:'Lucida Sans Unicode', 'Lucida Grande', sans-serif !important;; background-color:#ffffff;" | 3. Kaki&Kin’s solution is an improvement for both Workflow 1 & Workflow 2
 +
|-
 +
| colspan="4" style="text-align: center; font-weight:bold; font-family:'Lucida Sans Unicode', 'Lucida Grande', sans-serif !important;; background-color:#5086d2; color:#ffffff;" | Test Subjects from KeyReply
 +
|-
 +
| colspan="4" style="font-family:'Lucida Sans Unicode', 'Lucida Grande', sans-serif !important;; background-color:#ffffff;" | 1. CTO
 +
|-
 +
| colspan="4" style="font-family:'Lucida Sans Unicode', 'Lucida Grande', sans-serif !important;; background-color:#ffffff;" | 2. Chatbot Developer
 +
|-
 +
| colspan="4" style="text-align: center; font-weight:bold; font-family:'Lucida Sans Unicode', 'Lucida Grande', sans-serif !important;; background-color:#5086d2; color:#ffffff;" | Workflow 1
 +
|-
 +
| colspan="4" style="font-family:'Lucida Sans Unicode', 'Lucida Grande', sans-serif !important;; background-color:#ffffff;" | 1. Time spent processing a file with 400+ utterances takes around 3 hours manually. The AIR system by Kaki&Kin takes around an hour
 +
|-
 +
| colspan="4" style="font-family:'Lucida Sans Unicode', 'Lucida Grande', sans-serif !important;; background-color:#ffffff;" | 2. Process is less painful with the removal of manual typing and eyeballing
 +
|-
 +
| colspan="4" style="font-family:'Lucida Sans Unicode', 'Lucida Grande', sans-serif !important;; background-color:#ffffff;" | 3. General clustering gives good overall view of data
 +
|-
 +
| colspan="4" style="font-family:'Lucida Sans Unicode', 'Lucida Grande', sans-serif !important;; background-color:#ffffff;" | 4. Project shows good potential but is not yet ready for live use
 +
|-
 +
| colspan="4" style="font-family:'Lucida Sans Unicode', 'Lucida Grande', sans-serif !important;; background-color:#ffffff;" | 5. Splitting of intents & merging of intents is not yet ready but can be done using workarounds
 +
|-
 +
| colspan="4" style="text-align: center; font-weight:bold; font-family:'Lucida Sans Unicode', 'Lucida Grande', sans-serif !important;; background-color:#5086d2; color:#ffffff;" | Workflow 2
 +
|-
 +
| colspan="4" style="font-family:'Lucida Sans Unicode', 'Lucida Grande', sans-serif !important;; background-color:#ffffff;" | 1. Roughly the same time savings as Workflow 1
 +
|-
 +
| colspan="4" style="font-family:'Lucida Sans Unicode', 'Lucida Grande', sans-serif !important;; background-color:#ffffff;" | 2. Process is less painful as user can now directly train on the chat logs instead of eyeballing
 +
|-
 +
| colspan="4" style="font-family:'Lucida Sans Unicode', 'Lucida Grande', sans-serif !important;; background-color:#ffffff;" | 3. Automatic reassignment of new variations into matching existing intents where possible is highly useful
 +
|-
 +
| colspan="4" style="font-family:'Lucida Sans Unicode', 'Lucida Grande', sans-serif !important;; background-color:#ffffff;" | 4. A potential issue is that new Intents created may be duplicates of existing intents
 +
|}
 
<br><br>
 
<br><br>
 
<br>
 
<br>
Line 549: Line 585:
 
| style="font-size:13px; font-family:'Lucida Sans Unicode', 'Lucida Grande', sans-serif !important;; background-color:#ffffff;" | 0
 
| style="font-size:13px; font-family:'Lucida Sans Unicode', 'Lucida Grande', sans-serif !important;; background-color:#ffffff;" | 0
 
|}
 
|}
 +
<br>
 
<h1><p>Sprint 9 - Sprint 11 (Finals)</p></h1>
 
<h1><p>Sprint 9 - Sprint 11 (Finals)</p></h1>
 
<h3>Unresolved Bug Points: 0</h3>
 
<h3>Unresolved Bug Points: 0</h3>

Revision as of 16:12, 20 November 2018

Kklogo.png
Kkhome.png
Kkau.png
Kkpo.png
Kkpm.png
Kkdsel.png
Kkd1.png
Kkd2.png
Kkd3.png
Kkd4.png
Kkd5.png
KKTest1.png


Acceptance User Testing

Overview
1. Conducted at KeyReply’s office on 14 August 2018
2. Four test subjects had to do a series of tasks on the prototype new system
3. Feedback was recorded from a questionnaire
4. Proposed system was rated as easier to use than current system
Test Subjects from KeyReply
1. CTO
2. Chatbot Trainer
3. Developer
4. UI/UX Designer
Task Recommendation Average Rating (1 - very hard, 6 very easy)
Uploading a text file into the system 1. More explanation of the types of accepted files

2. Show indication of upload progress

4.25
Finding a generated intent 1. Font too small

2. Hard to click

3. Graph lines small and redundant

3
Editing a sentence related to an intent 1. Should tell users that enter key can be used

2. Guide the users to the empty fields

3. Unclear that variations can be deleted

4
Adding alternative sentences for an intent 1. Should tell users that enter key can be used

2. Guiding text is too lengthy

3. No clear indication that fields are for variations

4. Have a button to ‘Add’ new fields

3.75
Adding a chatbot response/answer to an intent 1. Guiding text is too lengthy and unclear

2. Have a button to ‘Add’ answer

3. Not obvious where to input answer

3.5



Mid-Terms User Testing

Overview
1. Conducted at KeyReply’s office on 2 October 2018
2. Majority of feedback were on UI & UX
3. Feedback was recorded from a questionnaire
4. Kaki&Kin’s solution is still rated as being easier to use than existing method
Test Subjects from KeyReply
1. CTO
2. Manager, Engineering
3. UI/UX Designer
Activity / Review Recommendation /

Critical Feedback

Average Rating (1 - worst score 6 - best score) Insights
Upload file 1. More explanation on accepted file formats

2. Progress bar instead of spinner

6 - very easy More guidance could be given on upload file types. The spinner does not provide enough information on upload progress
Keyword accuracy as a representation of Intent 1. Too vague 2.67 - below average Further improvements to be made on NLP
Visualisation Structure 1. Representation is unclear

2. Merge ‘View’ & ‘Manage Content’ tabs

2.67 - below average Improve the current visualisation by making it easier to understand
Deleting an Intent 1. Drag and drop is easy but affected by list length

2. Trash icon button would be easier

3. Intent name should be editable

4.67 - easier than average Current method will be rendered hard to use by a long list of intent. Only allowing deletion of intent without editing intent name is too restrictive
Adding question variations and answers 1. Cannot see which intents are filled

2. Not enough hints and instructions

3. ‘Enter’ & ‘Backspace’ usage could cause input usability problems

2.67 - below average Change method for input and deletion. Show in a clear manner yet-to-be-filled intents.
‘Review’ tab 1. Take note of limitations with large intent numbers

2. Consider having collapsible boxes

3. Colour scheme makes it a bit hard to read

Unrated Take note of UI/UX suggestions


Finals User Acceptance Testing

Overview
1. Conducted at KeyReply’s office on 19 November 2018
2. Data set used was client's own private data
3. Kaki&Kin’s solution is an improvement for both Workflow 1 & Workflow 2
Test Subjects from KeyReply
1. CTO
2. Chatbot Developer
Workflow 1
1. Time spent processing a file with 400+ utterances takes around 3 hours manually. The AIR system by Kaki&Kin takes around an hour
2. Process is less painful with the removal of manual typing and eyeballing
3. General clustering gives good overall view of data
4. Project shows good potential but is not yet ready for live use
5. Splitting of intents & merging of intents is not yet ready but can be done using workarounds
Workflow 2
1. Roughly the same time savings as Workflow 1
2. Process is less painful as user can now directly train on the chat logs instead of eyeballing
3. Automatic reassignment of new variations into matching existing intents where possible is highly useful
4. A potential issue is that new Intents created may be duplicates of existing intents




KKTest2.png


Bug Metrics
Severity Description
Low Impact (1 pt) Unimportant. Typo error or small UI alignment issues
High Impact (5 pts) The system runs. However, some non-critical functionalities are not working.
Critical Impact (10 pts) The system is down or is usable after a short period of time. We have to fix the bugs to continue.
Mitigation Plan
Cumulative Current Unresolved Points Plan
Points <=5 Fix during buffer time
5 < points<10 Fix during planned debugging time
Points >= 10 Stop current development and resolve the bug immediately


Sprint 1 - Sprint 5 (Acceptance)

Unresolved Bug Points: 0

General back-end/NLP
Bug Description Frontend / Backend Tester Date Found Severity Points Allocation Resolved? Cause Action Debugged By Debugged Date Current Points
CORS concerns on backend that does not allow front end to connect Backend Aaron 8-Aug-18 High Impact 5 Resolved Allow(*) for cors, removed subscription option, false for websockets only Went down to KR to understand how they set up GraphQL Li Xuan 9-Aug-18 0
Deployment to heroku unsuccessful Backend Li Xuan 10-Aug-18 High Impact 5 Resolved server.js not found Need to commit js files, not ts and added a procfile Li Xuan & Zachery 13-Aug-18 0
Upload Module
Bug Description Frontend / Backend Tester Date Found Severity Points Allocation Resolved? Cause Action Debugged By Debugged Date Current Points
GraphQL runs on run time Frontend Eugene 16-Aug-18 High Impact 5 Resolved Codes were executed on runtime Execute the 'Skip' method in export function of GraphQL Aaron 16-Aug-18 0
Error thrown when read text file contents Backend Eugene 16-Aug-18 High Impact 5 Resolved The uploaded object was not read as a text object JSON.stringify() Aaron 16-Aug-18 0
Upon reading file, variables were not updated Frontend Eugene 16-Aug-18 High Impact 5 Resolved The GraphQL server skip() function was not used Used the skip() function Aaron 16-Aug-18 0
GraphQL not receiving string Object in Query Frontend Eugene 16-Aug-18 High Impact 5 Resolved The variable required was not converted into a JSON object Convert the variable into JSON object first Aaron 16-Aug-18 0
Edit Visualisation (View Interface)
Bug Description Frontend / Backend Tester Date Found Severity Points Allocation Resolved? Cause Action Debugged By Debugged Date Current Points
Error in data structure of flattened data returned Frontend Eugene 14-Aug-18 High Impact 5 Resolved Parsing and converting to string for JSON Fixed the issues in the JSON template Zachery 16-Aug-18 0
Error on JSON file used for testing Frontend Eugene 14-Aug-18 High Impact 5 Resolved Wrong assignment of variable Changed the assignment of variable Zachery 16-Aug-18 0
Manage Content Node
Bug Description Frontend / Backend Tester Date Found Severity Points Allocation Resolved? Cause Action Debugged By Debugged Date Current Points
Table not updated Frontend Eugene 14-Aug-18 Critical Impact 10 Resolved JSON object deep freezed. Requires parsing and stringify of return object Zachery 16-Aug-18 0



Sprint 6 - Sprint 8 (Mid-Terms)

Unresolved Bug Points: 0

General back-end/NLP
Bug Description Frontend / Backend Tester Date Found Severity Points Allocation Resolved? Cause Action Debugged By Debugged Date Current Points
Tree recursive structure not compatible with front-end Both Li Xuan 25-Sep-18 Critical Impact 10 Resolved JSON data structure mismatch Matched the JSON data structure Li Xuan 26-Sep-18 0
Edit Visualisation (View Interface)
Bug Description Frontend / Backend Tester Date Found Severity Points Allocation Resolved? Cause Action Debugged By Debugged Date Current Points
Flatten tree algorithm loses some children intents Frontend Eugene 25-Sep-18 High Impact 5 Resolved Recursion issue Redo recursion code Li Xuan & Gabriel 26-Sep-18 0
Default template appears sometimes instead of generated intents Frontend Eugene 2-Oct-18 High Impact 5 Resolved File has not finished uploading when the 'Continue' button was clicked. Therefore, the default template loaded instead. Show a circle loading animation while the file is being uploaded and do not allow the 'Continue' button to be clicked until the upload has finished Zachery 2-Oct-18 0
Not every child on lowest level is shown Frontend Li Xuan 2-Oct-18 High Impact 5 Resolved Recursion issue Issue was linked to earlier issue where flattened tree algorithm lost children Li Xuan & Gabriel 26-Sep-18 0
Some intents are generated even without Most representative text and variations Frontend Eugene 2-Oct-18 High Impact 0 Resolved Checking for index > 0 instead of >= 0 Removed check for index and print everything Zachary 12-Oct-18 0
Manage Content Node
Bug Description Frontend / Backend Tester Date Found Severity Points Allocation Resolved? Cause Action Debugged By Debugged Date Current Points
Tree does not populate with variations of the same category by default Frontend Eugene 18-Sep-18 Critical Impact 10 Resolved Printing issue Printing from the tagged sentences of each child Zachery 18-Sep-18 0
Editing 1 variation edits the entire parents tree Frontend Zachery 18-Sep-18 Critical Impact 10 Resolved Error in method to find children intents Resolved and ensured that the method to find children does not affect parent Zachery 20-Sep-18 0
Deleting 1 variation deletes some other variations as well Frontend Eugene 25-Sep-18 High Impact 5 Resolved Array Slicing Limiting the array slicing to 1 object Zachery 26-Sep-18 0
Unclickable first child Frontend Zachary 2-Oct-18 High Impact 5 Resolved List issue Changing :list="list" to :list="data.Selected" Zachery 12-Oct-18 0


Sprint 9 - Sprint 11 (Finals)

Unresolved Bug Points: 0

General back-end/NLP
Bug Description Frontend / Backend Tester Date Found Severity Points Allocation Resolved? Cause Action Debugged By Debugged Date Current Points
Data Cleansing does not remove '-' that are the first character on line or last character on line Backend Eugene 24-Oct-18 Low Impact 1 Resolved Regex issue Changed the regex expression Eugene 28-Oct-18 0
Some lines merge during data cleansing Backend Eugene 30-Oct-18 High Impact 5 Resolved Suspected incorrect Regex expression Changed the Regex expression in method remove_surrounding_colon_http_chat Eugene 30-Oct-18 0
Entry of chatlog with more than one sentence will be split in workflow 1 Backend Li Xuan 8-Nov-18 Critical Impact 10 Resolved Delimit by fullstops Merged related sentences together Li Xuan 15-Nov-18 0


KKTest3.png