HeaderSIS.jpg

Difference between revisions of "IS480 Team wiki: 2015T1 4Sight User Testing 2"

From IS480
Jump to navigation Jump to search
 
(29 intermediate revisions by the same user not shown)
Line 98: Line 98:
  
 
====Number of Clicks====
 
====Number of Clicks====
 +
<br/>
 
[[File:4Sight UT2 Claim1NoOfClicks.png|600px|center|link=]]<br/>
 
[[File:4Sight UT2 Claim1NoOfClicks.png|600px|center|link=]]<br/>
  
Based on the above chart, create appointment results has the largest deviation compared to the other features. Many users encountered the same issue - activation of heat map does not navigate the calendar to the current date. This has caused many confusions and additional clicks to toggle back to the current date heatmap view if they are on a different week view. Based on these feedback, we have made some changes to improve users' experience in using the heat map by removing the unnecessary clicks (toggle to the current date). In the improved version, user will be brought to the current week's heatmap upon selecting "create new appointment" form
+
Based on the above chart, create appointment results has the largest deviation compared to the other features. Many users encountered the same issue - activation of heat map does not navigate the calendar to the current date. This has caused many confusions and additional clicks to toggle back to the current date heatmap view if they are on a different week view. Based on these feedback, we have made some changes to improve users' experience in using the heat map by removing the unnecessary clicks (toggle to the current date). In the improved version, user will be brought to the current week's heatmap upon selecting "create new appointment" form.
  
[[File:4Sight UT2 claim1Changes.png|900px|center|link=]]<br/><br>
+
<br/>
 +
[[File:4Sight UT2 claim1Changes.png|900px|center|link=]]<br/>
  
 
====Number of Errors====
 
====Number of Errors====
Line 112: Line 114:
  
 
====Claim 1 Results====
 
====Claim 1 Results====
[[File:4Sight UT2 claim1Score.png|900px|center|link=]]<br/>
+
<br/>
 +
[[File:4Sight UT2 claim1Score.png|800px|center|link=]]<br/>
  
===Collated Feedback Score===
+
Taking the average of the component scores, the average mean score is 4.39 > 4. This supports our claim that participants find it easy to navigate the calendar and manage patients' appointments (CUD) effectively.
  
<div style="text-align: left; width:99%;  margin-left:auto; margin-right:auto;">
 
{| class="wikitable" width="100%" font-family="Roboto Lt"
 
|-
 
! style="font-family: Roboto Lt; text-align: center; bold;background: #34495e;color:#ecf0f1; width:10px; border:1px solid #999" | #
 
! style="font-family: Roboto Lt; text-align: center; bold;background: #34495e;color:#ecf0f1; width:30px; border:1px solid #999" | Comments on the application
 
! style="font-family: Roboto Lt; text-align: center; bold;background: #34495e;color:#ecf0f1; width:20px; border:1px solid #999" | Participant 1
 
! style="font-family: Roboto Lt; text-align: center; bold;background: #34495e;color:#ecf0f1; width:20px; border:1px solid #999" | Participant 2
 
! style="font-family: Roboto Lt; text-align: center; bold;background: #34495e;color:#ecf0f1; width:20px; border:1px solid #999" | Participant 3
 
! style="font-family: Roboto Lt; text-align: center; bold;background: #34495e;color:#ecf0f1; width:20px; border:1px solid #999" | Overall Mean
 
  
|-
+
<b>Some of the feedback/ comments made by the users are as follow:</b>
| style="text-align:center" | 1
+
* Interface is user-friendly, but one small thing is it’s a bit of a hassle to toggle between Dr. Ho and Dr Goh at some points of time
| style="text-align:center" | The application is easy to use
+
* Flow of entering information is natural and does not feel awkward. However, there must be a choice to save appointment without selecting mode of referral
| style="text-align:center" | 5
+
* It is easy to identify the appointment types with the help of color coding
| style="text-align:center" | 5
+
* I like how easy it is to switch between doctor's calendar and viewing the appointment types on the calendar
| style="text-align:center" | 4
+
 
| style="text-align:center" | 4.67
+
<br/>
 +
===Claim 2 Results===
 +
<b>Claim:</b> Participants find the patient queue useful in helping them to track patients’ attendance (average rating: 4/5)
 +
 
 +
[[File:4Sight UT2 Claim2NoOfClicks.png|700px|center|link=]]<br/>
 +
 
 +
The number of clicks users took to perform the tasks are consistent with the average number of clicks by the members. This shows that the user interface is intuitive and easy to use. There is also no mis-interpretation of the various features in the queue page. Generally all the users found it easy to manage patient queue.
 +
 
 +
[[File:4Sight UT2 Claim2NoOfErrors.png|700px|center|link=]]<br/>
 +
 
 +
There are no errors in most of the tasks in the queue management module. The only error found was that the user could not associate the cross button to add to the no show list initially.
 +
 
 +
[[File:4Sight UT2 claim2Score.png|700px|center|link=]]<br/>
 +
 
 +
Overall the feedback gotten from this module was positive. The average score across all the tasks was 4.47 > 4, supporting our claim of participants finding the patient queue useful in helping them to track patients' attendance.
 +
 
 +
<b>Below are some of the feedback from users:</b>
 +
* No show is good for keeping track of patient
 +
* The patient queue page is very easy to use
 +
* The no show feature helps save a lot of manual work required and time spent on compiling a list of no show list. The current work process requires us to manually check the appointments in the black book against the CMS and marketing form google spreadsheet that is managed by the marketing team. This is highly tedious and error prone.
 +
 
 +
===Claim 3 Results===
 +
<b>Claim:</b> Participants find the marketing conversion and appointment scheduling dashboard beneficial in their performance analysis
 +
 
 +
[[File:4Sight UT2 claim3Score.png|700px|center|link=]]<br/>
  
|-
+
For this claim, we are interested in finding out on the usability of the analytics module. Taking the average of the scores across different tasks, we have gotten an average of 3.975 < 4. This is slightly less than what we have claimed. Some of the reasons we have gathered that explained the scores given are as follow:
| style="text-align:center" | 2
+
* Include feature for exporting to excel (table)
| style="text-align:center" | The user interface is intuitive
+
* Analytics dashboards are useful but some of the charts are quite difficult to interpret
| style="text-align:center" | 4
 
| style="text-align:center" | 5
 
| style="text-align:center" | 4
 
| style="text-align:center" | 4.33
 
  
|-
+
===Claim 4 Results===
| style="text-align:center" | 2
+
<b>Claim:</b> Participants like and have positive comments about our user interface (average rating: 4/5)
| style="text-align:center" | The color scheme is appropriate
 
| style="text-align:center" | 5
 
| style="text-align:center" | 5
 
| style="text-align:center" | 4
 
| style="text-align:center" | 4.67
 
|}
 
</div>
 
  
 +
[[File:4Sight UT2 claim4Score.png|700px|center|link=]]<br/>
  
<center>
+
In general, users like our user interface and have positive feedback about it.
[[File:4Sight UserTestingFeedbackScore.JPG|570px|link=]]
 
[[File:4Sight UserTestingFeedbackMeanScore.JPG|500px|link=]]<br/>
 
</center>
 
  
 +
===Collated Feedback Score - Survey Questionnaire===
  
 
<div style="text-align: left; width:99%;  margin-left:auto; margin-right:auto;">
 
<div style="text-align: left; width:99%;  margin-left:auto; margin-right:auto;">
 
{| class="wikitable" width="100%" font-family="Roboto Lt"
 
{| class="wikitable" width="100%" font-family="Roboto Lt"
 
|-
 
|-
! style="font-family: Roboto Lt; text-align: center; bold;background: #34495e;color:#ecf0f1; width:30px; border:1px solid #999" | #
+
! style="font-family: Roboto Lt; text-align: center; bold;background: #34495e;color:#ecf0f1; width:5%; border:1px solid #999" | #
! style="font-family: Roboto Lt; text-align: center; bold;background: #34495e;color:#ecf0f1; width:40px; border:1px solid #999" | Comments on features of application
+
! style="font-family: Roboto Lt; text-align: center; bold;background: #34495e;color:#ecf0f1; width:30%; border:1px solid #999" | Question
! style="font-family: Roboto Lt; text-align: center; bold;background: #34495e;color:#ecf0f1; width:50px; border:1px solid #999" | Collated feedback
+
! style="font-family: Roboto Lt; text-align: center; bold;background: #34495e;color:#ecf0f1; width:60%; border:1px solid #999" | Collated Comments
 
 
 
|-
 
|-
 
| style="text-align:center" | 1
 
| style="text-align:center" | 1
| style="text-align:center" | Heatmap
+
| style="text-align:center" | What do you like about the application?
| style="text-align:center" | color coding of heatmap is good. <br> Prefer if timing is available.<br> Current layout of application is good.<br> Do not need to enable iSchedule on main calendar page
+
| style="text-align:center" | Very simple to use, simple to understand <br> Will make us efficient <br> Ease of use for essential tasks. Runs smoothly with no lag <br> I like the analytics dashboards as it simplifies the current process where a lot of my work was done manually, across multiple channels (CMS, paper, online shared doc and black book). An integrated platform like this helps reduce my workload <br> The app is easy to use. It will save us a lot of time especially in managing the appointments <br> Very simple to use and many of the features built are what we need <br> Very clear and nice <br> Very easy to use. Reduces human error, making us more efficient
  
 
|-
 
|-
 
| style="text-align:center" | 2
 
| style="text-align:center" | 2
| style="text-align:center" | Suggested timings
+
| style="text-align:center" | What do you not like about the application?
| style="text-align:center" | Allow for customization of suggested timings
+
| style="text-align:center" | Disable auto swap? Maybe can prompt to swap appointment when you delete appointment on crowded days <br> Learning curve for swap waiting list patients is steep <br> Auto sending out of appointment reminder. Some patients might not like it <br> Quite hard to understand and use the swap appointment function
  
 
|-
 
|-
 
| style="text-align:center" | 3
 
| style="text-align:center" | 3
| style="text-align:center" | Swapping of appointments
+
| style="text-align:center" | Comment on how easy it is to use the application
| style="text-align:center" | Very simple to use. <br> This feature is very useful as they do KIV patient's preferred time slot now. However, there are often negligence on the front desk staff and patient's request is often not followed up with a reply on any possibilities of a swap.
+
| style="text-align:center" | Easy to navigate around and update appointments <br> Application is intuitive <br> very easy to navigate
  
 
|-
 
|-
 
| style="text-align:center" | 4
 
| style="text-align:center" | 4
| style="text-align:center" | Any areas of improvement?
+
| style="text-align:center" | Comment on the overall usefulness of the application in helping you complete your daily tasks?
| style="text-align:center" | Include export function for graph<br>Show appointment type in search results<br>Check for international number as we do have patients from abroad<br>Refresh waiting list page on swap
+
| style="text-align:center" | Compared to the current work process, this application greatly reduces the time needed for us to complete some tasks such as calling patients to remind them of appointment, collating the patient no show list etc. <br> Eliminates many of the manual work that we have been doing. Save us a lot of time and reduces error especially during peak periods. <br> Maintains old function of our appointments book, and can be updated <br> I like the analytics dashboards as it simplifies the current process where a lot of my work was done manually, across multiple channels (CMS, paper, online shared doc and black book). An integrated platform like this helps reduce my workload <br> Streamlined work process <br> Charts provided are useful as it allows drilling down of information <br> Application reflects current functions but make a lot easier
 
 
|-
 
| style="text-align:center" | 5
 
| style="text-align:center" | Any thing you will like to see in the application?
 
| style="text-align:center" | Overall list of patients and if they came for the appointment (appointment log)<br> Blocking of time slots on calendar. This will be useful as it will prevent staff from making an appointment on days when doctors are on leave/ MC <br> Allow for adding of more doctors to calendar view <br>Is it possible to click on a time slot in the calendar and it will open the drawer with the create new appointment form, without me having to click on the create new appointment button and choosing the date and time <br> Drag and drop feature for appointments. eg: editing <br>
 
Include UI to manually input marketing expenditure for analytics dashboard <br> Add another appointment type called eyecare/ miscellaneous for general eye checkup etc. <br>
 
 
|}
 
|}
 
</div>
 
</div>
 
  
 
===User Study Summary===
 
===User Study Summary===
* Overall, users find the application easy to user, intuitive to perform an appointment booking
+
* Overall, users find the application easy to user and intuitive to perform all required tasks
 
* Usability scores did not vary drastically from user to user
 
* Usability scores did not vary drastically from user to user
* In general, users are satisfied with most functions, except for a few
+
* Number of clicks are fairly consistent compared to the members
* User feedback for important features are fairly useful
+
* Users are satisfied and have positive comments on all the features of the application
 +
* User provided good feedback for improvement
  
 
===User Study Takeaways===
 
===User Study Takeaways===
* A consistency in usability for different users. Consider that different users have different functions to perform in their day-to-day operation.
+
* Consistent usability score for different users, considering that different users have different functions to perform in their day-to-day operation.
* With users' feedback, we identified areas of improvement on these functions. Eg. Include user confirmation upon deleting an appointment
+
* With users' feedback, we identified areas of improvement on these functions. Eg. export function.
* Allow us to evaluate the usefulness of these features. Consideration on improvising the features and removing features that are redundant to users. Eg. Form for marketing expenditure. Most feedback has been accounted for in our implementation plan, which will then be confirmed into subsequent development.
+
* Allow us to evaluate the usefulness of these features. Team will improve on the features based on users feedback and manage all change requests (remove/ add feature). Most feedback has been documented and accounted for in our implementation plan in which the team will work on it in subsequent development.

Latest revision as of 09:33, 16 November 2015

4Sight team logo.png
4Sight Home.png HOME   4Sight Team.png ABOUT US   4Sight Project overview.png PROJECT OVERVIEW   4Sight Project management.png PROJECT MANAGEMENT   4Sight Documentation.png DOCUMENTATION  
Technical Diagrams Design Documents Meeting Minutes User Testings Project Documents

Venue

  • Location: Clearvision @ 6 Nutmeg Road
  • Date: 16 October 2015 (Wednesday)
  • Time: 15:00pm

Objectives

  • Verify that functionalities built are in line with user requirements
  • Determine if the user interface is intuitive
  • Gather feedback on current functionalities
  • Identify usability problems

Claims

Our team made 4 claims -
1. Participants find it easy to navigate the calendar and manage patients’ appointments (CUD) effectively (average rating: 4/5)
2. Participants find the patient queue useful in helping them track patient’s attendance (average rating: 3.5/5)
3. Participants find the marketing conversion and appointment scheduling dashboard beneficial in their performance analysis (average rating: 4/5)
4. Participants like and have positive comments about our user interface (average rating: 4/5)

Participants

  • Number of participants: 8
  • Roles of participants: Optometrist, admin clerk and marketing team member


IMG 1365.JPG IMG 1358.JPG IMG 1353.JPG IMG 1367.JPG

Tasks

Each of the users was given a list of tasks to complete. Click here to download tasklist. The tasklist also consists of some questionnaire to gather feedback from users.

Summary of Survey Results

Click here to download the raw data.

Usability Results

4Sight UT2 UsabilityScore.png



4Sight UT2 UsabilityScore2.png


Usability Analysis

  • No drastic variation in the usability scores between users of different roles
  • Simple UI that allows user to perform fundamental tasks easily
  • Confirmed the value of proposed features such as swap list
  • Discover potential improvement eg: Block off doctor’s time slots on calendar

Claim 1 Results

Claim: Participants find it easy to navigate the calendar and manage patients' appointments (CUD) effectively (average rating: 4/5)

Number of Clicks


4Sight UT2 Claim1NoOfClicks.png


Based on the above chart, create appointment results has the largest deviation compared to the other features. Many users encountered the same issue - activation of heat map does not navigate the calendar to the current date. This has caused many confusions and additional clicks to toggle back to the current date heatmap view if they are on a different week view. Based on these feedback, we have made some changes to improve users' experience in using the heat map by removing the unnecessary clicks (toggle to the current date). In the improved version, user will be brought to the current week's heatmap upon selecting "create new appointment" form.


4Sight UT2 claim1Changes.png


Number of Errors

4Sight UT2 Claim1NoOfErrors.png


Based on the above chart, create and update appointment have the highest number of errors, 3 and 2 respectively. Many users encountered the same issue - Participants change dropdown selections before heat map finishes loading. Based on the feedback we have gotten, we have made improvement to prevent users from selecting the options in the dropdown too quickly.

4Sight UT2 claim2Changes.png


Claim 1 Results


4Sight UT2 claim1Score.png


Taking the average of the component scores, the average mean score is 4.39 > 4. This supports our claim that participants find it easy to navigate the calendar and manage patients' appointments (CUD) effectively.


Some of the feedback/ comments made by the users are as follow:

  • Interface is user-friendly, but one small thing is it’s a bit of a hassle to toggle between Dr. Ho and Dr Goh at some points of time
  • Flow of entering information is natural and does not feel awkward. However, there must be a choice to save appointment without selecting mode of referral
  • It is easy to identify the appointment types with the help of color coding
  • I like how easy it is to switch between doctor's calendar and viewing the appointment types on the calendar


Claim 2 Results

Claim: Participants find the patient queue useful in helping them to track patients’ attendance (average rating: 4/5)

4Sight UT2 Claim2NoOfClicks.png


The number of clicks users took to perform the tasks are consistent with the average number of clicks by the members. This shows that the user interface is intuitive and easy to use. There is also no mis-interpretation of the various features in the queue page. Generally all the users found it easy to manage patient queue.

4Sight UT2 Claim2NoOfErrors.png


There are no errors in most of the tasks in the queue management module. The only error found was that the user could not associate the cross button to add to the no show list initially.

4Sight UT2 claim2Score.png


Overall the feedback gotten from this module was positive. The average score across all the tasks was 4.47 > 4, supporting our claim of participants finding the patient queue useful in helping them to track patients' attendance.

Below are some of the feedback from users:

  • No show is good for keeping track of patient
  • The patient queue page is very easy to use
  • The no show feature helps save a lot of manual work required and time spent on compiling a list of no show list. The current work process requires us to manually check the appointments in the black book against the CMS and marketing form google spreadsheet that is managed by the marketing team. This is highly tedious and error prone.

Claim 3 Results

Claim: Participants find the marketing conversion and appointment scheduling dashboard beneficial in their performance analysis

4Sight UT2 claim3Score.png


For this claim, we are interested in finding out on the usability of the analytics module. Taking the average of the scores across different tasks, we have gotten an average of 3.975 < 4. This is slightly less than what we have claimed. Some of the reasons we have gathered that explained the scores given are as follow:

  • Include feature for exporting to excel (table)
  • Analytics dashboards are useful but some of the charts are quite difficult to interpret

Claim 4 Results

Claim: Participants like and have positive comments about our user interface (average rating: 4/5)

4Sight UT2 claim4Score.png


In general, users like our user interface and have positive feedback about it.

Collated Feedback Score - Survey Questionnaire

# Question Collated Comments
1 What do you like about the application? Very simple to use, simple to understand
Will make us efficient
Ease of use for essential tasks. Runs smoothly with no lag
I like the analytics dashboards as it simplifies the current process where a lot of my work was done manually, across multiple channels (CMS, paper, online shared doc and black book). An integrated platform like this helps reduce my workload
The app is easy to use. It will save us a lot of time especially in managing the appointments
Very simple to use and many of the features built are what we need
Very clear and nice
Very easy to use. Reduces human error, making us more efficient
2 What do you not like about the application? Disable auto swap? Maybe can prompt to swap appointment when you delete appointment on crowded days
Learning curve for swap waiting list patients is steep
Auto sending out of appointment reminder. Some patients might not like it
Quite hard to understand and use the swap appointment function
3 Comment on how easy it is to use the application Easy to navigate around and update appointments
Application is intuitive
very easy to navigate
4 Comment on the overall usefulness of the application in helping you complete your daily tasks? Compared to the current work process, this application greatly reduces the time needed for us to complete some tasks such as calling patients to remind them of appointment, collating the patient no show list etc.
Eliminates many of the manual work that we have been doing. Save us a lot of time and reduces error especially during peak periods.
Maintains old function of our appointments book, and can be updated
I like the analytics dashboards as it simplifies the current process where a lot of my work was done manually, across multiple channels (CMS, paper, online shared doc and black book). An integrated platform like this helps reduce my workload
Streamlined work process
Charts provided are useful as it allows drilling down of information
Application reflects current functions but make a lot easier

User Study Summary

  • Overall, users find the application easy to user and intuitive to perform all required tasks
  • Usability scores did not vary drastically from user to user
  • Number of clicks are fairly consistent compared to the members
  • Users are satisfied and have positive comments on all the features of the application
  • User provided good feedback for improvement

User Study Takeaways

  • Consistent usability score for different users, considering that different users have different functions to perform in their day-to-day operation.
  • With users' feedback, we identified areas of improvement on these functions. Eg. export function.
  • Allow us to evaluate the usefulness of these features. Team will improve on the features based on users feedback and manage all change requests (remove/ add feature). Most feedback has been documented and accounted for in our implementation plan in which the team will work on it in subsequent development.