HeaderSIS.jpg

Difference between revisions of "IS480 Team wiki: 2013T1 Bankrevels Testing"

From IS480
Jump to navigation Jump to search
 
(43 intermediate revisions by 2 users not shown)
Line 22: Line 22:
 
|-
 
|-
 
! style="border-style: solid; border-width: 0px 0px 2px 0px; width: 12%; background-color:#FFFFFF"|  
 
! style="border-style: solid; border-width: 0px 0px 2px 0px; width: 12%; background-color:#FFFFFF"|  
[[IS480 Team wiki: 2013T1 Bankrevels Project Documentation | <span style="color:#999999;">Design & Prototyping</span>]]
+
[[IS480 Team wiki: 2013T1 Bankrevels Project Documentation | <span style="color:#999999">Design & Prototyping (Mobile)</span>]]
 +
! style="border-style: solid; border-width: 0px 0px 2px 0px; width: 15%; background-color:#FFFFFF"|
 +
[[IS480 Team wiki: 2013T1 Bankrevels Design & Prototyping (Internet) | <span style="color:#999999;">Design & Prototyping (Internet)</span>]]
 
! style="border-style: solid; border-width: 0px 0px 2px 0px; width: 12%; background-color:#FFFFFF"|  
 
! style="border-style: solid; border-width: 0px 0px 2px 0px; width: 12%; background-color:#FFFFFF"|  
[[IS480 Team wiki: 2013T1 Bankrevels System Design | <span style="color:#999999">System Design</span>]]
+
[[IS480 Team wiki: 2013T1 Bankrevels Heuristic Evaluation| <span style="color:#999999">Heuristic Evaluation</span>]]
 +
! style="border-style: solid; border-width: 0px 0px 2px 0px; width: 12%; background-color:#FFFFFF"|
 +
[[IS480 Team wiki: 2013T1 Bankrevels Diagrams| <span style="color:#999999">Diagrams</span>]]
 +
! style="border-style: solid; border-width: 0px 0px 2px 0px; width: 15%; background-color:#FFFFFF"|
 +
[[IS480 Team wiki: 2013T1 Bankrevels Testing | <span style="color:#999999;">Persona & Scenarios</span>]]
 
! style="border-style: solid; border-width: 1px 1px 2px 1px; width: 15%; background-color:#FFFFFF"|  
 
! style="border-style: solid; border-width: 1px 1px 2px 1px; width: 15%; background-color:#FFFFFF"|  
[[IS480 Team wiki: 2013T1 Bankrevels Testing | <span style="color:#000000">Testing</span>]]
+
[[IS480 Team wiki: 2013T1 Bankrevels Testing | <span style="color:#000000;">Testing</span>]]
! style="border-style: solid; border-width: 0px 0px 2px 0px; width: 15%; background-color:#FFFFFF"|
 
[[IS480 Team wiki: 2013T1 Bankrevels Screenshot/Videos | <span style="color:#999999;">Screenshots & Videos</span>]]
 
! style="border-style: solid; border-width: 0px 0px 2px 0px; width: 15%; background-color:#FFFFFF"|
 
[[IS480 Team wiki: 2013T1 Bankrevels Research | <span style="color:#999999;">Research</span>]]
 
 
! style="border-style: solid; border-width: 0px 0px 2px 0px; width: 15%; background-color:#FFFFFF"|  
 
! style="border-style: solid; border-width: 0px 0px 2px 0px; width: 15%; background-color:#FFFFFF"|  
 
[[IS480 Team wiki: 2013T1 Bankrevels Administrative | <span style="color:#999999;">Administrative</span>]]
 
[[IS480 Team wiki: 2013T1 Bankrevels Administrative | <span style="color:#999999;">Administrative</span>]]
Line 36: Line 38:
 
|}
 
|}
  
 +
<br>
 
</center>
 
</center>
 +
====<div style=" border-bottom: 1px dotted; background: #FFFFFF; padding: 12px; font-weight: bold; line-height: 0.5em"><font color= black size = 4>User Testing 2 (UT 2)</font></div>====
  
<br>
+
=====Objective=====
 +
The objectives of UT 2 are to compare the proficiency of experienced and inexperienced users to find out if learning curve of our app (RIB and RMB) is large, gather feedback on the ease of using the application and collect overall feedback of the feel and design of the application.
 +
 
 +
=====Devices tested for Android=====
 +
# Samsung S3
 +
# Samsung S4
 +
# HTC Desire X
 +
 
 +
=====Devices tested for iOS=====
 +
# iPhone 4
 +
# iPhone 4S
 +
# iPhone 5
 +
# iPhone 5S
 +
 
 +
=====Documents=====
 +
* [[Media:BR_UT 2_Plan.pdf|UT 2 Execution Plan]]
 +
* [[Media:BR_UT2_Observation_sheet_-_mobile.pdf|Observation Sheet (Mobile)]]
 +
* [[Media:BR_UT2_Observation_sheet_-_internet.pdf‎ |Observation Sheet (Internet)]]
 +
* [[Media:BR_Questionnaire_-_internet.pdf‎|Questionnaire (Internet)]]
 +
* [[Media:BR_Questionnaire_-_mobile.pdf|Questionnaire (Mobile)]]
 +
* [[Media:BR_Tasks_-_internet.pdf|Tasks (Internet)]]
 +
* [[Media:Tasks_-_mobile.pdf‎|Tasks (Mobile)]]
 +
* [[Media:RMB_UT2_collated_results.pdf|Raw Results (Mobile)]]
 +
* [[Media:BR_UT2_RIB collated results.pdf|Raw Results (Internet)]]
 +
* [[Media:BR_UT2_experienced_users.pdf|Experienced Users Raw Results (Mobile and Internet)]]
 +
* [[Media:BR_Quantitative_analysis_RMB.xlsx|Quantitative Analysis (Mobile)]]
 +
* [[Media:BR_Quantitative_analysis_RIB.xlsx|Quantitative Analysis (Internet)]]
 +
* [[Media:BR_UT2_results.pdf|Summary of UT 2 results]]
  
== Overview ==
+
=====Purpose=====
{| class="wikitable"
+
The purpose of the UT 2 is to find out the following from the users:
|-
 
! Test Conducted !! Date(s) !! Venue !! Documentation !! Test Results
 
|-
 
| Heuristic Evaluation <br/>(Paper Prototype) || 27 June 2013 (Thursday), <br/> 29 June 2013 (Saturday) || SMU || HE Execution Plan || HE Compiled Results
 
|-
 
|}
 
  
== Heuristic Evalutation ==
+
# Is the learning curve of our app (internet and mobile) large?
===Objective===
+
# Is the QR code QuikPay function intuitive enough?
The objective of this Heuristic Evaluation is to understand the user-friendliness of the mock-up of the application and identify any potential usability issues.
+
# What do users feel about the QR code login for internet banking?
 +
# Can the user use all the functions of the application with ease?
 +
# How easy it is for the user to navigate through the whole application?
 +
# How does the user feel about the overall design of the application?
 +
<br>
  
===Participants, Timing and Venue held===
+
=====Participants, Timing and Venue held=====
Number of participants: 5 <br/>
+
Number of participants: 20 (10 for Internet Banking, 5 for Android, 5 for iOS) <br/>
Date and Time held: 27 June 2013 (5.30pm-6.30pm) and 29 June 2013 (1pm-2pm) <br/>
+
Date and Time held: 15 November 2013 (11am - 7pm)<br/>
 
Venue: SMU
 
Venue: SMU
  
===Implementation===
+
{|align="center"
Participants are given an overview of the application and the possible scenarios they are able to explore with the current prototype. Participants are guided by the facilitator to walkthrough all the possible scenarios. After exploring the application, participants are required to come up with 10 heuristic problems they encounter.
+
|[[Image:Brut21.jpg|thumb|left|200px|200px]]
 +
|[[Image:Brut22.jpg|thumb|left|200px|200px]]
 +
|[[Image:Brut23.jpg|thumb|left|200px|200px]]
 +
|[[Image:Brut24.jpg|thumb|left|200px|200px]]
 +
|}
 +
<br/>
 +
{|align="center"
 +
|[[Image:Brut25.jpg|thumb|left|200px|200px]]
 +
|[[Image:Brut26.jpg|thumb|left|200px|200px]]
 +
|[[Image:Brut27.jpg|thumb|left|200px|200px]]
 +
|[[Image:Brut28.jpg|thumb|left|200px|200px]]
 +
|}
 +
 
 +
=====Implementation=====
 +
Participants will be welcomed to the user testing by the facilitator. Thereafter, participants will be given a brief overview of the application. Participants will then be given the specific tasks that they are required to complete without the guidance of the facilitator. In face of any problems that they are unable to resolve themselves, participants can ask the facilitator for help. The problem faced by the participants will then be recorded down the Observation Sheet provided for each facilitator.
 +
<br><br>
 +
After going through the user testing of the application, participants are required to complete an evaluation of the application on the Questionnaire provided. Participants are encouraged to discuss any problems they encounter with the facilitator.
 +
<br><br>
 +
The User Testing will end after the completion of the Questionnaire. The facilitator will then thank the participants for their time and feedback provided.
  
===Functions Tested===
+
=====Functions Tested=====
 
{| class="wikitable" style="text-align: center; height:25px"
 
{| class="wikitable" style="text-align: center; height:25px"
 
|+  
 
|+  
Line 69: Line 116:
 
|-
 
|-
 
!scope="row"  style=" text-align: center;"| 1
 
!scope="row"  style=" text-align: center;"| 1
|View account balance
+
|Login 2FA (RIB Mobile login 2FA)
 
|-
 
|-
 
!scope="row"  style=" text-align: center;"| 2
 
!scope="row"  style=" text-align: center;"| 2
|View transaction history
+
|View account information
 
|-
 
|-
 
!scope="row"  style=" text-align: center;"| 3
 
!scope="row"  style=" text-align: center;"| 3
|Manage fund transfer
+
|View transaction history
 
|-
 
|-
 
!scope="row"  style=" text-align: center;"| 4
 
!scope="row"  style=" text-align: center;"| 4
|Verify user identity using 2FA
+
|Immediate Fund Transfer
 
|-
 
|-
 
!scope="row"  style=" text-align: center;"| 5
 
!scope="row"  style=" text-align: center;"| 5
|Manage bill payments
+
|Standing Instructions Fund Transfer
 
|-
 
|-
 
!scope="row"  style=" text-align: center;"| 6
 
!scope="row"  style=" text-align: center;"| 6
|Create a new loan
+
|Bill Payment to billing organizations
 
|-
 
|-
 
!scope="row"  style=" text-align: center;"| 7
 
!scope="row"  style=" text-align: center;"| 7
|Manage stocks
+
|GIRO arrangement with billing organizations
 +
|-
 +
!scope="row"  style=" text-align: center;"| 8
 +
|QuikPay QR Code (for mobile only)
 +
|-
 +
!scope="row"  style=" text-align: center;"| 9
 +
|View and edit profile settings (for web only)
 +
|-
 +
!scope="row"  style=" text-align: center;"| 10
 +
|Loan Calculator (Calculate Loan only)
 +
|-
 +
|}
 +
 
 +
=====Data Collected=====
 +
{| class="wikitable centered" width="90%"
 +
!Qualitative Data
 +
!Quantitative Data
 +
 
 +
|-
 +
|
 +
'''Questionnaire'''
 +
# Feedback on the overall design of the application
 +
 
 +
''For RMB only''
 +
# Feedback on QR Code QuikPay function
 +
 
 +
''For RIB only''
 +
# Feedback on mobile QR Code login 2FA
 +
 
 +
'''Observation sheet'''
 +
# Recorded general comments from users
 +
 
 +
|
 +
'''Questionnaire'''
 +
# Age
 +
# Gender
 +
# Frequent User (Yes or No)
 +
# Ease of using Accounts Summary & Transaction History (1 - very difficult, 5 - very easy)
 +
# Ease of using Immediate Funds Transfer (1 - very difficult, 5 - very easy)
 +
# Ease of using Standing Instructions Funds Transfer (1 - very difficult, 5 - very easy)
 +
# Ease of using Bill Payment (1 - very difficult, 5 - very easy)
 +
# Ease of using GIRO Arrangement (1 - very difficult, 5 - very easy)
 +
# Ease of using Loan Calculator – Calculate Loan (1 - very difficult, 5 - very easy)
 +
# Ease of navigating through the application (1 - very difficult, 5 - very easy)
 +
# Ratings on the overall design of the application (1 - very poor, 5 - Excellent)
 +
 
 +
''For RMB only''
 +
# Ease of using QR code QuikPay (1 - very difficult, 5 - very easy)
 +
 
 +
''For RIB only''
 +
# Ease of using profile settings (1 - very difficult, 5 - very easy)
 +
# Ease of using mobile QR Code Login 2FA (1 - very difficult, 5 - very easy)
 +
 
 +
'''Observation Sheet'''
 +
# Time taken for each task
 +
 
 
|-
 
|-
 
|}
 
|}
  
===Results and Follow-up===
+
=====Results and Follow Up=====
Problems are compiled and organized according to their severity ratings. Actions to be taken by the team are identified as well.
+
Data collected through the Questionnaire will be organized and aggregate to give a summary of the results to be published on wiki at a later date. Quantitative analysis will be done  to compare the proficiency of experienced and inexperienced users to find out if learning curve of our app is large. A team meeting will be held to discuss the changes to be made.
  
 +
=====Key Findings=====
 +
======Mobile Banking======
 +
* Most users have given positive feedback that using the QR code to do fund transfer is convenient and fast.  However, there are some users who find it a little troublesome while they not used to this mode of funds transfer.
 
{| class="wikitable" style="text-align: center; height:25px"
 
{| class="wikitable" style="text-align: center; height:25px"
 
|+  
 
|+  
 
|-
 
|-
! scope="col"  width="50" style="background:#CCCCCC| No.
+
! scope="col"  width="50" style="background:#CCCCCC| Occurences
! scope="col" width="350" style="background:#CCCCCC| Heuristic Problem
+
! scope="col" width="250" style="background:#CCCCCC| Comments
! scope="col" width="70" style="background:#CCCCCC| Severity Rating
 
! scope="col" width="350" style="background:#CCCCCC| Action to be taken
 
 
|-
 
|-
!scope="row"  style=" text-align: center;"| 1
+
!scope="row"  style=" text-align: center;"| 5
|Add / Remove button in bill payments || 4 || Rectified before HE on 29 June 2013
+
|Convenient
 
|-
 
|-
 
!scope="row"  style=" text-align: center;"| 2
 
!scope="row"  style=" text-align: center;"| 2
|Don’t understand how QuikPay works || 3 || Prototype to be updated with change
+
|Quick/Fast
 +
|-
 +
|}
 +
* Average ratings of the ease of using each functions: 3.8 - 4.3 (Relatively Easy to Very Easy)
 +
* Average ratings for the overall design of the application: 4.0 (Very good)
 +
{| class="wikitable" style="text-align: center; height:25px"
 +
|+
 +
|-
 +
! scope="col"  width="50" style="background:#CCCCCC| Occurences
 +
! scope="col" width="250" style="background:#CCCCCC| Comments
 +
|-
 +
!scope="row"  style=" text-align: center;"| 7
 +
|Easy to use/understand/navigate
 
|-
 
|-
 
!scope="row"  style=" text-align: center;"| 3
 
!scope="row"  style=" text-align: center;"| 3
|No mention of currency for fund transfer || 3 || Prototype to be updated with change
+
|Simple and Neat
 +
|-
 +
!scope="row"  style=" text-align: center;"| 3
 +
|Good use of colours
 +
|-
 +
!scope="row"  style=" text-align: center;"| 2
 +
|Looks nice
 +
|-
 +
|}
 +
Many users reflect that our application easy to use and navigate. Some mentioned our application is visually appealing, not cluttered and uses good colour themes.
 +
 
 +
======Internet Banking======
 +
* Most users have given positive feedback in using the QR code to log in, such as good speed and accuracy and secured, in addition to easy to use and its convenience. However, there are users who felt that it was quite confusing as they are unsure of which QR code they are required to scan at each step.
 +
{| class="wikitable" style="text-align: center; height:25px"
 +
|+
 +
|-
 +
! scope="col"  width="50" style="background:#CCCCCC| Occurences
 +
! scope="col" width="250" style="background:#CCCCCC| Comments
 +
|-
 +
!scope="row"  style=" text-align: center;"| 2
 +
|Easy to use
 +
|-
 +
!scope="row"  style=" text-align: center;"| 2
 +
|Convenient
 +
|-
 +
|}
 +
* Average ratings of the ease of using each functions: 3.5 - 4.4 (Relatively Easy to Very Easy)
 +
* Average ratings for the overall design of the application: 4.2 (Very good)
 +
{| class="wikitable" style="text-align: center; height:25px"
 +
|+
 +
|-
 +
! scope="col"  width="50" style="background:#CCCCCC| Occurences
 +
! scope="col" width="250" style="background:#CCCCCC| Comments
 +
|-
 +
!scope="row"  style=" text-align: center;"| 5
 +
|Easy to use/understand/navigate/clear
 +
|-
 +
!scope="row"  style=" text-align: center;"| 5
 +
|Clean/simple/Neat
 
|-
 
|-
 
!scope="row"  style=" text-align: center;"| 4
 
!scope="row"  style=" text-align: center;"| 4
|User Interface is not consistent || 3 || Prototype to be updated with change
+
|Visually appealing
 
|-
 
|-
!scope="row"  style=" text-align: center;"| 5
+
|}
|Not sure where to find help when face problems || 3 || Problem to be rectified at the later stage of the project, team may publish a PDF manual on how to use app
+
Many users reflect that our application easy to use and navigate, and at the same time, looks clean and neat. Some mentioned our application looks nice and pretty.
 +
 
 +
======Quantitative Analysis======
 +
''Methodology''
 +
<br>To measure if the learning curve of first-time users is great, we compared the timings of inexperienced users versus experienced users in completing assigned tasks. If the timings are similar, then we can conclude that the learning curve is not steep. However, if the inexperienced users take a significantly longer time (i.e. 30 seconds or more), this implies that there is a substantial learning curve.
 +
Our team has decided to use a significance level of 0.05 (5%). A two-sample unequal variance T-test is used as both sets of users are different.
 +
<br><br>
 +
'''H0 (Null Hypothesis):''' There is no (or little) difference between time taken for experienced and inexperienced users to complete tasks assigned to them
 +
<br>
 +
'''H1:''' Experienced users take a much faster time to complete the tasks as compared to inexperienced users
 +
<br><br>
 +
''Results''
 +
<br><br>
 +
[[Image:Br_QA_results1.PNG|1000px]]
 +
[[Image:Br_QA_results2.PNG|1000px]]
 +
 
 +
====<div style=" border-bottom: 1px dotted; background: #FFFFFF; padding: 12px; font-weight: bold; line-height: 0.5em"><font color= black size = 4>User Testing 1 (UT 1)</font></div>====
 +
 
 +
=====Objective=====
 +
The objectives of UT 1 are to understand the intuitiveness, ease of using the application, as well as to get feedback from users on the design and layout of the application on all two platforms (Android and web).
 +
 
 +
=====Devices tested for Android=====
 +
# Samsung S3
 +
# Samsung Note 2
 +
# LG Optimus G
 +
 
 +
=====Documents=====
 +
* [[Media:BR_UT_1_Plan.pdf|UT 1 Execution Plan]]
 +
* [[Media:BR_Observation_sheet_-_android.pdf|Observation Sheet (Android)]]
 +
* [[Media:BR_Observation_sheet_-_internet.pdf|Observation Sheet (Internet)]]
 +
* [[Media:Br_Questionnaire_-_internet.pdf|Questionnaire (Internet)]]
 +
* [[Media:Br_Questionnaire_-_mobile.pdf|Questionnaire (Android)]]
 +
* [[Media:Br_Tasks_-_internet.pdf|Tasks (Internet)]]
 +
* [[Media:Br_Tasks_-_mobile.pdf|Tasks (Android)]]
 +
* [[Media:BR_RMB_collated_results.pdf|Raw Results (Android)]]
 +
* [[Media:BR_RIB_collated_results.pdf|Raw Results (Internet)]]
 +
* [[Media:BR_UT1_results.pdf|Summary of UT 1 results]]
 +
 
 +
=====Purpose=====
 +
The purpose of the UT 1 is to find out the following from the users:
 +
 
 +
# How does user find the login 2FA using QR code instead of the conventional hardware token?
 +
# Is the QuikPay function intuitive enough?
 +
# Can the user use the Fund Transfer function with ease?
 +
# How easy it is for the user to navigate through the whole application?
 +
# How does the user feel about the layout and design of our application?
 +
# What are the user's emotions when using the application?
 +
# What are the user's general impressions of the application?
 +
<br>
 +
 
 +
=====Data Collected=====
 +
{| class="wikitable centered" width="90%"
 +
!Qualitative Data
 +
!Quantitative Data
 +
 
 
|-
 
|-
!scope="row"  style=" text-align: center;"| 6
+
|
|Unclear cancel button for the fund transfer function|| 3 || Prototype to be updated with change
+
'''Questionnaire'''
 +
# Feedback on the layout of Account Summary Page
 +
# Feedback on the design of Account Summary Page
 +
# Feedback on the improvements for Account Summary Page
 +
# Feedback on the layout of Transaction History Page
 +
# Feedback on the design of Transaction History Page
 +
# Feedback on the improvements for Transaction History Page
 +
# Feedback on the things they like
 +
# Feedback on the things they dislike
 +
# Feedback on general impressions of the application
 +
 
 +
'''Observation sheet'''
 +
# Recorded comments on Login 2FA
 +
# Recorded comments on Funds Transfer
 +
# Recorded feelings of user throughout the test
 +
# Recorded general observations of users
 +
 
 +
''For RMB only''
 +
# Recorded comments Login 2FA
 +
# Recorded comments on QuikPay
 +
 
 +
|
 +
'''Questionnaire'''
 +
# Age
 +
# Gender
 +
# Frequent User (Yes or No)
 +
# Ease of using Funds Transfer (1 - very difficult, 5 - very easy)
 +
# Ease of adding a payee (1 - very difficult, 5 - very easy)
 +
# Ease of removing a payee (1 - very difficult, 5 - very easy)
 +
# Ease of navigating the application (1 - very difficult, 5 - very easy)
 +
 
 +
''For RMB only''
 +
# Ease of using QR code as login 2FA (1 - very difficult, 5 - very easy)
 +
# Ease of using QuikPay (1 - very difficult, 5 - very easy)
 +
 
 +
'''Observation Sheet'''
 +
# Time taken for login 2FA
 +
# Time taken for Fund Transfer
 +
# Number of times the user asked for help for login 2FA
 +
# Number of times the user asked for help for Fund Transfer
 +
# Number of times the user asked for help for the rest of the functions
 +
 
 +
''For RMB only''
 +
# Time taken for QuikPay 
 +
# Number of times the user asked for help for QuikPay
 +
 
 
|-
 
|-
!scope="row"  style=" text-align: center;"| 7
+
|}
|Putting manual fund transfer and NFC fund transfer makes the page very messy || 2 || Prototype to be updated with change
+
 
|-|-
+
=====Participants, Timing and Venue held=====
!scope="row" style=" text-align: center;"| 8
+
Number of participants: 20 <br/>
|Did not know the branch tab at the loan calculator page is expanadable || 2 || Prototype to be updated with change
+
Date and Time held: 19 - 20 September 2013 (10am - 7pm)<br/>
 +
Venue: SMU
 +
 
 +
{|align="center"
 +
|[[Image:Brut11.JPG|thumb|left|200px|200px]]
 +
|[[Image:Brut12.JPG|thumb|left|200px|200px]]
 +
|[[Image:Brut13.JPG|thumb|left|200px|200px]]
 +
|[[Image:Brut14.JPG|thumb|left|200px|200px]]
 +
|}
 +
<br/>
 +
 
 +
=====Implementation=====
 +
Participants will be welcomed to the user testing by the facilitator. Thereafter, participants will be given a brief overview of the application and given a list of tasks they need to complete. Participants will be required to complete all tasks, without the guidance of the facilitator. However, the participants can ask the facilitator for help should they face any problems and such instances will be recorded down on the Observation Sheet provided.
 +
 
 +
After going through the user testing of the application, participants are required to do an evaluation of the application on the Questionnaire provided. Participants are encouraged to discuss any problems they encounter with the facilitator.
 +
 
 +
The User Testing will end after the participants complete the Questionnaire. The facilitator will then thank the participants for their time and feedback provided.
 +
 
 +
=====Functions Tested=====
 +
{| class="wikitable" style="text-align: center; height:25px"
 +
|+
 
|-
 
|-
!scope="row"  style=" text-align: center;"| 9
+
! scope="colwidth="50" style="background:#CCCCCC| No.
|Bill payment (and other more important functions) should have double confirmation once it is completed  || 2 || Prototype to be updated with change
+
! scope="col" width="250" style="background:#CCCCCC| Functions Tested
 
|-
 
|-
!scope="row"  style=" text-align: center;"| 10
+
!scope="row"  style=" text-align: center;"| 1
|Long scrolls to get to the correct payee for fund transfer || 2 || Prototype to be updated with change
+
|Login 2FA
 
|-
 
|-
!scope="row"  style=" text-align: center;"| 11
+
!scope="row"  style=" text-align: center;"| 2
|Not sure of what the application actually does, or serve what function to the user || 2 || Problem will not be solved, redundant.
+
|View account information
 
|-
 
|-
!scope="row"  style=" text-align: center;"| 12
+
!scope="row"  style=" text-align: center;"| 3
|Want to scroll instead of clicking “more” for more transaction history || 2 || Prototype to be updated with change
+
|View transaction history
 
|-
 
|-
!scope="row"  style=" text-align: center;"| 13
+
!scope="row"  style=" text-align: center;"| 4
|Dropdown List to select account || 2 || Prototype to be updated with change
+
|Fund Transfer
 
|-
 
|-
!scope="row"  style=" text-align: center;"| 14
+
!scope="row"  style=" text-align: center;"| 5
|Notifications should not only appear in the app but reminds the user when the app is not launched || 2 || Not a problem, we already intend to implement for the app, just that it is not reflected in the paper prototype
+
|QuikPay
 
|-
 
|-
!scope="row"  style=" text-align: center;"| 15
+
!scope="row"  style=" text-align: center;"| 6
|Headers and buttons look the same || 2 || Prototype to be updated with change
+
|Add new payee
 
|-
 
|-
!scope="row"  style=" text-align: center;"| 16
+
!scope="row"  style=" text-align: center;"| 7
|The calendar and time selection for the loan calculator function may be hard to view and use on the phone || 2 || Prototype to be updated with change
+
|Remove existing payee
 
|-
 
|-
!scope="row" style=" text-align: center;"| 17
+
|}
|Unsure to key account number in fund transfer with or without dashes || 2 || Prototype to be updated with change
+
 
 +
=====Results and Follow Up=====
 +
Data will be collected using the Questionnaire provided by the team which participants are required to fill it up after going through the application. After the collection of all the data required, the team will collate and organize the raw data into appropriate tables and give a summary of the results of this user testing. A team meeting will be held to discuss the changes to be made.
 +
 
 +
=====Key Findings=====
 +
''Mobile Banking''
 +
# Most users have given positive feedback about using QR code such as increased convenience and ease of use. However, several users also point out the lack of security of QR code as it does not generate a dynamic password unlike the hardware token.
 +
# Almost all users find using QR code for login 2FA easy
 +
# No issue in usability for Account Summary and Transaction History
 +
# Almost all users have no problems using QuikPay, Standard Funds Transfer and Adding a new payee
 +
# Some users mentioned the lack of intuitiveness and instructions required for the delete payee function
 +
# Navigation of the whole application is easy among the users
 +
# Users had feedback that the application is easy to use and navigate and commented that they like the bright colour scheme of the application
 +
# For points that users dislike, many users reflect that they had problems deleting the payee as the function is not obvious. Security issue is another point brought up by users for the usage of QR code login 2FA
 +
# The general impression of users is our application is relatively easy to use and is able to navigate without any major issues. Some commented that they like the trendy feel the application gives
 +
 
 +
''Internet Banking''
 +
# A variety of feedbacks from users on Account Summary were collected. The difficulty in reading grey fonts stood out the most among the other feedbacks collected
 +
# A variety of feedbacks from users on Transaction History were collected. The inappropriate font size stood out the most among the other feedbacks collected. Users commented that font size used is too small to read
 +
# No major issue in usability for Standard Funds Transfer, Adding a payee and Deleting a payee
 +
# Navigation through the whole application is deemed to be moderately difficult among users. Users commented that the segments of the web application are unclear
 +
# For points that users like, many users commented that it is easy to navigate the web application. The basic neat design is well-liked by the users
 +
# For points that users dislike, some users reflect that they had problems reading the font due to its size and it being grey in colour. Few users reflect that there is too much white space for some pages
 +
# The general impression of the application is that it is relatively easy to use
 +
 
 +
====<div style=" border-bottom: 1px dotted; background: #FFFFFF; padding: 12px; font-weight: bold; line-height: 0.5em"><font color= black size = 4>Deployment exercise</font></div>====
 +
 
 +
=====Objective=====
 +
The objectives of the deployment exercise is to carry out stress testing of the our application (web and android) and to identify any bugs present in our application that we have not surfaced out during our own testing.
 +
 
 +
=====Participants, Timing and Venue held=====
 +
Number of participants: '''45''' students from the Retail Banking class <br/>
 +
Date and Time held: 2 October 2013 (8.15am - 9.30am)<br/>
 +
Venue: SMU
 +
<br/>
 +
 
 +
=====Implementation=====
 +
Participants are required to go through a Retail Banking Lab that involves the mobile and internet banking functions. 2 of the members will be facilitators of the exercise to help the participants should they face any problems with the application.
 +
 
 +
After going through the lab exercise, participants are encouraged to give any feedback to the facilitators.
 +
 
 +
=====Functions Tested=====
 +
''Internet Banking''
 +
{| class="wikitable" style="text-align: center; height:25px"
 +
|+
 
|-
 
|-
!scope="row"  style=" text-align: center;"| 18
+
! scope="colwidth="50" style="background:#CCCCCC| No.
|Unable to tell what RM stands for || 1 || Prototype to be updated with change
+
! scope="col" width="250" style="background:#CCCCCC| Functions Tested
 
|-
 
|-
!scope="row"  style=" text-align: center;"| 19
+
!scope="row"  style=" text-align: center;"| 1
|“Add new GIRO Arrangement” button is very small || 1 || Prototype to be updated with change. And button should not be at the bottom
+
|Login 2FA
 
|-
 
|-
!scope="row"  style=" text-align: center;"| 20
+
!scope="row"  style=" text-align: center;"| 2
|Two dots for the stocks on the home page looks weird || 1 || Prototype to be updated with change
+
|View account information
 
|-
 
|-
!scope="row"  style=" text-align: center;"| 21
+
!scope="row"  style=" text-align: center;"| 3
|Effective search function for adding new stock tickers to watchlist || 1 || No action required. Search is already intended to be intuitive, just that it is not clearly reflected by the paper prototype
+
|View transaction history
 
|-
 
|-
!scope="row"  style=" text-align: center;"| 22
+
!scope="row"  style=" text-align: center;"| 4
|Other services button unnecessary || 1 || No action required. Team consensus to keep as “Other Services” first
+
|Fund Transfer
 
|-
 
|-
!scope="row"  style=" text-align: center;"| 23
+
!scope="row"  style=" text-align: center;"| 5
|Unable to view transaction history at a glance || 1 || Problem will not be rectified, not within project scope
+
|Set up Customer Preferences
 
|-
 
|-
!scope="row" style=" text-align: center;"| 24
+
|}
|Auto fill / Populated name || 1 || Prototype to be updated with change
+
 
 +
''Mobile Banking''
 +
{| class="wikitable" style="text-align: center; height:25px"
 +
|+
 
|-
 
|-
!scope="row"  style=" text-align: center;"| 25
+
! scope="colwidth="50" style="background:#CCCCCC| No.
|SMU tBank logo which functions as a home button || 1 || Prototype to be updated with change
+
! scope="col" width="250" style="background:#CCCCCC| Functions Tested
 
|-
 
|-
!scope="row"  style=" text-align: center;"| 26
+
!scope="row"  style=" text-align: center;"| 1
|Spelling error of QuikPay || 1 || No action required. Problem will not be rectified, team consensus to use back ‘QuikPay’
+
|Login 2FA
 
|-
 
|-
!scope="row"  style=" text-align: center;"| 27
+
!scope="row"  style=" text-align: center;"| 2
|Stock ticker only appears on the home page || 1 || No action required. Problem will not be rectified, team consensus to stick with the original display for stock tickers
+
|View account information
 
|-
 
|-
!scope="row"  style=" text-align: center;"| 28
+
!scope="row"  style=" text-align: center;"| 3
|Filter the largest expense to the smallest expense || 0 || No action required. Filter function is already in place for transaction history.
+
|View transaction history
 
|-
 
|-
!scope="row"  style=" text-align: center;"| 29
+
!scope="row"  style=" text-align: center;"| 4
|Have notifications when users hit the minimum level that users have set previously to prevent overspending || 0 || No action required. Problem will not be rectified now, will be implemented if time allows in the future.
+
|Fund Transfer
 
|-
 
|-
 +
|}
  
|}
+
=====Follow Up=====
 +
The facilitators will collect the feedback and communicate to the members of the group to make any necessary changes to the application (Android and web).

Latest revision as of 11:28, 3 December 2013

Group Logo.png
HomeThe TeamProject OverviewProject ManagementProject DocumentationTeam ResourcesTeam ReflectionMenu bar br.jpg

Design & Prototyping (Mobile)

Design & Prototyping (Internet)

Heuristic Evaluation

Diagrams

Persona & Scenarios

Testing

Administrative


User Testing 2 (UT 2)

Objective

The objectives of UT 2 are to compare the proficiency of experienced and inexperienced users to find out if learning curve of our app (RIB and RMB) is large, gather feedback on the ease of using the application and collect overall feedback of the feel and design of the application.

Devices tested for Android
  1. Samsung S3
  2. Samsung S4
  3. HTC Desire X
Devices tested for iOS
  1. iPhone 4
  2. iPhone 4S
  3. iPhone 5
  4. iPhone 5S
Documents
Purpose

The purpose of the UT 2 is to find out the following from the users:

  1. Is the learning curve of our app (internet and mobile) large?
  2. Is the QR code QuikPay function intuitive enough?
  3. What do users feel about the QR code login for internet banking?
  4. Can the user use all the functions of the application with ease?
  5. How easy it is for the user to navigate through the whole application?
  6. How does the user feel about the overall design of the application?


Participants, Timing and Venue held

Number of participants: 20 (10 for Internet Banking, 5 for Android, 5 for iOS)
Date and Time held: 15 November 2013 (11am - 7pm)
Venue: SMU

Brut21.jpg
Brut22.jpg
Brut23.jpg
Brut24.jpg


Brut25.jpg
Brut26.jpg
Brut27.jpg
Brut28.jpg
Implementation

Participants will be welcomed to the user testing by the facilitator. Thereafter, participants will be given a brief overview of the application. Participants will then be given the specific tasks that they are required to complete without the guidance of the facilitator. In face of any problems that they are unable to resolve themselves, participants can ask the facilitator for help. The problem faced by the participants will then be recorded down the Observation Sheet provided for each facilitator.

After going through the user testing of the application, participants are required to complete an evaluation of the application on the Questionnaire provided. Participants are encouraged to discuss any problems they encounter with the facilitator.

The User Testing will end after the completion of the Questionnaire. The facilitator will then thank the participants for their time and feedback provided.

Functions Tested
No. Functions Tested
1 Login 2FA (RIB Mobile login 2FA)
2 View account information
3 View transaction history
4 Immediate Fund Transfer
5 Standing Instructions Fund Transfer
6 Bill Payment to billing organizations
7 GIRO arrangement with billing organizations
8 QuikPay QR Code (for mobile only)
9 View and edit profile settings (for web only)
10 Loan Calculator (Calculate Loan only)
Data Collected
Qualitative Data Quantitative Data

Questionnaire

  1. Feedback on the overall design of the application

For RMB only

  1. Feedback on QR Code QuikPay function

For RIB only

  1. Feedback on mobile QR Code login 2FA

Observation sheet

  1. Recorded general comments from users

Questionnaire

  1. Age
  2. Gender
  3. Frequent User (Yes or No)
  4. Ease of using Accounts Summary & Transaction History (1 - very difficult, 5 - very easy)
  5. Ease of using Immediate Funds Transfer (1 - very difficult, 5 - very easy)
  6. Ease of using Standing Instructions Funds Transfer (1 - very difficult, 5 - very easy)
  7. Ease of using Bill Payment (1 - very difficult, 5 - very easy)
  8. Ease of using GIRO Arrangement (1 - very difficult, 5 - very easy)
  9. Ease of using Loan Calculator – Calculate Loan (1 - very difficult, 5 - very easy)
  10. Ease of navigating through the application (1 - very difficult, 5 - very easy)
  11. Ratings on the overall design of the application (1 - very poor, 5 - Excellent)

For RMB only

  1. Ease of using QR code QuikPay (1 - very difficult, 5 - very easy)

For RIB only

  1. Ease of using profile settings (1 - very difficult, 5 - very easy)
  2. Ease of using mobile QR Code Login 2FA (1 - very difficult, 5 - very easy)

Observation Sheet

  1. Time taken for each task
Results and Follow Up

Data collected through the Questionnaire will be organized and aggregate to give a summary of the results to be published on wiki at a later date. Quantitative analysis will be done to compare the proficiency of experienced and inexperienced users to find out if learning curve of our app is large. A team meeting will be held to discuss the changes to be made.

Key Findings
Mobile Banking
  • Most users have given positive feedback that using the QR code to do fund transfer is convenient and fast. However, there are some users who find it a little troublesome while they not used to this mode of funds transfer.
Occurences Comments
5 Convenient
2 Quick/Fast
  • Average ratings of the ease of using each functions: 3.8 - 4.3 (Relatively Easy to Very Easy)
  • Average ratings for the overall design of the application: 4.0 (Very good)
Occurences Comments
7 Easy to use/understand/navigate
3 Simple and Neat
3 Good use of colours
2 Looks nice

Many users reflect that our application easy to use and navigate. Some mentioned our application is visually appealing, not cluttered and uses good colour themes.

Internet Banking
  • Most users have given positive feedback in using the QR code to log in, such as good speed and accuracy and secured, in addition to easy to use and its convenience. However, there are users who felt that it was quite confusing as they are unsure of which QR code they are required to scan at each step.
Occurences Comments
2 Easy to use
2 Convenient
  • Average ratings of the ease of using each functions: 3.5 - 4.4 (Relatively Easy to Very Easy)
  • Average ratings for the overall design of the application: 4.2 (Very good)
Occurences Comments
5 Easy to use/understand/navigate/clear
5 Clean/simple/Neat
4 Visually appealing

Many users reflect that our application easy to use and navigate, and at the same time, looks clean and neat. Some mentioned our application looks nice and pretty.

Quantitative Analysis

Methodology
To measure if the learning curve of first-time users is great, we compared the timings of inexperienced users versus experienced users in completing assigned tasks. If the timings are similar, then we can conclude that the learning curve is not steep. However, if the inexperienced users take a significantly longer time (i.e. 30 seconds or more), this implies that there is a substantial learning curve. Our team has decided to use a significance level of 0.05 (5%). A two-sample unequal variance T-test is used as both sets of users are different.

H0 (Null Hypothesis): There is no (or little) difference between time taken for experienced and inexperienced users to complete tasks assigned to them
H1: Experienced users take a much faster time to complete the tasks as compared to inexperienced users

Results

Br QA results1.PNG Br QA results2.PNG

User Testing 1 (UT 1)

Objective

The objectives of UT 1 are to understand the intuitiveness, ease of using the application, as well as to get feedback from users on the design and layout of the application on all two platforms (Android and web).

Devices tested for Android
  1. Samsung S3
  2. Samsung Note 2
  3. LG Optimus G
Documents
Purpose

The purpose of the UT 1 is to find out the following from the users:

  1. How does user find the login 2FA using QR code instead of the conventional hardware token?
  2. Is the QuikPay function intuitive enough?
  3. Can the user use the Fund Transfer function with ease?
  4. How easy it is for the user to navigate through the whole application?
  5. How does the user feel about the layout and design of our application?
  6. What are the user's emotions when using the application?
  7. What are the user's general impressions of the application?


Data Collected
Qualitative Data Quantitative Data

Questionnaire

  1. Feedback on the layout of Account Summary Page
  2. Feedback on the design of Account Summary Page
  3. Feedback on the improvements for Account Summary Page
  4. Feedback on the layout of Transaction History Page
  5. Feedback on the design of Transaction History Page
  6. Feedback on the improvements for Transaction History Page
  7. Feedback on the things they like
  8. Feedback on the things they dislike
  9. Feedback on general impressions of the application

Observation sheet

  1. Recorded comments on Login 2FA
  2. Recorded comments on Funds Transfer
  3. Recorded feelings of user throughout the test
  4. Recorded general observations of users

For RMB only

  1. Recorded comments Login 2FA
  2. Recorded comments on QuikPay

Questionnaire

  1. Age
  2. Gender
  3. Frequent User (Yes or No)
  4. Ease of using Funds Transfer (1 - very difficult, 5 - very easy)
  5. Ease of adding a payee (1 - very difficult, 5 - very easy)
  6. Ease of removing a payee (1 - very difficult, 5 - very easy)
  7. Ease of navigating the application (1 - very difficult, 5 - very easy)

For RMB only

  1. Ease of using QR code as login 2FA (1 - very difficult, 5 - very easy)
  2. Ease of using QuikPay (1 - very difficult, 5 - very easy)

Observation Sheet

  1. Time taken for login 2FA
  2. Time taken for Fund Transfer
  3. Number of times the user asked for help for login 2FA
  4. Number of times the user asked for help for Fund Transfer
  5. Number of times the user asked for help for the rest of the functions

For RMB only

  1. Time taken for QuikPay
  2. Number of times the user asked for help for QuikPay
Participants, Timing and Venue held

Number of participants: 20
Date and Time held: 19 - 20 September 2013 (10am - 7pm)
Venue: SMU

Brut11.JPG
Brut12.JPG
Brut13.JPG
Brut14.JPG


Implementation

Participants will be welcomed to the user testing by the facilitator. Thereafter, participants will be given a brief overview of the application and given a list of tasks they need to complete. Participants will be required to complete all tasks, without the guidance of the facilitator. However, the participants can ask the facilitator for help should they face any problems and such instances will be recorded down on the Observation Sheet provided.

After going through the user testing of the application, participants are required to do an evaluation of the application on the Questionnaire provided. Participants are encouraged to discuss any problems they encounter with the facilitator.

The User Testing will end after the participants complete the Questionnaire. The facilitator will then thank the participants for their time and feedback provided.

Functions Tested
No. Functions Tested
1 Login 2FA
2 View account information
3 View transaction history
4 Fund Transfer
5 QuikPay
6 Add new payee
7 Remove existing payee
Results and Follow Up

Data will be collected using the Questionnaire provided by the team which participants are required to fill it up after going through the application. After the collection of all the data required, the team will collate and organize the raw data into appropriate tables and give a summary of the results of this user testing. A team meeting will be held to discuss the changes to be made.

Key Findings

Mobile Banking

  1. Most users have given positive feedback about using QR code such as increased convenience and ease of use. However, several users also point out the lack of security of QR code as it does not generate a dynamic password unlike the hardware token.
  2. Almost all users find using QR code for login 2FA easy
  3. No issue in usability for Account Summary and Transaction History
  4. Almost all users have no problems using QuikPay, Standard Funds Transfer and Adding a new payee
  5. Some users mentioned the lack of intuitiveness and instructions required for the delete payee function
  6. Navigation of the whole application is easy among the users
  7. Users had feedback that the application is easy to use and navigate and commented that they like the bright colour scheme of the application
  8. For points that users dislike, many users reflect that they had problems deleting the payee as the function is not obvious. Security issue is another point brought up by users for the usage of QR code login 2FA
  9. The general impression of users is our application is relatively easy to use and is able to navigate without any major issues. Some commented that they like the trendy feel the application gives

Internet Banking

  1. A variety of feedbacks from users on Account Summary were collected. The difficulty in reading grey fonts stood out the most among the other feedbacks collected
  2. A variety of feedbacks from users on Transaction History were collected. The inappropriate font size stood out the most among the other feedbacks collected. Users commented that font size used is too small to read
  3. No major issue in usability for Standard Funds Transfer, Adding a payee and Deleting a payee
  4. Navigation through the whole application is deemed to be moderately difficult among users. Users commented that the segments of the web application are unclear
  5. For points that users like, many users commented that it is easy to navigate the web application. The basic neat design is well-liked by the users
  6. For points that users dislike, some users reflect that they had problems reading the font due to its size and it being grey in colour. Few users reflect that there is too much white space for some pages
  7. The general impression of the application is that it is relatively easy to use

Deployment exercise

Objective

The objectives of the deployment exercise is to carry out stress testing of the our application (web and android) and to identify any bugs present in our application that we have not surfaced out during our own testing.

Participants, Timing and Venue held

Number of participants: 45 students from the Retail Banking class
Date and Time held: 2 October 2013 (8.15am - 9.30am)
Venue: SMU

Implementation

Participants are required to go through a Retail Banking Lab that involves the mobile and internet banking functions. 2 of the members will be facilitators of the exercise to help the participants should they face any problems with the application.

After going through the lab exercise, participants are encouraged to give any feedback to the facilitators.

Functions Tested

Internet Banking

No. Functions Tested
1 Login 2FA
2 View account information
3 View transaction history
4 Fund Transfer
5 Set up Customer Preferences

Mobile Banking

No. Functions Tested
1 Login 2FA
2 View account information
3 View transaction history
4 Fund Transfer
Follow Up

The facilitators will collect the feedback and communicate to the members of the group to make any necessary changes to the application (Android and web).