HeaderSIS.jpg

Difference between revisions of "IS480 Team wiki: 2013T1 Bankrevels Testing"

From IS480
Jump to navigation Jump to search
 
(31 intermediate revisions by the same user not shown)
Line 38: Line 38:
 
|}
 
|}
  
 +
<br>
 +
</center>
 +
====<div style=" border-bottom: 1px dotted; background: #FFFFFF; padding: 12px; font-weight: bold; line-height: 0.5em"><font color= black size = 4>User Testing 2 (UT 2)</font></div>====
 +
 +
=====Objective=====
 +
The objectives of UT 2 are to compare the proficiency of experienced and inexperienced users to find out if learning curve of our app (RIB and RMB) is large, gather feedback on the ease of using the application and collect overall feedback of the feel and design of the application.
 +
 +
=====Devices tested for Android=====
 +
# Samsung S3
 +
# Samsung S4
 +
# HTC Desire X
 +
 +
=====Devices tested for iOS=====
 +
# iPhone 4
 +
# iPhone 4S
 +
# iPhone 5
 +
# iPhone 5S
 +
 +
=====Documents=====
 +
* [[Media:BR_UT 2_Plan.pdf|UT 2 Execution Plan]]
 +
* [[Media:BR_UT2_Observation_sheet_-_mobile.pdf|Observation Sheet (Mobile)]]
 +
* [[Media:BR_UT2_Observation_sheet_-_internet.pdf‎ |Observation Sheet (Internet)]]
 +
* [[Media:BR_Questionnaire_-_internet.pdf‎|Questionnaire (Internet)]]
 +
* [[Media:BR_Questionnaire_-_mobile.pdf|Questionnaire (Mobile)]]
 +
* [[Media:BR_Tasks_-_internet.pdf|Tasks (Internet)]]
 +
* [[Media:Tasks_-_mobile.pdf‎|Tasks (Mobile)]]
 +
* [[Media:RMB_UT2_collated_results.pdf|Raw Results (Mobile)]]
 +
* [[Media:BR_UT2_RIB collated results.pdf|Raw Results (Internet)]]
 +
* [[Media:BR_UT2_experienced_users.pdf|Experienced Users Raw Results (Mobile and Internet)]]
 +
* [[Media:BR_Quantitative_analysis_RMB.xlsx|Quantitative Analysis (Mobile)]]
 +
* [[Media:BR_Quantitative_analysis_RIB.xlsx|Quantitative Analysis (Internet)]]
 +
* [[Media:BR_UT2_results.pdf|Summary of UT 2 results]]
 +
 +
=====Purpose=====
 +
The purpose of the UT 2 is to find out the following from the users:
 +
 +
# Is the learning curve of our app (internet and mobile) large?
 +
# Is the QR code QuikPay function intuitive enough?
 +
# What do users feel about the QR code login for internet banking?
 +
# Can the user use all the functions of the application with ease?
 +
# How easy it is for the user to navigate through the whole application?
 +
# How does the user feel about the overall design of the application?
 
<br>
 
<br>
  
</center>
+
=====Participants, Timing and Venue held=====
 +
Number of participants: 20 (10 for Internet Banking, 5 for Android, 5 for iOS) <br/>
 +
Date and Time held: 15 November 2013 (11am - 7pm)<br/>
 +
Venue: SMU
 +
 
 +
{|align="center"
 +
|[[Image:Brut21.jpg|thumb|left|200px|200px]]
 +
|[[Image:Brut22.jpg|thumb|left|200px|200px]]
 +
|[[Image:Brut23.jpg|thumb|left|200px|200px]]
 +
|[[Image:Brut24.jpg|thumb|left|200px|200px]]
 +
|}
 +
<br/>
 +
{|align="center"
 +
|[[Image:Brut25.jpg|thumb|left|200px|200px]]
 +
|[[Image:Brut26.jpg|thumb|left|200px|200px]]
 +
|[[Image:Brut27.jpg|thumb|left|200px|200px]]
 +
|[[Image:Brut28.jpg|thumb|left|200px|200px]]
 +
|}
 +
 
 +
=====Implementation=====
 +
Participants will be welcomed to the user testing by the facilitator. Thereafter, participants will be given a brief overview of the application. Participants will then be given the specific tasks that they are required to complete without the guidance of the facilitator. In face of any problems that they are unable to resolve themselves, participants can ask the facilitator for help. The problem faced by the participants will then be recorded down the Observation Sheet provided for each facilitator.
 +
<br><br>
 +
After going through the user testing of the application, participants are required to complete an evaluation of the application on the Questionnaire provided. Participants are encouraged to discuss any problems they encounter with the facilitator.
 +
<br><br>
 +
The User Testing will end after the completion of the Questionnaire. The facilitator will then thank the participants for their time and feedback provided.
 +
 
 +
=====Functions Tested=====
 +
{| class="wikitable" style="text-align: center; height:25px"
 +
|+
 +
|-
 +
! scope="col"  width="50" style="background:#CCCCCC| No.
 +
! scope="col" width="250" style="background:#CCCCCC| Functions Tested
 +
|-
 +
!scope="row"  style=" text-align: center;"| 1
 +
|Login 2FA (RIB Mobile login 2FA)
 +
|-
 +
!scope="row"  style=" text-align: center;"| 2
 +
|View account information
 +
|-
 +
!scope="row"  style=" text-align: center;"| 3
 +
|View transaction history
 +
|-
 +
!scope="row"  style=" text-align: center;"| 4
 +
|Immediate Fund Transfer
 +
|-
 +
!scope="row"  style=" text-align: center;"| 5
 +
|Standing Instructions Fund Transfer
 +
|-
 +
!scope="row"  style=" text-align: center;"| 6
 +
|Bill Payment to billing organizations
 +
|-
 +
!scope="row"  style=" text-align: center;"| 7
 +
|GIRO arrangement with billing organizations
 +
|-
 +
!scope="row"  style=" text-align: center;"| 8
 +
|QuikPay QR Code (for mobile only)
 +
|-
 +
!scope="row"  style=" text-align: center;"| 9
 +
|View and edit profile settings (for web only)
 +
|-
 +
!scope="row"  style=" text-align: center;"| 10
 +
|Loan Calculator (Calculate Loan only)
 +
|-
 +
|}
 +
 
 +
=====Data Collected=====
 +
{| class="wikitable centered" width="90%"
 +
!Qualitative Data
 +
!Quantitative Data
 +
 
 +
|-
 +
|
 +
'''Questionnaire'''
 +
# Feedback on the overall design of the application
 +
 
 +
''For RMB only''
 +
# Feedback on QR Code QuikPay function
 +
 
 +
''For RIB only''
 +
# Feedback on mobile QR Code login 2FA
 +
 
 +
'''Observation sheet'''
 +
# Recorded general comments from users
 +
 
 +
|
 +
'''Questionnaire'''
 +
# Age
 +
# Gender
 +
# Frequent User (Yes or No)
 +
# Ease of using Accounts Summary & Transaction History (1 - very difficult, 5 - very easy)
 +
# Ease of using Immediate Funds Transfer (1 - very difficult, 5 - very easy)
 +
# Ease of using Standing Instructions Funds Transfer (1 - very difficult, 5 - very easy)
 +
# Ease of using Bill Payment (1 - very difficult, 5 - very easy)
 +
# Ease of using GIRO Arrangement (1 - very difficult, 5 - very easy)
 +
# Ease of using Loan Calculator – Calculate Loan (1 - very difficult, 5 - very easy)
 +
# Ease of navigating through the application (1 - very difficult, 5 - very easy)
 +
# Ratings on the overall design of the application (1 - very poor, 5 - Excellent)
 +
 
 +
''For RMB only''
 +
# Ease of using QR code QuikPay (1 - very difficult, 5 - very easy)
 +
 
 +
''For RIB only''
 +
# Ease of using profile settings (1 - very difficult, 5 - very easy)
 +
# Ease of using mobile QR Code Login 2FA (1 - very difficult, 5 - very easy)
 +
 
 +
'''Observation Sheet'''
 +
# Time taken for each task
 +
 
 +
|-
 +
|}
 +
 
 +
=====Results and Follow Up=====
 +
Data collected through the Questionnaire will be organized and aggregate to give a summary of the results to be published on wiki at a later date. Quantitative analysis will be done  to compare the proficiency of experienced and inexperienced users to find out if learning curve of our app is large. A team meeting will be held to discuss the changes to be made.
 +
 
 +
=====Key Findings=====
 +
======Mobile Banking======
 +
* Most users have given positive feedback that using the QR code to do fund transfer is convenient and fast.  However, there are some users who find it a little troublesome while they not used to this mode of funds transfer.
 +
{| class="wikitable" style="text-align: center; height:25px"
 +
|+
 +
|-
 +
! scope="col"  width="50" style="background:#CCCCCC| Occurences
 +
! scope="col" width="250" style="background:#CCCCCC| Comments
 +
|-
 +
!scope="row"  style=" text-align: center;"| 5
 +
|Convenient
 +
|-
 +
!scope="row"  style=" text-align: center;"| 2
 +
|Quick/Fast
 +
|-
 +
|}
 +
* Average ratings of the ease of using each functions: 3.8 - 4.3 (Relatively Easy to Very Easy)
 +
* Average ratings for the overall design of the application: 4.0 (Very good)
 +
{| class="wikitable" style="text-align: center; height:25px"
 +
|+
 +
|-
 +
! scope="col"  width="50" style="background:#CCCCCC| Occurences
 +
! scope="col" width="250" style="background:#CCCCCC| Comments
 +
|-
 +
!scope="row"  style=" text-align: center;"| 7
 +
|Easy to use/understand/navigate
 +
|-
 +
!scope="row"  style=" text-align: center;"| 3
 +
|Simple and Neat
 +
|-
 +
!scope="row"  style=" text-align: center;"| 3
 +
|Good use of colours
 +
|-
 +
!scope="row"  style=" text-align: center;"| 2
 +
|Looks nice
 +
|-
 +
|}
 +
Many users reflect that our application easy to use and navigate. Some mentioned our application is visually appealing, not cluttered and uses good colour themes.
 +
 
 +
======Internet Banking======
 +
* Most users have given positive feedback in using the QR code to log in, such as good speed and accuracy and secured, in addition to easy to use and its convenience. However, there are users who felt that it was quite confusing as they are unsure of which QR code they are required to scan at each step.
 +
{| class="wikitable" style="text-align: center; height:25px"
 +
|+
 +
|-
 +
! scope="col"  width="50" style="background:#CCCCCC| Occurences
 +
! scope="col" width="250" style="background:#CCCCCC| Comments
 +
|-
 +
!scope="row"  style=" text-align: center;"| 2
 +
|Easy to use
 +
|-
 +
!scope="row"  style=" text-align: center;"| 2
 +
|Convenient
 +
|-
 +
|}
 +
* Average ratings of the ease of using each functions: 3.5 - 4.4 (Relatively Easy to Very Easy)
 +
* Average ratings for the overall design of the application: 4.2 (Very good)
 +
{| class="wikitable" style="text-align: center; height:25px"
 +
|+
 +
|-
 +
! scope="col"  width="50" style="background:#CCCCCC| Occurences
 +
! scope="col" width="250" style="background:#CCCCCC| Comments
 +
|-
 +
!scope="row"  style=" text-align: center;"| 5
 +
|Easy to use/understand/navigate/clear
 +
|-
 +
!scope="row"  style=" text-align: center;"| 5
 +
|Clean/simple/Neat
 +
|-
 +
!scope="row"  style=" text-align: center;"| 4
 +
|Visually appealing
 +
|-
 +
|}
 +
Many users reflect that our application easy to use and navigate, and at the same time, looks clean and neat. Some mentioned our application looks nice and pretty.
 +
 
 +
======Quantitative Analysis======
 +
''Methodology''
 +
<br>To measure if the learning curve of first-time users is great, we compared the timings of inexperienced users versus experienced users in completing assigned tasks. If the timings are similar, then we can conclude that the learning curve is not steep. However, if the inexperienced users take a significantly longer time (i.e. 30 seconds or more), this implies that there is a substantial learning curve.
 +
Our team has decided to use a significance level of 0.05 (5%). A two-sample unequal variance T-test is used as both sets of users are different.
 +
<br><br>
 +
'''H0 (Null Hypothesis):''' There is no (or little) difference between time taken for experienced and inexperienced users to complete tasks assigned to them
 +
<br>
 +
'''H1:''' Experienced users take a much faster time to complete the tasks as compared to inexperienced users
 +
<br><br>
 +
''Results''
 +
<br><br>
 +
[[Image:Br_QA_results1.PNG|1000px]]
 +
[[Image:Br_QA_results2.PNG|1000px]]
 +
 
 +
====<div style=" border-bottom: 1px dotted; background: #FFFFFF; padding: 12px; font-weight: bold; line-height: 0.5em"><font color= black size = 4>User Testing 1 (UT 1)</font></div>====
 +
 
 +
=====Objective=====
 +
The objectives of UT 1 are to understand the intuitiveness, ease of using the application, as well as to get feedback from users on the design and layout of the application on all two platforms (Android and web).
 +
 
 +
=====Devices tested for Android=====
 +
# Samsung S3
 +
# Samsung Note 2
 +
# LG Optimus G
 +
 
 +
=====Documents=====
 +
* [[Media:BR_UT_1_Plan.pdf|UT 1 Execution Plan]]
 +
* [[Media:BR_Observation_sheet_-_android.pdf|Observation Sheet (Android)]]
 +
* [[Media:BR_Observation_sheet_-_internet.pdf|Observation Sheet (Internet)]]
 +
* [[Media:Br_Questionnaire_-_internet.pdf|Questionnaire (Internet)]]
 +
* [[Media:Br_Questionnaire_-_mobile.pdf|Questionnaire (Android)]]
 +
* [[Media:Br_Tasks_-_internet.pdf|Tasks (Internet)]]
 +
* [[Media:Br_Tasks_-_mobile.pdf|Tasks (Android)]]
 +
* [[Media:BR_RMB_collated_results.pdf|Raw Results (Android)]]
 +
* [[Media:BR_RIB_collated_results.pdf|Raw Results (Internet)]]
 +
* [[Media:BR_UT1_results.pdf|Summary of UT 1 results]]
 +
 
 +
=====Purpose=====
 +
The purpose of the UT 1 is to find out the following from the users:
 +
 
 +
# How does user find the login 2FA using QR code instead of the conventional hardware token?
 +
# Is the QuikPay function intuitive enough?
 +
# Can the user use the Fund Transfer function with ease?
 +
# How easy it is for the user to navigate through the whole application?
 +
# How does the user feel about the layout and design of our application?
 +
# What are the user's emotions when using the application?
 +
# What are the user's general impressions of the application?
 +
<br>
 +
 
 +
=====Data Collected=====
 +
{| class="wikitable centered" width="90%"
 +
!Qualitative Data
 +
!Quantitative Data
 +
 
 +
|-
 +
|
 +
'''Questionnaire'''
 +
# Feedback on the layout of Account Summary Page
 +
# Feedback on the design of Account Summary Page
 +
# Feedback on the improvements for Account Summary Page
 +
# Feedback on the layout of Transaction History Page
 +
# Feedback on the design of Transaction History Page
 +
# Feedback on the improvements for Transaction History Page
 +
# Feedback on the things they like
 +
# Feedback on the things they dislike
 +
# Feedback on general impressions of the application
 +
 
 +
'''Observation sheet'''
 +
# Recorded comments on Login 2FA
 +
# Recorded comments on Funds Transfer
 +
# Recorded feelings of user throughout the test
 +
# Recorded general observations of users
 +
 
 +
''For RMB only''
 +
# Recorded comments Login 2FA
 +
# Recorded comments on QuikPay
 +
 
 +
|
 +
'''Questionnaire'''
 +
# Age
 +
# Gender
 +
# Frequent User (Yes or No)
 +
# Ease of using Funds Transfer (1 - very difficult, 5 - very easy)
 +
# Ease of adding a payee (1 - very difficult, 5 - very easy)
 +
# Ease of removing a payee (1 - very difficult, 5 - very easy)
 +
# Ease of navigating the application (1 - very difficult, 5 - very easy)
 +
 
 +
''For RMB only''
 +
# Ease of using QR code as login 2FA (1 - very difficult, 5 - very easy)
 +
# Ease of using QuikPay (1 - very difficult, 5 - very easy)
 +
 
 +
'''Observation Sheet'''
 +
# Time taken for login 2FA
 +
# Time taken for Fund Transfer
 +
# Number of times the user asked for help for login 2FA
 +
# Number of times the user asked for help for Fund Transfer
 +
# Number of times the user asked for help for the rest of the functions
 +
 
 +
''For RMB only''
 +
# Time taken for QuikPay 
 +
# Number of times the user asked for help for QuikPay
 +
 
 +
|-
 +
|}
 +
 
 +
=====Participants, Timing and Venue held=====
 +
Number of participants: 20 <br/>
 +
Date and Time held: 19 - 20 September 2013 (10am - 7pm)<br/>
 +
Venue: SMU
 +
 
 +
{|align="center"
 +
|[[Image:Brut11.JPG|thumb|left|200px|200px]]
 +
|[[Image:Brut12.JPG|thumb|left|200px|200px]]
 +
|[[Image:Brut13.JPG|thumb|left|200px|200px]]
 +
|[[Image:Brut14.JPG|thumb|left|200px|200px]]
 +
|}
 +
<br/>
 +
 
 +
=====Implementation=====
 +
Participants will be welcomed to the user testing by the facilitator. Thereafter, participants will be given a brief overview of the application and given a list of tasks they need to complete. Participants will be required to complete all tasks, without the guidance of the facilitator. However, the participants can ask the facilitator for help should they face any problems and such instances will be recorded down on the Observation Sheet provided.
 +
 
 +
After going through the user testing of the application, participants are required to do an evaluation of the application on the Questionnaire provided. Participants are encouraged to discuss any problems they encounter with the facilitator.
 +
 
 +
The User Testing will end after the participants complete the Questionnaire. The facilitator will then thank the participants for their time and feedback provided.
 +
 
 +
=====Functions Tested=====
 +
{| class="wikitable" style="text-align: center; height:25px"
 +
|+
 +
|-
 +
! scope="col"  width="50" style="background:#CCCCCC| No.
 +
! scope="col" width="250" style="background:#CCCCCC| Functions Tested
 +
|-
 +
!scope="row"  style=" text-align: center;"| 1
 +
|Login 2FA
 +
|-
 +
!scope="row"  style=" text-align: center;"| 2
 +
|View account information
 +
|-
 +
!scope="row"  style=" text-align: center;"| 3
 +
|View transaction history
 +
|-
 +
!scope="row"  style=" text-align: center;"| 4
 +
|Fund Transfer
 +
|-
 +
!scope="row"  style=" text-align: center;"| 5
 +
|QuikPay
 +
|-
 +
!scope="row"  style=" text-align: center;"| 6
 +
|Add new payee
 +
|-
 +
!scope="row"  style=" text-align: center;"| 7
 +
|Remove existing payee
 +
|-
 +
|}
 +
 
 +
=====Results and Follow Up=====
 +
Data will be collected using the Questionnaire provided by the team which participants are required to fill it up after going through the application. After the collection of all the data required, the team will collate and organize the raw data into appropriate tables and give a summary of the results of this user testing. A team meeting will be held to discuss the changes to be made.
 +
 
 +
=====Key Findings=====
 +
''Mobile Banking''
 +
# Most users have given positive feedback about using QR code such as increased convenience and ease of use. However, several users also point out the lack of security of QR code as it does not generate a dynamic password unlike the hardware token.
 +
# Almost all users find using QR code for login 2FA easy
 +
# No issue in usability for Account Summary and Transaction History
 +
# Almost all users have no problems using QuikPay, Standard Funds Transfer and Adding a new payee
 +
# Some users mentioned the lack of intuitiveness and instructions required for the delete payee function
 +
# Navigation of the whole application is easy among the users
 +
# Users had feedback that the application is easy to use and navigate and commented that they like the bright colour scheme of the application
 +
# For points that users dislike, many users reflect that they had problems deleting the payee as the function is not obvious. Security issue is another point brought up by users for the usage of QR code login 2FA
 +
# The general impression of users is our application is relatively easy to use and is able to navigate without any major issues. Some commented that they like the trendy feel the application gives
 +
 
 +
''Internet Banking''
 +
# A variety of feedbacks from users on Account Summary were collected. The difficulty in reading grey fonts stood out the most among the other feedbacks collected
 +
# A variety of feedbacks from users on Transaction History were collected. The inappropriate font size stood out the most among the other feedbacks collected. Users commented that font size used is too small to read
 +
# No major issue in usability for Standard Funds Transfer, Adding a payee and Deleting a payee
 +
# Navigation through the whole application is deemed to be moderately difficult among users. Users commented that the segments of the web application are unclear
 +
# For points that users like, many users commented that it is easy to navigate the web application. The basic neat design is well-liked by the users
 +
# For points that users dislike, some users reflect that they had problems reading the font due to its size and it being grey in colour. Few users reflect that there is too much white space for some pages
 +
# The general impression of the application is that it is relatively easy to use
 +
 
 +
====<div style=" border-bottom: 1px dotted; background: #FFFFFF; padding: 12px; font-weight: bold; line-height: 0.5em"><font color= black size = 4>Deployment exercise</font></div>====
 +
 
 +
=====Objective=====
 +
The objectives of the deployment exercise is to carry out stress testing of the our application (web and android) and to identify any bugs present in our application that we have not surfaced out during our own testing.
 +
 
 +
=====Participants, Timing and Venue held=====
 +
Number of participants: '''45''' students from the Retail Banking class <br/>
 +
Date and Time held: 2 October 2013 (8.15am - 9.30am)<br/>
 +
Venue: SMU
 +
<br/>
 +
 
 +
=====Implementation=====
 +
Participants are required to go through a Retail Banking Lab that involves the mobile and internet banking functions. 2 of the members will be facilitators of the exercise to help the participants should they face any problems with the application.
 +
 
 +
After going through the lab exercise, participants are encouraged to give any feedback to the facilitators.
 +
 
 +
=====Functions Tested=====
 +
''Internet Banking''
 +
{| class="wikitable" style="text-align: center; height:25px"
 +
|+
 +
|-
 +
! scope="col"  width="50" style="background:#CCCCCC| No.
 +
! scope="col" width="250" style="background:#CCCCCC| Functions Tested
 +
|-
 +
!scope="row"  style=" text-align: center;"| 1
 +
|Login 2FA
 +
|-
 +
!scope="row"  style=" text-align: center;"| 2
 +
|View account information
 +
|-
 +
!scope="row"  style=" text-align: center;"| 3
 +
|View transaction history
 +
|-
 +
!scope="row"  style=" text-align: center;"| 4
 +
|Fund Transfer
 +
|-
 +
!scope="row"  style=" text-align: center;"| 5
 +
|Set up Customer Preferences
 +
|-
 +
|}
 +
 
 +
''Mobile Banking''
 +
{| class="wikitable" style="text-align: center; height:25px"
 +
|+
 +
|-
 +
! scope="col"  width="50" style="background:#CCCCCC| No.
 +
! scope="col" width="250" style="background:#CCCCCC| Functions Tested
 +
|-
 +
!scope="row"  style=" text-align: center;"| 1
 +
|Login 2FA
 +
|-
 +
!scope="row"  style=" text-align: center;"| 2
 +
|View account information
 +
|-
 +
!scope="row"  style=" text-align: center;"| 3
 +
|View transaction history
 +
|-
 +
!scope="row"  style=" text-align: center;"| 4
 +
|Fund Transfer
 +
|-
 +
|}
 +
 
 +
=====Follow Up=====
 +
The facilitators will collect the feedback and communicate to the members of the group to make any necessary changes to the application (Android and web).

Latest revision as of 11:28, 3 December 2013

Group Logo.png
HomeThe TeamProject OverviewProject ManagementProject DocumentationTeam ResourcesTeam ReflectionMenu bar br.jpg

Design & Prototyping (Mobile)

Design & Prototyping (Internet)

Heuristic Evaluation

Diagrams

Persona & Scenarios

Testing

Administrative


User Testing 2 (UT 2)

Objective

The objectives of UT 2 are to compare the proficiency of experienced and inexperienced users to find out if learning curve of our app (RIB and RMB) is large, gather feedback on the ease of using the application and collect overall feedback of the feel and design of the application.

Devices tested for Android
  1. Samsung S3
  2. Samsung S4
  3. HTC Desire X
Devices tested for iOS
  1. iPhone 4
  2. iPhone 4S
  3. iPhone 5
  4. iPhone 5S
Documents
Purpose

The purpose of the UT 2 is to find out the following from the users:

  1. Is the learning curve of our app (internet and mobile) large?
  2. Is the QR code QuikPay function intuitive enough?
  3. What do users feel about the QR code login for internet banking?
  4. Can the user use all the functions of the application with ease?
  5. How easy it is for the user to navigate through the whole application?
  6. How does the user feel about the overall design of the application?


Participants, Timing and Venue held

Number of participants: 20 (10 for Internet Banking, 5 for Android, 5 for iOS)
Date and Time held: 15 November 2013 (11am - 7pm)
Venue: SMU

Brut21.jpg
Brut22.jpg
Brut23.jpg
Brut24.jpg


Brut25.jpg
Brut26.jpg
Brut27.jpg
Brut28.jpg
Implementation

Participants will be welcomed to the user testing by the facilitator. Thereafter, participants will be given a brief overview of the application. Participants will then be given the specific tasks that they are required to complete without the guidance of the facilitator. In face of any problems that they are unable to resolve themselves, participants can ask the facilitator for help. The problem faced by the participants will then be recorded down the Observation Sheet provided for each facilitator.

After going through the user testing of the application, participants are required to complete an evaluation of the application on the Questionnaire provided. Participants are encouraged to discuss any problems they encounter with the facilitator.

The User Testing will end after the completion of the Questionnaire. The facilitator will then thank the participants for their time and feedback provided.

Functions Tested
No. Functions Tested
1 Login 2FA (RIB Mobile login 2FA)
2 View account information
3 View transaction history
4 Immediate Fund Transfer
5 Standing Instructions Fund Transfer
6 Bill Payment to billing organizations
7 GIRO arrangement with billing organizations
8 QuikPay QR Code (for mobile only)
9 View and edit profile settings (for web only)
10 Loan Calculator (Calculate Loan only)
Data Collected
Qualitative Data Quantitative Data

Questionnaire

  1. Feedback on the overall design of the application

For RMB only

  1. Feedback on QR Code QuikPay function

For RIB only

  1. Feedback on mobile QR Code login 2FA

Observation sheet

  1. Recorded general comments from users

Questionnaire

  1. Age
  2. Gender
  3. Frequent User (Yes or No)
  4. Ease of using Accounts Summary & Transaction History (1 - very difficult, 5 - very easy)
  5. Ease of using Immediate Funds Transfer (1 - very difficult, 5 - very easy)
  6. Ease of using Standing Instructions Funds Transfer (1 - very difficult, 5 - very easy)
  7. Ease of using Bill Payment (1 - very difficult, 5 - very easy)
  8. Ease of using GIRO Arrangement (1 - very difficult, 5 - very easy)
  9. Ease of using Loan Calculator – Calculate Loan (1 - very difficult, 5 - very easy)
  10. Ease of navigating through the application (1 - very difficult, 5 - very easy)
  11. Ratings on the overall design of the application (1 - very poor, 5 - Excellent)

For RMB only

  1. Ease of using QR code QuikPay (1 - very difficult, 5 - very easy)

For RIB only

  1. Ease of using profile settings (1 - very difficult, 5 - very easy)
  2. Ease of using mobile QR Code Login 2FA (1 - very difficult, 5 - very easy)

Observation Sheet

  1. Time taken for each task
Results and Follow Up

Data collected through the Questionnaire will be organized and aggregate to give a summary of the results to be published on wiki at a later date. Quantitative analysis will be done to compare the proficiency of experienced and inexperienced users to find out if learning curve of our app is large. A team meeting will be held to discuss the changes to be made.

Key Findings
Mobile Banking
  • Most users have given positive feedback that using the QR code to do fund transfer is convenient and fast. However, there are some users who find it a little troublesome while they not used to this mode of funds transfer.
Occurences Comments
5 Convenient
2 Quick/Fast
  • Average ratings of the ease of using each functions: 3.8 - 4.3 (Relatively Easy to Very Easy)
  • Average ratings for the overall design of the application: 4.0 (Very good)
Occurences Comments
7 Easy to use/understand/navigate
3 Simple and Neat
3 Good use of colours
2 Looks nice

Many users reflect that our application easy to use and navigate. Some mentioned our application is visually appealing, not cluttered and uses good colour themes.

Internet Banking
  • Most users have given positive feedback in using the QR code to log in, such as good speed and accuracy and secured, in addition to easy to use and its convenience. However, there are users who felt that it was quite confusing as they are unsure of which QR code they are required to scan at each step.
Occurences Comments
2 Easy to use
2 Convenient
  • Average ratings of the ease of using each functions: 3.5 - 4.4 (Relatively Easy to Very Easy)
  • Average ratings for the overall design of the application: 4.2 (Very good)
Occurences Comments
5 Easy to use/understand/navigate/clear
5 Clean/simple/Neat
4 Visually appealing

Many users reflect that our application easy to use and navigate, and at the same time, looks clean and neat. Some mentioned our application looks nice and pretty.

Quantitative Analysis

Methodology
To measure if the learning curve of first-time users is great, we compared the timings of inexperienced users versus experienced users in completing assigned tasks. If the timings are similar, then we can conclude that the learning curve is not steep. However, if the inexperienced users take a significantly longer time (i.e. 30 seconds or more), this implies that there is a substantial learning curve. Our team has decided to use a significance level of 0.05 (5%). A two-sample unequal variance T-test is used as both sets of users are different.

H0 (Null Hypothesis): There is no (or little) difference between time taken for experienced and inexperienced users to complete tasks assigned to them
H1: Experienced users take a much faster time to complete the tasks as compared to inexperienced users

Results

Br QA results1.PNG Br QA results2.PNG

User Testing 1 (UT 1)

Objective

The objectives of UT 1 are to understand the intuitiveness, ease of using the application, as well as to get feedback from users on the design and layout of the application on all two platforms (Android and web).

Devices tested for Android
  1. Samsung S3
  2. Samsung Note 2
  3. LG Optimus G
Documents
Purpose

The purpose of the UT 1 is to find out the following from the users:

  1. How does user find the login 2FA using QR code instead of the conventional hardware token?
  2. Is the QuikPay function intuitive enough?
  3. Can the user use the Fund Transfer function with ease?
  4. How easy it is for the user to navigate through the whole application?
  5. How does the user feel about the layout and design of our application?
  6. What are the user's emotions when using the application?
  7. What are the user's general impressions of the application?


Data Collected
Qualitative Data Quantitative Data

Questionnaire

  1. Feedback on the layout of Account Summary Page
  2. Feedback on the design of Account Summary Page
  3. Feedback on the improvements for Account Summary Page
  4. Feedback on the layout of Transaction History Page
  5. Feedback on the design of Transaction History Page
  6. Feedback on the improvements for Transaction History Page
  7. Feedback on the things they like
  8. Feedback on the things they dislike
  9. Feedback on general impressions of the application

Observation sheet

  1. Recorded comments on Login 2FA
  2. Recorded comments on Funds Transfer
  3. Recorded feelings of user throughout the test
  4. Recorded general observations of users

For RMB only

  1. Recorded comments Login 2FA
  2. Recorded comments on QuikPay

Questionnaire

  1. Age
  2. Gender
  3. Frequent User (Yes or No)
  4. Ease of using Funds Transfer (1 - very difficult, 5 - very easy)
  5. Ease of adding a payee (1 - very difficult, 5 - very easy)
  6. Ease of removing a payee (1 - very difficult, 5 - very easy)
  7. Ease of navigating the application (1 - very difficult, 5 - very easy)

For RMB only

  1. Ease of using QR code as login 2FA (1 - very difficult, 5 - very easy)
  2. Ease of using QuikPay (1 - very difficult, 5 - very easy)

Observation Sheet

  1. Time taken for login 2FA
  2. Time taken for Fund Transfer
  3. Number of times the user asked for help for login 2FA
  4. Number of times the user asked for help for Fund Transfer
  5. Number of times the user asked for help for the rest of the functions

For RMB only

  1. Time taken for QuikPay
  2. Number of times the user asked for help for QuikPay
Participants, Timing and Venue held

Number of participants: 20
Date and Time held: 19 - 20 September 2013 (10am - 7pm)
Venue: SMU

Brut11.JPG
Brut12.JPG
Brut13.JPG
Brut14.JPG


Implementation

Participants will be welcomed to the user testing by the facilitator. Thereafter, participants will be given a brief overview of the application and given a list of tasks they need to complete. Participants will be required to complete all tasks, without the guidance of the facilitator. However, the participants can ask the facilitator for help should they face any problems and such instances will be recorded down on the Observation Sheet provided.

After going through the user testing of the application, participants are required to do an evaluation of the application on the Questionnaire provided. Participants are encouraged to discuss any problems they encounter with the facilitator.

The User Testing will end after the participants complete the Questionnaire. The facilitator will then thank the participants for their time and feedback provided.

Functions Tested
No. Functions Tested
1 Login 2FA
2 View account information
3 View transaction history
4 Fund Transfer
5 QuikPay
6 Add new payee
7 Remove existing payee
Results and Follow Up

Data will be collected using the Questionnaire provided by the team which participants are required to fill it up after going through the application. After the collection of all the data required, the team will collate and organize the raw data into appropriate tables and give a summary of the results of this user testing. A team meeting will be held to discuss the changes to be made.

Key Findings

Mobile Banking

  1. Most users have given positive feedback about using QR code such as increased convenience and ease of use. However, several users also point out the lack of security of QR code as it does not generate a dynamic password unlike the hardware token.
  2. Almost all users find using QR code for login 2FA easy
  3. No issue in usability for Account Summary and Transaction History
  4. Almost all users have no problems using QuikPay, Standard Funds Transfer and Adding a new payee
  5. Some users mentioned the lack of intuitiveness and instructions required for the delete payee function
  6. Navigation of the whole application is easy among the users
  7. Users had feedback that the application is easy to use and navigate and commented that they like the bright colour scheme of the application
  8. For points that users dislike, many users reflect that they had problems deleting the payee as the function is not obvious. Security issue is another point brought up by users for the usage of QR code login 2FA
  9. The general impression of users is our application is relatively easy to use and is able to navigate without any major issues. Some commented that they like the trendy feel the application gives

Internet Banking

  1. A variety of feedbacks from users on Account Summary were collected. The difficulty in reading grey fonts stood out the most among the other feedbacks collected
  2. A variety of feedbacks from users on Transaction History were collected. The inappropriate font size stood out the most among the other feedbacks collected. Users commented that font size used is too small to read
  3. No major issue in usability for Standard Funds Transfer, Adding a payee and Deleting a payee
  4. Navigation through the whole application is deemed to be moderately difficult among users. Users commented that the segments of the web application are unclear
  5. For points that users like, many users commented that it is easy to navigate the web application. The basic neat design is well-liked by the users
  6. For points that users dislike, some users reflect that they had problems reading the font due to its size and it being grey in colour. Few users reflect that there is too much white space for some pages
  7. The general impression of the application is that it is relatively easy to use

Deployment exercise

Objective

The objectives of the deployment exercise is to carry out stress testing of the our application (web and android) and to identify any bugs present in our application that we have not surfaced out during our own testing.

Participants, Timing and Venue held

Number of participants: 45 students from the Retail Banking class
Date and Time held: 2 October 2013 (8.15am - 9.30am)
Venue: SMU

Implementation

Participants are required to go through a Retail Banking Lab that involves the mobile and internet banking functions. 2 of the members will be facilitators of the exercise to help the participants should they face any problems with the application.

After going through the lab exercise, participants are encouraged to give any feedback to the facilitators.

Functions Tested

Internet Banking

No. Functions Tested
1 Login 2FA
2 View account information
3 View transaction history
4 Fund Transfer
5 Set up Customer Preferences

Mobile Banking

No. Functions Tested
1 Login 2FA
2 View account information
3 View transaction history
4 Fund Transfer
Follow Up

The facilitators will collect the feedback and communicate to the members of the group to make any necessary changes to the application (Android and web).