HeaderSIS.jpg

IS480 Team wiki: 2016T1 Charlies-Angels-QA

From IS480
Revision as of 22:46, 18 November 2016 by Xwzheng.2014 (talk | contribs)
Jump to navigation Jump to search


Charlies Angels Logo.png


Charlies Angels Home.png  HOME

 

Charlies Angels About Us.png  ABOUT US

 

Charlies Angels Project Overview.png  PROJECT OVERVIEW

 

Charlies Angels Project Management.png  PROJECT MANAGEMENT

 

Charlies Angels Project Documentation.png  DOCUMENTATION

 
Technical Documents Wireframes Charlies Angels Current Stage.png Quality Assurance Meeting Minutes Deliverables

Survey Findings

Survey Findings

Key Findings from Survey

This survey is meant to gather sentiments towards the features of our application that is intended to be rolled out (i.e. Dashboard, Robot Advisor). The targeted audience in this survey are millenials whom have little or no investment experience.

Key Findings:

  • Total no. of participants: 50
  • We have a fair mix of composition of gender, with 64% males and 36% females
  • All participants were aged between 20-29
  • 96% of our participants were Chinese, and 4% were Malay
  • 22% of our participants have began working full time, 76% have not began, and the remaining 2% are currently looking for jobs
  • We discovered that 92% of participants place between $0-$500 monthly into investments, or 8% > $1000. Surprisingly, there seem to not be a 'moderate' range of $501- $1000
  • 68% of participants have not began investment, while the other 32% being distributed across several ranges
  • 96% of participants felt that a feature to consider CPF and Housing Loan would be a useful one in a portfolio management

Graphical breakdowns of the survey can be found below:
Charlies Angels Survey Age.pngCharlies Angels Survey Gender.png
Charlies Angels Survey Race.pngCharlies Angels Survey Working.png
Charlies Angels Survey InvestmentExp.pngCharlies Angels Survey InvestmentAmt.png
Charlies Angels Survey CPF.png



We have also took our initial design of the application, specifically for the Robot Advisor to the users, and have requested for their feedback regarding the User Interface and Experience. The reason why we narrowed down to specifically Robot Advisor as it is a never-before-seen feature in any portfolio management application. The following are screenshots of the design that was shown to the users:

Screenshots of CPF-Robot Advisor

Screenshot #1: Charlies Angels CPF 1.jpg
Screenshot #2: Charlies Angels CPF 2.jpg
Screenshot #3: Charlies Angels CPF 3.jpg

User Feedback via Survey

Key Comments:

  • Having to input so many fields at once is really troublesome. It is a turn off!
  • I have not even began working, I don't have CPF, so effectively having me to enter 0s does not make sense
  • I am not buying a house so soon, housing loan does not affect me
  • Maybe a segmented survey would be more pleasing to the eyes

With these comments, we have properly evaluated that the user interface created was horrible and required immediate actions to be taken. This is despite of the good intentions of being thorough and precise when it came to calculations. Instead, we have considered what are the most essential of inputs, and have changed the user interface to ensure a more seamless experience for the user to utilise the feature.

Survey #2 Results

Charlies Angels Survey 2 Results 1.PNGCharlies Angels Survey 2 Results 2.PNG
Charlies Angels Survey 2 Results 3.PNG Charlies Angels Survey 2 Results 4.PNG
Charlies Angels Survey 2 Results 5.PNG Charlies Angels Survey 2 Results 6.PNG
Charlies Angels Survey 2 Results 7.PNG Charlies Angels Survey 2 Results 8.PNG

Survey #3 Results

Charlies Angels Survey 3 Results.jpg

Regression Testing

Regression Testing
Results of Internal Unit Testing
Iteration No. Date Results
3 06 JUL 2016 33/36; Bug Metrics of 7 - Hitting 1 High Impact & 2 Low Impact
4 27 JUL 2016 69/74; Bug Metrics of 9 - Hitting 1 High Impact & 4 Low Impact
5 10 AUG 2016 87/90; Bug Metrics of 7 - Hitting 1 High Impact & 2 Low Impact
6 29 AUG 2016 101/105; Bug Metrics of 8 - Hitting 1 High Impact & 3 Low Impact
7 5 SEP 2016 105/107; Bug Metrics of 6 - Hitting 1 High Impact & 1 Low Impact
8 13 SEP 2016 119/120; Bug Metrics of 5 - Hitting 1 High Impact

UT 1 Results

UT Results & Findings


Date: 05 Aug 2016 - 10 Aug 2016
Venue: Online
Duration: At their own timing
Number of Participants: 10
Target Audience: Users with Beginners' Investment Background



Objectives:

  1.   Identify functionalities and usability issues
  2.   Determine is the user interface is intuitive
  3.   Discover ways to improve features and functions of the website based on the results of User Testing #1




Scope:
Functionalities tested are ones developed between Iteration 3 and 4

  1.   Account Management
    • Log In
    • Log Out
    • Registration
    • Change Password
    • Forget Password
  2.   Search and Display
    • Display Fundamental Information (Live Data)
    • Display Financial Information (Live Data)
  3.    iStockTrade
    • Add/Delete Stock
    • Create/Delete Portfolio
    • Delete Transactions
  4.    StockBook Portfolio Stocks
    • Display Fundamental, Financial Information (Live Data) & Transaction History of the owned stocks
  5.    StockBook Portfolio Summary
    • Display a summary of user's portfolio




Procedure:

  1.   A UT document with instructions on how to test the website have been given to the testers
  2.   Participants will log down the testing outcomes & time taken per task in the sheet
  3.   Participants are welcomed to type their general comments and feedback into the feedback textbox provided at the end of the test plan document




Tasks:
The UT document requests participants to perform the following task(s):

  1. Registration
  2. Login
  3. Logout
  4. Change Password
  5. Forget Password
  6. Create a Portfolio
  7. Add (Buy) Few Stocks to the Portfolio
  8. Remove (Sell) a Stock from the Portfolio
  9. Delete Portfolio


For every task, participants were required to document down the following information:

  • Execution of Task (Successful or Unsuccessful)
  • Outcome
  • Time Taken


At the end of the test plan, we also asked the participants the following questions:

  • Do you have investment knowledge prior to using this application?
  • Any Comments?


Results

For UT #1, we tasked participants to perform the testing online, without assistance from the team, to best determine the state of the application

S/N Objective Achieved? % of Participants that found this task easy or very easy
1 Participants should be able to register for an account Yes 100%
2 Participants should be able to login Yes 100%
3 Participants should be able to change their password Yes 100%
4 Participants should be able to create a new portfolio Partially 66.67%
5 Participants should be able to add three stocks of their choice Partially 50%
6 Participants should be able to delete a stock from their portfolio No 25%
7 Participants should be able to delete their portfolio Yes 100%
8 Participants should be able to log out Yes 100%


Key Findings


Overall Findings

  • The website has a very clean and modern design
  • The website is intuitive
  • Nothing new about what has been created so far from what is out there
  • Some functions are still breaking, need to ensure fully fixed before testing



Key Improvements to be made

  • Ensure that the application does not break before deployed for testing
  • There is nothing unique of this application as yet. However, we recognise that this is because the foundation of the application needs to come first
  • Interface looks good, can continue adding colour schemes to make the UI more intuitive
  • Future UTs shall not be conducted via online means. Instead, hold a testing environment with physical presence to better gauge time taken for task completion


UT 2 Results

UT Results & Findings

Date: 15 Sep 2016 - 16 Sep 2016
Venue: School of Information Systems, Singapore Management University
Duration: 15 Sep 2016: 0900 - 1200 Hours @ SIS CR 3-2 & 1800 - 2100 Hours @ SIS CR 3-1; 16 Sep 2016: 0900 - 1200 & 1800 - 2100 Hours @ SIS CR 3-2
Number of Participants: 20
Target Audience: Millennials; aged 16-36 with Basic or No Investment Background



Objectives:

  1.   Identify functionalities and usability issues
  2.   Determine is the user interface is intuitive
  3.   Discover ways to improve features and functions of the website based on the results of User Testing #2
  4.   Introduce CPF & Housing Loan calculations - Never seen before feature in the market




Scope:
Functionalities tested are ones developed between Iteration 3 and 8




Procedure:

  1.   A UT document with instructions on how to test the website have been given to the testers
  2.   Participants will log down the testing outcomes & team members would note the time taken per task in the sheet
  3.   Participants are welcomed to type their general comments and feedback into the feedback textbox provided at the end of the test plan document




Tasks:
The UT document requests participants to perform the following task(s):
For every task, participants were required to document down the following information:

  • Execution of Task (Successful or Unsuccessful)
  • Outcome
  • Time Taken


  • Login & Registration Test Cases
    • Register a New Account
    • Retrieve your Password
    • Login
    • Change your Password
  • Portfolio Test Cases
    • Create a Portfolio
    • Search for a Stock
    • Buy a Specific Stock
    • Buy Two Other Stocks
    • Sell One Stock
  • PDF & Excel Test Cases
    • View your Portfolio Summary
    • Generate PDF Report of your Portfolio
    • View your PDF Report summary and verify if the information is correct
    • Generate Excel Report of your Portfolio
    • View your Excel Sheet and verify if the information is correct
    • Delete your Portfolio
  • Watch List Test Cases
    • Add any Two Stocks into your Portfolio
    • Delete the stocks from your watch list


At the end of the test plan, we also asked the participants the following questions:

  • Demographics Questions such as Age, Race, Work Experience, Investment Habits
  • Do you have investment knowledge prior to using this application?
  • Any Comments?


Results

Demographics of our testers

  • Total number of testers: 20
  • Age group:
    • 0 testers aged between 16 and 19
    • 19 testers aged between 20 and 29
    • 1 tester aged above 30 years old
  • Working Profile:
    • 3 testers are currently working
    • 17 testers are currently not working
  • Investment Background:
    • 12 of 20 testers indicated no background
    • 3 out of 20 testers indicated investment background of half a year
    • 3 out of 20 testers indicated investment background of 1 year
    • Remaining 2 testers indicated investment background of 2 years


Login & Registration Test Cases

Function No. of Testers No. of Errors Avg. Time (Sec) Fastest (Sec) Slowest (Sec) Comments
Registration 20 0 36.75 16 56 Username req too long
Retrieve Password 20 0 32 7 118 Email can be formatted better; Password that is resetted is too long; Let user see password field
Login 20 0 12.31 2 41 NIL
Change Password 20 0 14.63 3 32 Consider a confirm password field


Portfolio Test Cases

Function No. of Testers No. of Errors Avg. Time (Sec) Fastest (Sec) Slowest (Sec) Comments
Create Portfolio 20 1 11.63 6 22 Can't create a portfolio called "test" but it appears in dropdown when I buy stocks
Search for DBS Stock 20 0 10.19 5 24 NIL
Buy DBS Stock 20 0 18.38 6 33 Page to show all stocks would be good; Buy/Sell like trading. Consider changing to add/remove.
Buy 2 more Stocks 20 0 58 2 126 Populate last done price in modal & quantity too
Sell any 1 Stock 20 1 46.13 14 95 Loading bar is too small; Confirmation box upon selling


PDF & Excel Test Cases

Function No. of Testers No. of Errors Avg. Time (Sec) Fastest (Sec) Slowest (Sec) Comments
View Portfolio 20 0 4.13 2 16 NIL
Generate PDF Report 20 0 20 3 54 Inform user that they already have clicked the PDF and show them the progress; Feature looks good. Very Useful; Might want to add disclaimer; Popup blocker will disable download, inform users if it is the case
View PDF 20 0 13 4 48 Sold all Genting stocks but they are still under winning stocks; Trust/warrants not in the PDF.
Generate Excel Report 20 3 7 2 126 Inform user that download has started; Icon looks like delete button
View Excel Report 20 3 46.13 14 95 Format Excel nicely please
Delete Portfolio 20 0 5 1 15 Confirmation box upon delete


Watchlist Test Cases

Function No. of Testers No. of Errors Avg. Time (Sec) Fastest (Sec) Slowest (Sec) Comments
Add any 2 Stocks 20 1 29.67 10 55 NIL
Delete the Stocks 20 1 7.73 3 17 Confirmation box upon delete


No. Risk Profiling
1 A bit of doubt on accuracy of the app but good to have. Good functions.
2 Icon is misleading. Thought its settings icon; Setup during registration better for me; Put in profile should not be changed all the time. Some investors are greedy and wanna see rewards in short amt of time
3 Useful function; Put more questions for more accuracy
4 Form was ok
5 Assumes few questions to generate this > GOOD! Very good profiling. Questioning on accuracy of quarters
6 Robot advisor icon not clear. Did not know what that button is for. Breakdown is too general, because large cap stocks have different yield and prices
7 Good amt of questions. Trust that the report is accurate. Useful!
8 Good feature. Recommendation is fair enough. Default profile set looks suspicious
9 Decent. Consider as a pop-up to force reader to read.
10 Limited and some questions are similar. Q2 is too long. Breakdown is not clear. Change to another name!
11 include reasons for breakdown. Default not convincing. Change robot word to financial advisor.
12 Questions are okay. Function is useful. Can provide directions (what you should do)
13 -Remove default values, describe functionality instead and tell user what to do; Q2 too long, intimidating; Q1-3, 2-4, 3-2, 4-4, 5-1; Breakdown is very very very conservative for my input; Explain the terms like tooltip/link e.g. ETF
14 Useful and tell which things to buy. No. of questions are good. Not too long
15 Good feature. Questions just nice.
16 Q4 time range? Define it. Taken aback by form's default values. Should put robot vertically.
17 The default recommendation confused me
18 I like the allocation done, but some parts seem wordy.
19 Description abit wordy but otherwise its ok.
20 Question 2 needs to be rephrase a bit


No. Robot Advisor
1 Good feature! Retirement analyser needs to have more output
2 Error should not stay so long; Increment and decrement makes no sense; Assumptions why got scroll bar; Describe that the est interest rate is automatically populated; Link with robot advisor? How to invest?; Give $ to earn as retirement goal
3 Form is ok, need to be more detailed; Can include additional salary per month, death age and future value at retirement.
4 Very useful. Inputs are ok ------- More details about retirement given would be helpful
5 Light bulb looks like an icon; Too many fields to fill in; Best if can use 1 accordion; Too many tabs is too troublesome for me; More details provided would be good, like telling me exactly how far I am away from my goal, maybe display a chart or something?
6 Good! Cpf and housing loan feature is fresh and very relevant; The application showed me how much I need to earn every month to reach my retirement goal. Its too vague and I don’t know what to do with the information!
7 Good feature, but input a lot of stuff and only getting 1 line of results is a bit disappointing
8 Sentencing structure is weird; Useful feature but questions is how it ties to business environment; Investment should be closer to how my portfolio is doing and advise
9 Useful to have; How would this affect my current investment goals? //might be a problem if they deleted their portfolio?

Would be good if there are clearer action steps for me.

10 Retirement nest value = 2 decimal places; if I can't hit the goals, what to do next?
11 Good function; Expected monthly salary? Or what salary?; Monthly salary based on monthy savings?; Make the function more prominent. It looks like fine print.

Would be good if you show me how exactly I can hit my retirement goals

12 Calculation might be wrong for retirement nest; Lack of info makes it unappealing.
13 Inflation rate put 4%, which is cpf rate. So usually investment must be more than 4%; Target pension term in sg context is it useful?; how much do you need per month (based on?); 2 decimal places; make assumptions tooltip more obvious, e.g. color; more explanation about retirement portion would be good
14 Very good. Information to earn is there; No. of questions is just nice.
15 A lot of inputs but will have a more accurate answer; Try to fit the inputs in a form, very lenghty; The amount of inputs I have to go through just to get 1 sentence of calculation does not seem worth it
16 -Expected salary? Entry level salary? ; Bank interest rate in savings acc is negligible; CPF details link to cpf site; Average salary income increment would be good; This feature is good. ; Feature is useful but too vague to be useful.
17 Telling me how to hit my retirement plans would be more useful than telling me to earn 10k a month for the next 40 years.
18 Feature is interesting, but lacks useful information.
19 NIL
20 Useful feature, but no advice was provided
Key Findings


Overall Findings

No. Overall Comment(s)
1 Love it. Simple and intuitive to use.
2 Ui is not bad. Just the icon; Customize dashboard?; Homepage alignment; Filter for STI components; User icon beside username should always be there.
3 Simple ui.
4 Portfolio summary stocks have alert?; Easy to use, very clear ui
5 More data in graphs! But very good features so far
6 At dashboard the portfolio sector shows 2 stocks but I only have 1 stock; System is slow. Suggest to replace p/e, yield, p/b to ask/bid price; Fix decimal places. Show 2 decimal places; Portfolio robot advisor got error
7 Nil
8 UI easy and straightforward; Some icons needs to be clearer (robot advisor)
9 Dashboard can be neater! Make sure information and actors ar ebetter represented.
10 Make stockbook more obvious at the side nav bar; Search bar still contains stock after logging out.
11 Ui needs improvement. Keep up the good work.
12 Two scrollbar, horizontal bar not needed; Export pdf text should be made clickable. Design is nice.
13 Confirmation box for functions; Color and change text to remove after stocks is added; Information is straightforwrd, easy to find. Things are informative; Search bar is easy to find. Dashboard page quite long, need more organization; Ui is nice and smooth; Purple is weird for stock investment, does not feel professional; Confused as to targeted audience of app, some parts are filled with technical jargons while others feel like a guiding app; Robot advisor and retirement calculation is a good way to stand out; Try to add more value towards this direction since portfolio mgmt app is easily available.
14 UI is quite nice. Easy to navigate around the app.
15 UI cluttered; Under portfolio, icon seems out of place; No search results notification. Indicate that stocks are already in watchlist.
16 Filter equities to the top for search; UI is plain; Homepage feels weird. Icon should be linked to home page; STI component time looks off; Put div data in a table and change PE ratio to P/E ratio. Excel to have opening/closing price over a range of timing.
17 UI needs more graphs. A bit plain looking now.
18 Application looks promising but there are rooms for improvement. Retirement analyser portion really needs more info. If that is supposed to be your unique feature with a Singaporean element, make it more useful by providing useful info.
19 Keep up the good work!
20 Ui is simple and straightforward. Like it a lot.



Key Improvements to be made

  • Consider more visualizations
  • Continue enhancing the User Interface
  • Continue user-centric development as it is proving to be great!



UT 3 Results

UT Results & Findings

Date: 16 Nov 2016
Venue: School of Information Systems, Singapore Management University
Duration: 16 Nov 2016: 1530 - 1900 Hours @ LKCSB CR 2-1
Number of Participants: 40
Target Audience: Millennials; aged 16-36 with Basic or No Investment Background



Objectives:

  1.   Identify functionalities and usability issues
  2.   Determine is the user interface is intuitive
  3.   Discover ways to improve features and functions of the website based on the results of User Testing #3




Scope:
Functionalities tested are ones developed between Iteration 3 and 11




Procedure:

  1.   A UT document with instructions on how to test the website have been given to the testers
  2.   Participants will log down the testing outcomes & team members would note the time taken per task in the sheet
  3.   Participants are welcomed to type their general comments and feedback into the feedback textbox provided at the end of the test plan document




Tasks:
The UT document requests participants to perform the following task(s):
For every task, participants were required to document down the following information:

  • Execution of Task (Successful or Unsuccessful)
  • Outcome
  • Time Taken


  • Login & Registration Test Cases
    • Register a New Account
    • Retrieve your Password
    • Login
    • Change your Password
  • Portfolio Test Cases
    • Create a Portfolio
    • Search for a Stock
    • Buy a Specific Stock
    • Buy Two Other Stocks
    • Sell One Stock
  • PDF & Excel Test Cases
    • View your Portfolio Summary
    • Generate PDF Report of your Portfolio
    • View your PDF Report summary and verify if the information is correct
    • Generate Excel Report of your Portfolio
    • View your Excel Sheet and verify if the information is correct
  • Watch List Test Cases
    • Add any Two Stocks into your Portfolio
    • Delete the stocks from your watch list
  • Risk Profiling Test Case
    • Perform the risk profile and give feedback accordingly
  • Robot Advisor Test Case
    • Perform the Robot Advisor and give feedback accordingly
  • Dividend Tracker Test Case
    • Generate the Dividend Tracker Report for any one of the stock added to your portfolio
  • User Help Display Feedback
    • After observing the user interface and give feedback accordingly


At the end of the test, we also asked the participants to fill the following post-UT questionnaire at the following URL: http://tinyurl.com/stockbook


Results

Demographics of our testers

  • Total number of testers: 40
  • Age group:
    • 4 testers aged between 16 and 21
    • 33 testers aged between 22 and 25
    • 7 testers aged between 26 and 30
    • 0 testers aged above 30 years old
  • Working Profile:
    • 6 testers are currently working
    • 34 testers are currently not working
  • Investment Background:
    • 34 of 40 testers indicated investment background within a year
    • 5 out of 40 testers indicated investment background of 2 to 4 years
    • 1 out of 40 testers indicated investment background of more than 4 years
  • Have you tested for us before in UT 2?
    • 9 out of 40 testers tested for us before
    • 31 out of 40 testers are new to the application


Login & Registration Test Cases

Function No. of Testers No. of Errors Avg. Time (Sec) Fastest (Sec) Slowest (Sec) Comments
Registration 40 0 35.95 16 78
Retrieve Password 40 0 28.12 10 84 Inform me that password is sent to my email
Login 40 0 6.4 2 18
Change Password 40 0 17 2 80 Consider a Confirm Password field.
Consider showing password strength


Portfolio Test Cases

Function No. of Testers No. of Errors Avg. Time (Sec) Fastest (Sec) Slowest (Sec) Comments
Create a Portfolio 40 0 29.8 5 95 Button can be properly aligned
CDP Import button is confusing
Search for DBS Group Holdings stock 40 0 18.6 3 98 Show a list of stocks available
Buy DBS Group Holdings stock 40 1 17.12 2 152 Font a bit too small
Error message for decimal quantity wrong
Unable to select portfolio, solved after refresh
Buy any 2 other stocks 40 0 59.9 12 153 Loading bar can put in centre?
Change colour when hovering over
Sell any stock from your portfolio 40 0 43.55 7 155 A little difficult to find. Consider adding a delete button by the side of the stock.
Would be good if you showed indication that the stocks are listed below


PDF & Excel Test Cases

Function No. of Testers No. of Errors Avg. Time (Sec) Fastest (Sec) Slowest (Sec) Comments
View Portfolio 40 0 3.85 1 30 Colour the words 'buy' and 'sell' differently
Generate PDF Report 40 0 39.87 2 152 Feature is intuitive. It is useful
Popup blocker will disable download. Inform users if that is the case.
View PDF 40 0 3.15 1 10 Put your brand on the PDF report
Generate Excel Report 40 0 10.6 1 46
View Excel Report 40 0 3.1 1 10 Put your brand on the Excel report


Watchlist Test Cases

Function No. of Testers No. of Errors Avg. Time (Sec) Fastest (Sec) Slowest (Sec) Comments
Add any 2 stocks 40 1 44.05 7 90 Easy to use
Delete the stocks 40 0 7.85 1 14


Dividend Tracker Test Cases

Function No. of Testers No. of Errors Avg. Time (Sec) Fastest (Sec) Slowest (Sec) Comments
Generate the Dividend Tracker Report of any of the stock you have added earlier in your portfolio 40 3 31.66 9 139 Some users experienced Error 500.


Function No. of Testers No. of Errors Avg. Time (Sec) Fastest (Sec) Slowest (Sec) Comments
In Progress In Progress In Progress In Progress In Progress In Progress In Progress
Key Findings


Overall Findings

No. Overall Comment(s)
In Progress In Progress



Key Improvements to be made

  • In Progress