HeaderSIS.jpg

IS480 Team wiki: 2014T1 Happy Sounds User Test 2

From IS480
Revision as of 21:39, 10 November 2014 by Qiyang.tan.2012 (talk | contribs)
Jump to navigation Jump to search

Happysounds logo.png

HomeButton.png HOME   Glyphicons 042 pie chart.png PROJECT OVERVIEW   Calendaricon.png PROJECT MANAGEMENT   Tableicon.png DOCUMENTATION   Aboutus.png ABOUT US
DIAGRAMS   TESTING   PROTOTYPE   PRESENTATION MATERIALS
INTERNAL TESTING USER TEST 1 USER TEST 2

User Test 2 (UT2) Test Plan

Number of Users: 75 existing Carousell users & New users
Venue: At user's convenience
Date:04/10/2014 to 06/10/2014
Duration: 45 mins per user
Objectives:

  • Gather feedback regarding user interface of developed functions
  • Detect usability issues based on user behavior
  • Find out if new users are able to easily learn and use the application
  • Find out if Offer Management is effective for different groups of users
  • Find out which developed functions are the most important & provides the best user experience
  • To use UT results to improve our web application

Scope of Testing:

  • Browse Product Listings
  • Search Product
  • Filter Product
  • Search Users
  • View User Profile and Listings by User
  • View Product Details
  • View & Add Product Comment
  • Buy Now (Chat)
  • Offer Management (Chat to Buy / Filters / Sort / Search)

User Test 2 (UT2) Consolidated Results

The following are the consolidated UT2 results from the 75 users from this UT.

Breakdown of the 75 testers

56 of the testers are existing Carousell users and the remaining 19 are new users who have not used Carousell mobile application prior to the test.
To obtain an in-depth analysis of the 56 existing Carousell users, we categorized them into 3 different groups, namely the Power Users, Buyers and Sellers. Users are categorized as Power Users if they launched the Carousell mobile application more than 7 times a week. After grouping the users, there are a total of 17 power users, 26 buyers and 13 sellers.

HS userbreakdown 1.png HS userbreakdown 2.png

Average Rating for Functionality and User Experience (Mobile vs Web Application)

The 56 existing Carousell users were first asked to rate the Functionality and User Experience for the mobile and web application on a scale from 1 to 5, 1 being the worst and 5 being the best. New Carousell users were not asked this question as they do not have the experience with Carousell mobile application and will be unable to provide the rating for the mobile application.

Overall

From the 56 responses, we calculated the average rating (on the same scale from 1 to 5) based on the individual rating and created a bar chart to visualize the result.

HS ut2 web mobile comparison.png

Both the average rating for functionality and user experience between web and mobile application are similar with the mobile application slightly edging out the web application in both aspects.
For functionality rating, mobile application got an average score of 3.9, which is higher than web application by a minor difference of 0.1. As for user experience rating, mobile application got an average rating of 3.9 and the difference is 0.2.
Although we are only half-way through the project, web application loses out to mobile application by only a small margin. Our team hopes that by the end of the project, the web application will be able to offer Carousell users a similar or better functionality and user experience compared to the mobile application

Group by User

We then grouped the existing users into Buyer, Seller and Power User and calculated the average rating for the functionality and user experience for the mobile and web application.

HS ut2 web mobile comparison grouped.png

As seen from the graph above, buyers generally gave the same rating for both functionality and user experience across the 2 platforms as average rating ranges from 3.7 to 3.8.
Sellers have the lowest rating of 3.6 for web application functionality between the 3 groups. This might be due to the lack of seller’s functionality such as Listing and Selling of product using the web application.
As for power users, they seem to be rather satisfied with the functionality developed for the web application so far as this group of user has the highest average rating of 4.2. However, given that the average rating of 3.8 for web user experience is lower than 4.1 for mobile application, power users do not feel that the web has a better or similar user experience as compared to the mobile application.

Average Functionality Rating based on User Experience

Overall

Users were then asked to rate the 12 tested functionalities based on User Experience on a scale from 1(Worst) to 5 (Best). The average values for the functionality rating based on user experience ranges from 3.5 to 4.1.

HS ut2 overall UE.png

The top 3 functionalities that offer the best user experience are Search Products, View Listings by user and View & Add Comments to Products. The other functionalities, with the exception of Search for Users, also had good average rating for user experience as the average rating ranges from 3.9 to 4.1. Search for Users had an average rating of 3.5 and this is mainly due to the function being not intuitive as most users were unaware that it is required to click on the search toggle to switch between product and user search.

Group by User

We then categorized the users into the 3 groups and calculated the average rating for the tested functionalities to analyze how the user experience fare across different groups of user.

HS ut2 grouped UE.png

According to the average rating for buyers, they generally felt that the user experience for all the tested functionalities were good as the average rating was ≥ 4 with the only exception being Search for Users which is rated as 3.8.
Sellers felt that View Listings by User and Offers Management (chat) user experience is good as the average rating is 4.3 and 4.1 respectively. On the other hand, the functions that offered the least user experience are Search for Users and Buy Now (Chat) with the average rating of 3.6. The lower average rating for Buy Now (Chat) might be due to sellers preferring Offers Management (Chat) as it is easier to manage their offers received from buyers as opposed to navigating to the individual product detail pages to reply offers received.
Power users are generally more critical as the average rating for Search for Users is 3.2, which is the lowest across the user groups. The average rating of 4.5 for Search Products is also the highest across the user groups.
As for new users, the average rating of the functionalities with the exception of Search for users ranges from 3.8 to 4.2. This proves that our application is intuitive and does not have a very steep learning curve. However, similar to the other user groups, Search for users did not offer them a good experience as they were unaware of this function due to the search toggle.
From the average rating for functionalities user experience, we learnt that Search for Users is still not intuitive despite the change from UT1 as many users had trouble locating this function. Furthermore, we also found out that Offer Management (Chat) average rating is consistent across the user groups and it ranges from 4 to 4.1. This proves that Offer Management (Chat) does not have much usability issues despite it being an entirely new function that is available only on the web application.

Average Functionality Rating based on Importance

Overall

Users were also asked to rate the functionalities based on Importance on the same scale from 1(Worst) to 5 (Best). The average values for the functionality importance rating ranges from 3.8 to 4.4 and the difference is not big.

HS ut2 overall Importance.png

Group by User

HS ut2 grouped Importance.png

According to the chart above, buyers felt that View Listings by User is the least important while Browse Product Listings, Product Filters and Search Products are the most important. This seems to validate our assumption that Browse and Search Product is more important to buyers.
As for sellers, the most important functionality is Search Product while least important functionalities are Product Filters, View & Add Product Comments and Offers Management Filters & Sort. Although, the average rating of 4.1 for Offer Management (Chat) is higher than Browse Products average rating of 3.9, it is not consistent with our assumption as thought that sellers will feel that Offers Management (Chat) will be the most important functionality.
Power users felt that the more important functionalities are Browse Product Listings, Search Product, View Product Details and Offers Management (Chat). The least important function is Search for user with the lowest rating of 3.6 across the user groups.
New users’ least important functionalities are Offer Management (Chat) and Offer Management (Search). This might be because new users lack the experience and are unaware of the difficulties with managing multiple offers on the mobile application.
From the average rating for functionalities importance, we found out that Search Product is the most important while Offers Management (Filters) is the least important and this is consistent across the different user groups. This might be due to the use of Offers Management (Search) more than the filters. However, it should be noted that the average ratings are very close and it might possibly mean that all the functions are important.

Qualitative Feedback & Problems

  • Category pictures on landing page are too big and this resulted in too much unnecessary scrolling
  • Search toggle for product and user search is not intuitive as many users have trouble finding Search for User function
  • Some icons / elements are not meaningful and users do not know what it represents. For example, the “+”, “-“, “o” icons on Product Details and User Profile page
  • Users were unsure if login is required to be able to add comments on Product Details page
  • The text displayed when no listings are found (No listings found. Here are some recently liked products instead!) is not obvious
  • The button text “Buy Now” on Product Details page is inaccurate as the user has to negotiate the product price with the seller before purchasing the product
  • Unsure of how to use Offers Management (Filters)
  • Unable to like products and follow user

Recommendations

  • Able to edit product details
  • More specific product categories such as Accessories in For Her category
  • Reduce size of category pictures on landing page
  • Add in message (information) of icons/elements on hover
  • Improve on Search for Users
  • Able to like product on Browse page
  • Zoom-in feature for main product image on Product Details page

Follow-up actions (Changes to be made)

  • Include text (product / user) or change background color for Search toggle to make user search more obvious
  • Reduce size of category pictures on landing page to minimize unnecessary scrolling
  • Add HTML title attribute (info) for icon / element on hover
  • Add message that indicate login is required for adding comments
  • Increase the spacing between the text “No listings found. Here are some recently liked products instead!”, and recommended product image cards
  • Change button text “Buy Now” to “Chat to Buy” on Product Details page

Changes Before & After UT2

UT2 Documentations