IS480 Team wiki: 2014T1 Happy Sounds User Test 2
HOME | PROJECT OVERVIEW | PROJECT MANAGEMENT | DOCUMENTATION | ABOUT US |
DIAGRAMS | TESTING | PROTOTYPE | PRESENTATION MATERIALS |
INTERNAL TESTING | USER TEST 1 | USER TEST 2 | USER TEST 3 |
---|
Contents
- 1 User Test 2 (UT2) Test Plan
- 2 User Test 2 (UT2) Consolidated Results
- 2.1 Breakdown of the 75 users
- 2.2 Average Rating for Functionality and User Experience (Mobile vs Web Application)
- 2.3 Group by User and Application type
- 2.4 Average Functionality Rating based on User Experience
- 2.5 Average Functionality Rating based on Importance
- 2.6 Qualitative Feedback & Problems
- 2.7 Recommendations
- 2.8 Follow-up actions (Changes to be made)
- 3 Changes Before & After UT2
- 4 UT2 Documentations
User Test 2 (UT2) Test Plan
Number of Users: 75 existing Carousell users & New users
Venue: At user's convenience
Date:04/10/2014 to 06/10/2014
Duration: 45 mins per user
Objectives:
- Gather feedback regarding user interface of developed functions
- Detect usability issues based on user behavior
- Find out if new users are able to easily learn and use the application
- Find out if Offer Management is effective for different groups of users
- Find out which developed functions are the most important & provides the best user experience
- To use UT results to improve our web application
Scope of Testing:
- Browse Product Listings
- Search Product
- Filter Product
- Search Users
- View User Profile and Listings by User
- View Product Details
- View & Add Product Comment
- Buy Now (Chat)
- Offer Management (Chat to Buy / Filters / Sort / Search)
User Test 2 (UT2) Consolidated Results
The following are the consolidated UT2 results from the 75 users from this UT.
Breakdown of the 75 users
56 of the users are existing Carousell users and the remaining 19 are new users who have not used Carousell mobile application prior to the test.
To obtain an in-depth analysis of the 56 existing Carousell users, we categorized them into 3 different groups, namely the Power Users, Buyers and Sellers. Users are categorized as Power Users if they launched the Carousell mobile application more than 7 times a week. After grouping the users, there are a total of 17 power users, 26 buyers and 13 sellers.
Average Rating for Functionality and User Experience (Mobile vs Web Application)
The 56 existing Carousell users were first asked to rate the Functionality and User Experience for the mobile and web application on a scale from 1 to 5, 1 being the worst and 5 being the best. New Carousell users were not asked this question as they do not have the experience with Carousell mobile application and will be unable to provide the rating for the mobile application.
Overall
From the 56 responses, we calculated the average rating (on the same scale from 1 to 5) based on the individual rating and created a bar chart to visualize the result.
Both the average rating for functionality and user experience between web and mobile application are similar with the mobile application slightly edging out the web application in both aspects.
For functionality rating, mobile application got an average score of 3.9, which is higher than web application by a minor difference of 0.1. As for user experience rating, mobile application got an average rating of 3.9 and the difference is 0.2.
Although we are only half-way through the project, web application loses out to mobile application by only a small margin. Our team hopes that by the end of the project, the web application will be able to offer Carousell users a similar or better functionality and user experience compared to the mobile application
Group by User
We then grouped the existing users into Buyer, Seller and Power User and calculated the average rating for the functionality and user experience for the mobile and web application.
As seen from the graph above, buyers generally gave the same rating for both functionality and user experience across the 2 platforms as average rating ranges from 3.7 to 3.8.
Sellers have the lowest rating of 3.6 for web application functionality between the 3 groups. This might be due to the lack of seller’s functionality such as Listing and Selling of product using the web application.
As for power users, they seem to be rather satisfied with the functionality developed for the web application so far as this group of user has the highest average rating of 4.2. However, given that the average rating of 3.8 for web user experience is lower than 4.1 for mobile application, power users do not feel that the web has a better or similar user experience as compared to the mobile application.
Group by User and Application type
A further comparison between mobile and web application shows that the the minute difference of 0.1 for web app functionality from mobile app functionality is largely due to the rating provided by the sellers.
Average rating for sellers for mobile application is 4.0 while the average rating for web application is 3.6. A possible reason to this is due to the absence of the List/Sell functionality in this UT.
As for the average rating for user experience between web and mobile application, the average rating by buyers and sellers are similar with only 0.1 difference. However, for power user, their average rating for both version of the application differs by 0.3 and this affected the overall rating the most. Some power users felt that the difference in user experience is due to the design of the web application as it felt like an oversized mobile application. The white spaces in the different pages could be better utilized and unnecessary scrolling should be minimized.
Average Functionality Rating based on User Experience
Overall
Users were then asked to rate the 12 tested functionalities based on User Experience on a scale from 1(Worst) to 5 (Best). The average values for the functionality rating based on user experience ranges from 3.5 to 4.1.
The top 3 functionalities that offer the best user experience are Search Products, View Listings by user and View & Add Comments to Products. The other functionalities, with the exception of Search for Users, also had good average rating for user experience as the average rating ranges from 3.9 to 4.1. Search for Users had an average rating of 3.5 and this is mainly due to the function being not intuitive as most users were unaware that it is required to click on the search toggle to switch between product and user search.
Group by User
We then categorized the users into the 3 groups and calculated the average rating for the tested functionalities to analyze how the user experience fare across different groups of user.
According to the average rating for buyers, they generally felt that the user experience for all the tested functionalities were good as the average rating was ≥ 4 with the only exception being Search for Users which is rated as 3.8.
Sellers felt that View Listings by User and Offers Management (chat) user experience is good as the average rating is 4.3 and 4.1 respectively. On the other hand, the functions that offered the least user experience are Search for Users and Buy Now (Chat) with the average rating of 3.6. The lower average rating for Buy Now (Chat) might be due to sellers preferring Offers Management (Chat) as it is easier to manage their offers received from buyers as opposed to navigating to the individual product detail pages to reply offers received.
Power users are generally more critical as the average rating for Search for Users is 3.2, which is the lowest across the user groups. The average rating of 4.5 for Search Products is also the highest across the user groups.
As for new users, the average rating of the functionalities with the exception of Search for users ranges from 3.8 to 4.2. This proves that our application is intuitive and does not have a very steep learning curve. However, similar to the other user groups, Search for users did not offer them a good experience as they were unaware of this function due to the search toggle.
From the average rating for functionalities user experience, we learnt that Search for Users is still not intuitive despite the change from UT1 as many users had trouble locating this function. Furthermore, we also found out that Offer Management (Chat) average rating is consistent across the user groups and it ranges from 4 to 4.1. This proves that Offer Management (Chat) does not have much usability issues despite it being an entirely new function that is available only on the web application.
Average Functionality Rating based on Importance
Overall
Users were also asked to rate the functionalities based on Importance on the same scale from 1(Worst) to 5 (Best). The average values for the functionality importance rating ranges from 3.8 to 4.4 and the difference is not big.
Group by User
According to the chart above, buyers felt that View Listings by User is the least important while Browse Product Listings, Product Filters and Search Products are the most important. This seems to validate our assumption that Browse and Search Product is more important to buyers.
As for sellers, the most important functionality is Search Product while least important functionalities are Product Filters, View & Add Product Comments and Offers Management Filters & Sort. Although, the average rating of 4.1 for Offer Management (Chat) is higher than Browse Products average rating of 3.9, it is not consistent with our assumption as thought that sellers will feel that Offers Management (Chat) will be the most important functionality.
Power users felt that the more important functionalities are Browse Product Listings, Search Product, View Product Details and Offers Management (Chat). The least important function is Search for user with the lowest rating of 3.6 across the user groups.
New users’ least important functionalities are Offer Management (Chat) and Offer Management (Search). This might be because new users lack the experience and are unaware of the difficulties with managing multiple offers on the mobile application.
From the average rating for functionalities importance, we found out that Search Product is the most important while Offers Management (Search / Sort/ Filters) are less important and this is consistent across the different user groups. Possible reasons might be due to Offer Management being a new function that is available only on the web application and users might not find these features useful as they do not know how to use the features under Offer Management. However, it should be noted that the average ratings are very close and it might possibly mean that all the functions are important.
Qualitative Feedback & Problems
- Category pictures on landing page are too big and this resulted in too much unnecessary scrolling
- Search toggle for product and user search is not intuitive as many users have trouble finding Search for User function
- Some icons / elements are not meaningful and users do not know what it represents. For example, the “+”, “-“, “o” icons on Product Details and User Profile page
- Users were unsure if login is required to be able to add comments on Product Details page
- The text displayed when no listings are found (No listings found. Here are some recently liked products instead!) is not obvious
- The button text “Buy Now” on Product Details page is inaccurate as the user has to negotiate the product price with the seller before purchasing the product
- Unsure of how to use Offers Management (Filters)
- Unable to like products and follow user
Recommendations
- Able to edit product details
- More specific product categories such as Accessories in For Her category
- Reduce size of category pictures on landing page
- Add in message (information) of icons/elements on hover
- Improve on Search for Users
- Able to like product on Browse page
- Zoom-in feature for main product image on Product Details page
Follow-up actions (Changes to be made)
- Include text (product / user) or change background color for Search toggle to make user search more obvious
- Reduce size of category pictures on landing page to minimize unnecessary scrolling
- Change product filters to make it more explicit and obvious
- Change product filters text input to slider
- Add message that indicate login is required for adding comments
- Increase the spacing between the text “No listings found. Here are some recently liked products instead!”, and recommended product image cards
- Change button text “Buy Now” to “Chat to Buy” on Product Details page
- Login prompt to inform user that login is required to use certain features
Changes Before & After UT2
Changes made | Before | After |
---|---|---|
Changed Search toggle div background to make the toggle more obvious | ||
"Buy Now" button on product details page to "Chat to Buy" | ||
Reduced category image size on landing page to minimize unnecessary scrolling | ||
Changed Product Filters buttons to make it more explicit and obvious to the user | ||
Changed Product Filters text input to slider | ||
Added message to indicate login is required to leave a comment | ||
Increased spacing between “No listings found. Here are some recently liked products instead!” message and product cards |
New Additions
Changes | After Change |
---|---|
Added Login Prompt to inform unauthenticated users that login is required to use certain features (Like Product, Offers, Chat to Buy) |