IS480 Team wiki: 2014T1 Happy Sounds User Test 2
|HOME||PROJECT OVERVIEW||PROJECT MANAGEMENT||DOCUMENTATION||ABOUT US|
|INTERNAL TESTING||USER TEST 1||USER TEST 2||USER TEST 3|
- 1 User Test 2 (UT2) Test Plan
- 2 User Test 2 (UT2) Consolidated Results
- 2.1 Breakdown of the 75 users
- 2.2 Average Rating for Functionality and User Experience (Mobile vs Web Application)
- 2.3 Overall Functionality Rating (Power vs Moderate User)
- 2.4 Average Functionality Rating based on User Experience
- 2.5 Average Functionality Rating based on Importance
- 2.6 Qualitative Feedback & Problems
- 2.7 Recommendations
- 2.8 Follow-up actions (Changes to be made)
- 3 Changes Before & After UT2
- 4 UT2 Documentations
User Test 2 (UT2) Test Plan
Number of Users: 75 existing Carousell users & New users
Venue: At user's convenience
Date:04/10/2014 to 06/10/2014
Duration: 45 mins per user
- Gather feedback regarding user interface of developed functions
- Detect usability issues based on user behavior
- Find out if new users are able to easily learn and use the application
- Find out if Offer Management is effective for different groups of users
- Find out which developed functions are the most important & provides the best user experience
- To use UT results to improve our web application
Scope of Testing:
- Browse Product Listings
- Search Product
- Filter Product
- Search Users
- View User Profile and Listings by User
- View Product Details
- View & Add Product Comment
- Buy Now (Chat)
- Offer Management (Chat to Buy / Filters / Sort / Search)
User Test 2 (UT2) Consolidated Results
The following are the consolidated UT2 results from the 75 users from this UT.
Breakdown of the 75 users
56 of the users are existing Carousell users and the remaining 19 are new users who have not used Carousell mobile application prior to the test.
In order to do an in-depth analysis of the 56 existing Carousell users, we categorized them into 2 groups, namely the Power Users and Moderate User. Users are categorized as Power User if they launched the Carousell mobile application more than 7 times a week. After grouping the users, there are a total of 17 power users and 39 moderate user.
We also categorized our existing users into Buyer and seller. Out of the 56 existing users, 36 of them are buyers and the remaining 20 users are sellers.
Average Rating for Functionality and User Experience (Mobile vs Web Application)
The 56 existing Carousell users were first asked to rate the Functionality and User Experience for the mobile and web application on a scale from 1 to 5, 1 being the worst and 5 being the best. New Carousell users were not asked this question as they do not have the experience with Carousell mobile application and will be unable to provide the rating for the mobile application.
From the 56 responses, we calculated the average rating based on the individual rating as shown below.
Both the average rating for functionality and user experience between web and mobile application are similar with the mobile application slightly edging out the web application in both aspects.
For functionality rating, mobile application got an average score of 3.9, which is higher than web application by a minor difference of 0.1. As for user experience rating, mobile application got an average rating of 3.9 and the difference is 0.2.
The result is encouraging as we are almost on par with the mobile application even though our web is not fully developed and we are only half-way through our project.
Overall User Experience Rating (Power vs Moderate User)
We then did a further analysis by comparing the user experience average rating between the power user and moderate user.
From the results, the user experience rating for moderate users are similar with only a difference of 0.1.
As for the user experience rating of power users, the difference is 0.3, which is slightly higher than the moderate users. After analyzing the feedback from the users, we found out that power users felt that the web application design needs to be further polished as some of them felt that unnecessary scrolling could be minimized in the landing page and white space could be better utilized.
Overall Functionality Rating (Power vs Moderate User)
We also compared the results for mobile and web functionality rating between the power user and moderate user.
From the analysis, we found out that power user rated web application slightly better than the mobile application. However, moderate users felt that web functionality loses out to mobile slightly. This might be because some mobile functionalities are not currently available on the web application as our project is still in progress.
Average Functionality Rating based on User Experience
Users were then asked to rate the 12 tested functionalities based on User Experience on a scale from 1(Worst) to 5 (Best). The average values for the functionality rating based on user experience ranges from 3.5 to 4.1.
This chart shows the average functionality rating based on user experience. As highlighted in the chart, search for users had the lowest average rating as it was still not intuitive after the change from UT1 as many users were still unable to locate this functionality. However, from the importance rating and the general user feedback, we gathered that this feature was not heavily used. Thus, this was not a very critical issue.
Comparison of individual functionality User Experience rating (Power & Moderate User)
We then compared the individual functionality rating based on the 3 user categories, Power User, Moderate User and new user.
From our analysis based on the comparison, we found out that power users like the user experience for Search Products, Listings by Users, Product Details and Offer Management (Search) as they rated these functionalities highly compared to the moderate and new users.
After analyzing the feedback provided by power users, we found out that search products and offers search were effective for them in looking for products and offers.
As for the user profile and product details page, it could be due to the changes made to the pages to make it somewhat equivalent to their shop front. They liked how the products are organized in the user profile page as it has the category filter which allows users to filter product categories listed by that user. This is different from the mobile as all the listings from different categories were interleaved
Comparison of individual functionality User Experience rating (Buyer & Seller)
We also did a further analysis for the individual functionality user experience between the buyers and sellers
The comparison showed that browse product listings, filters and search product being is more popular among the buyers. After looking through the users comments, we gathered that they like these features mainly because of the bigger screen. Buyers also liked buy now chat more due to the convenience of chatting from the product page as all the important details about the product is on the same page. Therefore, this provided buyers with better user experience as there is no need to go back and forth between pages/tabs
Average Functionality Rating based on Importance
Users were also asked to rate the functionalities based on Importance on the same scale from 1(Worst) to 5 (Best).
Although the other functionalities were generally well rated, Offer Management functionalities were not impressive. This is because users are not familiar with the concept of Offer Management as it is a new feature that is distinct from the mobile application. Furthermore, users were not familiar with the product in the test account for this UT as the test account was pre-populated with offers. Therefore, it was harder for the users to appreciate the filtering and sorting options available for Offers Management.
Comparison of individual functionality Importance rating (Power & Moderate User)
A further analysis on the individual functionalities was done to compared the difference in rating between the power and moderate users.
From on the chart above, power users consistently rated offer management functionalities highly compared to the other user groups. This showed that they felt offer management will help them to effectively manage their offers on Carousell. The lower average rating for offer management in the overall importance average rating shown in the previous section was due to the other user groups. As seen from the graph, new users rated offers management functionalities the lowest across the 3 groups.
Comparison of individual functionality Importance rating (Buyer & Seller)
Similarly, we compared the difference in ratings between the buyer and seller to discover any trends in the ratings. We discovered that product filter was more highly rated for buyers than sellers. They felt that product filters will help them to find and browse product listings more effectively.
Qualitative Feedback & Problems
- Category pictures on landing page are too big and this resulted in too much unnecessary scrolling
- Search toggle for product and user search is not intuitive as many users have trouble finding Search for User function
- Some icons / elements are not meaningful and users do not know what it represents. For example, the “+”, “-“, “o” icons on Product Details and User Profile page
- Users were unsure if login is required to be able to add comments on Product Details page
- The text displayed when no listings are found (No listings found. Here are some recently liked products instead!) is not obvious
- The button text “Buy Now” on Product Details page is inaccurate as the user has to negotiate the product price with the seller before purchasing the product
- Unsure of how to use Offers Management (Filters)
- Unable to like products and follow user
- Able to edit product details
- More specific product categories such as Accessories in For Her category
- Reduce size of category pictures on landing page
- Add in message (information) of icons/elements on hover
- Improve on Search for Users
- Able to like product on Browse page
- Zoom-in feature for main product image on Product Details page
Follow-up actions (Changes to be made)
- Include text (product / user) or change background color for Search toggle to make user search more obvious
- Reduce size of category pictures on landing page to minimize unnecessary scrolling
- Change product filters to make it more explicit and obvious
- Change product filters text input to slider
- Add message that indicate login is required for adding comments
- Increase the spacing between the text “No listings found. Here are some recently liked products instead!”, and recommended product image cards
- Change button text “Buy Now” to “Chat to Buy” on Product Details page
- Login prompt to inform user that login is required to use certain features
Changes Before & After UT2
|Changed Search toggle div background to make the toggle more obvious|
|Reduced category image size on landing page to minimize unnecessary scrolling|
|Changed Product Filters buttons to make it more explicit and obvious to the user|
|Changed Product Filters text input to slider|
|Changed the naming convention of Buy Now (Chat) to be similar to mobile app|
|Added message to indicate login is required to leave a comment|
|Added Login Prompt to inform unauthenticated users that login is required to use certain features (Like Product, Offers, Chat to Buy)|