HeaderSIS.jpg

IS480 Team wiki: 2011T2 Imateam User Testing

From IS480
Jump to navigation Jump to search
Steph.png






The Project Journal
"Lack of documentation is becoming a problem for acceptance."





Imahome.jpg Home   Imastake.jpg Team / Stakeholders   Imaover.jpgProject Overview   Project Documentation   Imamanagement.jpgProject Management   Imaresources.jpgResource & Reference   Imaloms.jpgLOMS & Reflections
  Imaaarrow2.png Diagrams   Imaaarrow2.png Prototypes   Imaaarrow2.png Minutes of Meeting   Imaaarrow2.png User Testing Materials   Imaaarrow2.png Presentation Materials  

User Testing (Midterm)

User Testing (Midterm) Methodology

Participants

The Participants
   

Participants

A pilot test will be conducted internally before the actual test begins.

We will select a total for 30 participants for the actual user test:
• Administrative Functions - 5 participants
• Speaker Functions - 5 participants
• Organizer Functions - 5 participants
• Attendees Functions - 15 participants

The participants' responsibilities will be to attempt to complete a set of representative task scenarios presented to them in as efficient and timely a manner as possible, and to provide feedback regarding the usability and acceptability of the user interface. The participants will be directed to provide honest opinions regarding the usability of the application, and to participate in post-session subjective questionnaires and debriefing.

The best candidates to be engaged are:
• Users who has experience with computers
• Users who have knowledge of using social networking websites
• Users who have attended business conferences or large-scale talks
• Users who have some prior usability experience
• Users with the necessary time to commit to the test
• Users who are willing to experiment (to try various methods to see what works and what doesn’t work)
• Patient and have a tolerance for ambiguity

Procedure

The Procedure
   

Participants will be divided down into 3 categories. These categories are namely IMAPAC staff, SMU professors and fellow students. IMAPAC staff will take on the role of Admin users. Our user tests will be held at Seminar Room 2-2 in Singapore Management University over 2 days. A computer with the web application will be loaded in the testing environment. The facilitator(s) seated in the same room will monitor the participant’s interaction with the web application. Note takers and data logger will monitor the sessions in a corner of the room to record quantitative and qualitative data. The test sessions will be photographed.

Participants' Arrival at Testing Location
Each participant will arrive at the testing location, at the time they have scheduled for their session. A facilitator will greet the participant and the facilitator will brief the participants on the conference web application and inform the participant that they are evaluating the application, rather than the facilitator evaluating the participant. Participants will sign an informed consent that acknowledges: the participation is voluntary, that participation can cease at any time, and that the session will be photographed but their privacy of identification will be safeguarded. The facilitator will ask the participant if they have any questions.

Orientation
Once participants have completed all of the necessary forms, a usability team member will provide them with a brief description of the project and the purposes of this user test. The participant will be reminded that this user test is centered on the Connet web application and not their particular ability to use technology. At this point, the participant will be given instructions to “think aloud” during the evaluation as well as given a reminder that they will be videotaped and monitored by a camera during their evaluation session.

Performance Evaluation
The evaluation will involve having individual participants complete a series of assigned tasks using the . The participant activity during this phase will be recorded by photos supervised by at least 2 testers. The evaluation will proceed accordingly:

  • Participants will be asked to sit down and open the browser on computer provided.
  • Participants will then be instructed on their assigned tasks.
  • Participants will then proceed through the usability evaluation by completing the tasks without guidance from the user testers. The participant will be encouraged to think aloud during this period and only ask for additional help if they become absolutely confused with the application.


Participant Debriefing
After the evaluation, when the participant has completed all the tasks, a debriefing statement will be read to the participant by one of the testers. The debrief will include the following sections:

  • Questionnaire – Participants will be asked to complete a brief questionnaires
  • Verbal Response – Participants will be asked to tell the tester about their experience using the web application. They will be asked to include any specific likes or dislikes within the web application.

The debrief session will be taken very seriously as it allows participants to convey their exact feelings about the web application. After the debrief session, the testers will thank the participants for their involvement.

Materials/Documents

Materials/Documents
   

The following materials have been developed for record and evaluation purposes.

Consent Form
Clickdl-brown.png View Consent Form

User Manual
Clickdl-brown.png View Admin User Manual

Questionnaires
Clickdl-brown.png View Pre-Test Questionnaire
Clickdl-brown.png View Post-Test Questionnaire

Test Cases
Clickdl-brown.png View Admin Test Cases
Clickdl-brown.png View Attendee Test Cases
Clickdl-brown.png View Speaker Test Cases
Clickdl-brown.png View Organizer Test Cases

User Testing (Midterm) Analysis

Data Collected

Results
   

Data Collected

Data we collected during the user testing sessions included:

  • User demographics
  • User observations recorded throughout the test e.g. noting user comments or difficulties
  • Qualitative Post-Test questionnaire:
    • Feedback on the usage of the overall application
    • Opinions on what can be improved or changed
  • Quantitative performance measures:
    • No. of passed and failed test cases for each user
    • No. of times a specific test case was failed across all users
    • Usability ratings of each individual function

Results Analysis

Results were analyzed then categorized according to the user roles because each role had different test cases to perform.

We derived at the following information from the data that we collected:

  • Average passed/failed test cases percentage (for Admin, Organizer, Speaker, Attendee roles)
  • Completion rate of each task (most often failed test case and the no. of times it failed)
  • Function Usability Ratings

Clickdl-brown.png Click to download our results analysis

Post-Midterm UT Changes

Changes Made

Changes Made
   

Design

  • Changed colour and look of conferences' tabs
  • Put more visual and familiar icons for links
  • Indicated which page users are on currently
  • New layout for Conference Page
  • Reduced number of tabs
  • Remove “Conference” from the tab names
  • Created clear navigation for users to create notes
  • Added new tab for Sponsors
  • Changed the link to Feedback form


Functions

  • Created business card or default card upon user creation/registration
  • Removed “Preset Name” and “Description”
  • Changed the name of “Preview Settings” tab
  • Included name, job title, company and email for business card
  • Improved link to change business card (Get help from Stephenie for visual links)
  • Put link under the picture or put link under the “Edit” tab – users keep going to “edit” tab to change business card
  • Changed layout of Business Card Creation
  • Changed “relationship” and “friend” terminologies to “Exchange Business Card” and “Networks”
  • Moved resources into Conference Library
  • Placed Conference Library link either in sidebar or main body
  • Separated upload file button – do not put inside the Conference Group Creation page (Get help from Stephenie for visual links)
  • Included additional fields in import list (6 fields in total): First Name, Last Name, Job Title, Company, Email, Password
  • Imported list can send custom email to individual user. Admin can customize the email
  • Included picture, etc for each conference on My Conferences page
  • Allowed organizer to edit the questions
  • Put speaker feedback and conference feedback under the same form
  • Included divider between each poll
  • Disabled poll voting for speakers
  • Fixed Bug – speakers cannot see poll if they cannot vote
  • Fixed Bug – going into one tab has additional tabs
  • Fixed Bug – users do not have permissions for files being uploaded

User Testing(Final)

User Testing (Final) Methodology

Participants

The Participants
   

Participants

A pilot test was conducted internally before the actual test began.

We selected a total of 30 participants for the actual user test:
• Administrative Functions - 10 participants
• Speaker Functions - 5 participants
• Organizer Functions - 5 participants
• Attendees Functions - 10 participants

The participants' responsibilities was to attempt to complete a set of representative task scenarios presented to them in as efficient and timely a manner as possible. Their test results were recorded by the facilitators who were seated next to them.

The best candidates to be engaged are:
• Users who has experience with computers
• Users who have knowledge of using social networking websites
• Users who have attended business conferences or large-scale talks
• Users who have some prior usability experience
• Users with the necessary time to commit to the test
• Users who are willing to experiment (to try various methods to see what works and what doesn’t work)
• Patient and have a tolerance for ambiguity

Procedure

The Procedure
   

Participants were divided down into 3 categories. These categories were namely IMAPAC staff, SMU professors and fellow students. Our user tests were held at Singapore Management University over 3 days. A computer with the web application was loaded in the testing environment. The facilitator(s) seated in the same room monitored the participant’s interaction with the web application. Note takers and data logger will monitor the sessions in a corner of the room to record quantitative and qualitative data.

Participants' Arrival at Testing Location
Each participant will arrive at the testing location, at the time they have scheduled for their session. A facilitator will greet the participant and the facilitator will brief the participants on the conference web application and inform the participant that they are evaluating the application, rather than the facilitator evaluating the participant. Participants will sign an informed consent that acknowledges: the participation is voluntary, that participation can cease at any time, and that the session will be photographed but their privacy of identification will be safeguarded. The facilitator will ask the participant if they have any questions.

Orientation
Once participants have completed all of the necessary forms, a usability team member will provide them with a brief description of the project and the purposes of this user test. The participant will be reminded that this user test is centered on the Connet web application and not their particular ability to use technology. At this point, the participant will be given instructions to “think aloud” during the evaluation.

Performance Evaluation
The evaluation will involve having individual participants complete a series of assigned tasks on the Connet application. The participant activity during this phase will be recorded by photos supervised by at least 2 testers. The evaluation will proceed accordingly:

  • Participants will be given the admin or user manual depending on his/her role.
  • Participants will be asked to sit down and open the browser on computer provided.
  • Participants will then be instructed on their assigned tasks.
  • Participants will then proceed through the usability evaluation by completing the tasks without guidance from the user testers. The participant will be encouraged to think aloud during this period and only ask for additional help if they become absolutely confused with the application.

Materials/Documents

Materials/Documents
   

The following materials have been developed for record and evaluation purposes.

Consent Form
Clickdl-brown.png View Consent Form

User Manuals
Clickdl-brown.png View Admin Manual Clickdl-brown.png View User Manual

Test Cases
Clickdl-brown.png View Admin Test Cases Clickdl-brown.png View Attendee Test Cases
Clickdl-brown.png View Speaker Test Cases Clickdl-brown.png View Organizer Test Cases

User Testing (Final) Analysis

Data Collected

Results
   

Data Collected

After receiving feedback from Prof. Richard Davis at the midterm presentation that quantitative data is more important, we decided to focus more on quantitative feedback for our Final UAT. However, we decided not to forgo all kinds of qualitative data as we felt that there is some value to qualitative data.

Data we collected during the user testing sessions included:

  • Qualitative performance measures:
    • User feedback and comments
  • Quantitative performance measures:
    • No. of passed and failed test cases for each user
    • No. of times a specific test case was failed across all users
    • No. of wrong moves made for each task before successful completion

Results Analysis

Results were analyzed then categorized according to the user roles because each role had different test cases to perform.

We derived at the following information from the quantitative data that we collected:

  • Average passed/failed test cases percentage (for Admin, Organizer, Speaker, Attendee roles)
  • Completion rate of each task (most often failed test case and the no. of times it failed)
  • Averge number of wrong moves made per task (demonstrates usability)
  • Average time taken for each task

Clickdl-brown.png Click to download our results analysis

Post-Final UT Changes

Changes Made

Changes Made
   

User Testing (Attendee)
Clickdl-brown.png View list of changes

User Testing (Admin/Organizer/Speaker)
Clickdl-brown.png View list of changes

User Testing (with sponsors)
Clickdl-brown.png View list of changes