Difference between revisions of "IS480 Team wiki: 2011T2 Imateam User Testing"
(21 intermediate revisions by 2 users not shown) | |||
Line 42: | Line 42: | ||
<font face="century gothic" size="2"> | <font face="century gothic" size="2"> | ||
+ | =User Testing (Midterm)= | ||
=='''User Testing (Midterm) Methodology'''== | =='''User Testing (Midterm) Methodology'''== | ||
===<div style="color: #ffffff; font-size:0%">Participants</div>=== | ===<div style="color: #ffffff; font-size:0%">Participants</div>=== | ||
Line 173: | Line 174: | ||
<div style = "text-align:right;"><font ="color:#000000; size:"12px;"><u>[[IS480_Team_wiki:_2011T2_Imateam_User_Testing#Top|Back to Top]]</u></font></div> | <div style = "text-align:right;"><font ="color:#000000; size:"12px;"><u>[[IS480_Team_wiki:_2011T2_Imateam_User_Testing#Top|Back to Top]]</u></font></div> | ||
+ | |||
=='''User Testing (Midterm) Analysis'''== | =='''User Testing (Midterm) Analysis'''== | ||
===<div style="color: #ffffff; font-size:0%">Data Collected</div>=== | ===<div style="color: #ffffff; font-size:0%">Data Collected</div>=== | ||
Line 237: | Line 239: | ||
*Added new tab for Sponsors<br> | *Added new tab for Sponsors<br> | ||
*Changed the link to Feedback form | *Changed the link to Feedback form | ||
− | + | <br> | |
'''Functions'''<br><br> | '''Functions'''<br><br> | ||
*Created business card or default card upon user creation/registration<br> | *Created business card or default card upon user creation/registration<br> | ||
Line 267: | Line 269: | ||
<div style = "text-align:right;"><font ="color:#000000; size:"12px;"><u>[[IS480_Team_wiki:_2011T2_Imateam_User_Testing#Top|Back to Top]]</u></font></div> | <div style = "text-align:right;"><font ="color:#000000; size:"12px;"><u>[[IS480_Team_wiki:_2011T2_Imateam_User_Testing#Top|Back to Top]]</u></font></div> | ||
− | + | =User Testing(Final)= | |
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
=='''User Testing (Final) Methodology'''== | =='''User Testing (Final) Methodology'''== | ||
===<div style="color: #ffffff; font-size:0%">Participants</div>=== | ===<div style="color: #ffffff; font-size:0%">Participants</div>=== | ||
Line 294: | Line 286: | ||
'''Participants'''<br><br> | '''Participants'''<br><br> | ||
− | A pilot test | + | A pilot test was conducted internally before the actual test began. |
− | We | + | We selected a total of 30 participants for the actual user test:<br> |
− | • Administrative Functions - | + | • Administrative Functions - 10 participants<br> |
• Speaker Functions - 5 participants<br> | • Speaker Functions - 5 participants<br> | ||
• Organizer Functions - 5 participants<br> | • Organizer Functions - 5 participants<br> | ||
− | • Attendees Functions - | + | • Attendees Functions - 10 participants<br><br> |
− | The participants' responsibilities | + | The participants' responsibilities was to attempt to complete a set of representative task scenarios presented to them in as efficient and timely a manner as possible. Their test results were recorded by the facilitators who were seated next to them.<br><br> |
The best candidates to be engaged are:<br> | The best candidates to be engaged are:<br> | ||
Line 333: | Line 325: | ||
<div style="padding:10px; text-align: justify;"> | <div style="padding:10px; text-align: justify;"> | ||
<font color="#000000"><font size ="2px"> | <font color="#000000"><font size ="2px"> | ||
− | Participants | + | Participants were divided down into 3 categories. These categories were namely IMAPAC staff, SMU professors and fellow students. Our user tests were held at Singapore Management University over 3 days. A computer with the web application was loaded in the testing environment. The facilitator(s) seated in the same room monitored the participant’s interaction with the web application. Note takers and data logger will monitor the sessions in a corner of the room to record quantitative and qualitative data.<br> |
<br> | <br> | ||
'''Participants' Arrival at Testing Location | '''Participants' Arrival at Testing Location | ||
Line 343: | Line 335: | ||
''' | ''' | ||
<Br> | <Br> | ||
− | Once participants have completed all of the necessary forms, a usability team member will provide them with a brief description of the project and the purposes of this user test. The participant will be reminded that this user test is centered on the Connet web application and not their particular ability to use technology. At this point, the participant will be given instructions to “think aloud” during the evaluation | + | Once participants have completed all of the necessary forms, a usability team member will provide them with a brief description of the project and the purposes of this user test. The participant will be reminded that this user test is centered on the Connet web application and not their particular ability to use technology. At this point, the participant will be given instructions to “think aloud” during the evaluation. |
<br><br> | <br><br> | ||
'''Performance Evaluation | '''Performance Evaluation | ||
''' | ''' | ||
<br> | <br> | ||
− | The evaluation will involve having individual participants complete a series of assigned tasks | + | The evaluation will involve having individual participants complete a series of assigned tasks on the Connet application. The participant activity during this phase will be recorded by photos supervised by at least 2 testers. The evaluation will proceed accordingly: |
+ | |||
+ | * Participants will be given the admin or user manual depending on his/her role. | ||
* Participants will be asked to sit down and open the browser on computer provided. | * Participants will be asked to sit down and open the browser on computer provided. | ||
Line 354: | Line 348: | ||
* Participants will then be instructed on their assigned tasks. | * Participants will then be instructed on their assigned tasks. | ||
− | * Participants will then proceed through the usability evaluation by completing the tasks without guidance from the user testers. The participant will be encouraged to think aloud during this period and only ask for additional help if they become absolutely confused with the application | + | * Participants will then proceed through the usability evaluation by completing the tasks without guidance from the user testers. The participant will be encouraged to think aloud during this period and only ask for additional help if they become absolutely confused with the application. |
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
|- | |- | ||
Line 391: | Line 376: | ||
[[Image:Clickdl-brown.png|70px]] [[Media:Informed Consent Form.docx|<span style="color: #000000; padding: 10px 15px 0 15px; font-size: 12px;">View Consent Form</span>]] | [[Image:Clickdl-brown.png|70px]] [[Media:Informed Consent Form.docx|<span style="color: #000000; padding: 10px 15px 0 15px; font-size: 12px;">View Consent Form</span>]] | ||
− | '''User | + | '''User Manuals'''<br> |
− | [[Image:Clickdl-brown.png|70px]] [[Media:Admin | + | [[Image:Clickdl-brown.png|70px]] [[Media:Admin Manual Final edited.doc|<span style="color: #000000; padding: 10px 15px 0 15px; font-size: 12px;">View Admin Manual</span>]] |
− | + | [[Image:Clickdl-brown.png|70px]] [[Media:User Manual Final.doc|<span style="color: #000000; padding: 10px 15px 0 15px; font-size: 12px;">View User Manual</span>]] | |
− | |||
− | [[Image:Clickdl-brown.png|70px]] [[Media: | ||
− | |||
'''Test Cases'''<br> | '''Test Cases'''<br> | ||
− | [[Image:Clickdl-brown.png|70px]] [[Media:Admin Test Cases.docx|<span style="color: #000000; padding: 10px 15px 0 15px; font-size: 12px;">View Admin Test Cases</span>]] | + | [[Image:Clickdl-brown.png|70px]] [[Media:Admin Test Cases Final.docx|<span style="color: #000000; padding: 10px 15px 0 15px; font-size: 12px;">View Admin Test Cases</span>]] |
− | [[Image:Clickdl-brown.png|70px]] [[Media:Attendee Test Cases.docx|<span style="color: #000000; padding: 10px 15px 0 15px; font-size: 12px;">View Attendee Test Cases</span>]]<br> | + | [[Image:Clickdl-brown.png|70px]] [[Media:Attendee Test Cases Final.docx|<span style="color: #000000; padding: 10px 15px 0 15px; font-size: 12px;">View Attendee Test Cases</span>]]<br> |
− | [[Image:Clickdl-brown.png|70px]] [[Media:Speaker Test Cases.docx|<span style="color: #000000; padding: 10px 15px 0 15px; font-size: 12px;">View Speaker Test Cases</span>]] | + | [[Image:Clickdl-brown.png|70px]] [[Media:Speaker Test Cases Final.docx|<span style="color: #000000; padding: 10px 15px 0 15px; font-size: 12px;">View Speaker Test Cases</span>]] |
− | [[Image:Clickdl-brown.png|70px]] [[Media:Organizer Test Cases.docx|<span style="color: #000000; padding: 10px 15px 0 15px; font-size: 12px;">View Organizer Test Cases</span>]] | + | [[Image:Clickdl-brown.png|70px]] [[Media:Organizer Test Cases Final.docx|<span style="color: #000000; padding: 10px 15px 0 15px; font-size: 12px;">View Organizer Test Cases</span>]] |
|- | |- | ||
| colspan="2" class="radius_bottom" style="background:#7e786f; border:2px solid #7e786f" |<div style="font-size:0">[[image:pix.gif|15px]]</div> | | colspan="2" class="radius_bottom" style="background:#7e786f; border:2px solid #7e786f" |<div style="font-size:0">[[image:pix.gif|15px]]</div> | ||
Line 409: | Line 391: | ||
<div style = "text-align:right;"><font ="color:#000000; size:"12px;"><u>[[IS480_Team_wiki:_2011T2_Imateam_User_Testing#Top|Back to Top]]</u></font></div> | <div style = "text-align:right;"><font ="color:#000000; size:"12px;"><u>[[IS480_Team_wiki:_2011T2_Imateam_User_Testing#Top|Back to Top]]</u></font></div> | ||
+ | |||
=='''User Testing (Final) Analysis'''== | =='''User Testing (Final) Analysis'''== | ||
===<div style="color: #ffffff; font-size:0%">Data Collected</div>=== | ===<div style="color: #ffffff; font-size:0%">Data Collected</div>=== | ||
Line 424: | Line 407: | ||
<font color="#000000"><font size ="2px"> | <font color="#000000"><font size ="2px"> | ||
'''Data Collected'''<br><br> | '''Data Collected'''<br><br> | ||
+ | After receiving feedback from Prof. Richard Davis at the midterm presentation that quantitative data is more important, we decided to focus more on quantitative feedback for our Final UAT. However, we decided not to forgo all kinds of qualitative data as we felt that there is some value to qualitative data. | ||
+ | |||
Data we collected during the user testing sessions included:<br> | Data we collected during the user testing sessions included:<br> | ||
− | + | *Qualitative performance measures: | |
− | + | **User feedback and comments | |
− | *Qualitative | ||
− | ** | ||
− | |||
*Quantitative performance measures: | *Quantitative performance measures: | ||
**No. of passed and failed test cases for each user | **No. of passed and failed test cases for each user | ||
**No. of times a specific test case was failed across all users | **No. of times a specific test case was failed across all users | ||
− | ** | + | **No. of wrong moves made for each task before successful completion<br><br> |
'''Results Analysis'''<br><br> | '''Results Analysis'''<br><br> | ||
Results were analyzed then categorized according to the user roles because each role had different test cases to perform. | Results were analyzed then categorized according to the user roles because each role had different test cases to perform. | ||
− | We derived at the following information from the data that we collected: | + | We derived at the following information from the '''quantitative''' data that we collected: |
*Average passed/failed test cases percentage (for Admin, Organizer, Speaker, Attendee roles) | *Average passed/failed test cases percentage (for Admin, Organizer, Speaker, Attendee roles) | ||
*Completion rate of each task (most often failed test case and the no. of times it failed) | *Completion rate of each task (most often failed test case and the no. of times it failed) | ||
− | * | + | *Averge number of wrong moves made per task (demonstrates usability) |
− | [[Image:Clickdl-brown.png]] [[Media: | + | *Average time taken for each task |
+ | [[Image:Clickdl-brown.png]] [[Media:FinalUTResults.pptx|<span style="color: #000000; padding: 10px 15px 0 15px; font-size: 14px;">Click to download our results analysis</span>]] | ||
|- | |- | ||
| colspan="2" class="radius_bottom" style="background:#7e786f; border:2px solid #7e786f" |<div style="font-size:0">[[image:pix.gif|15px]]</div> | | colspan="2" class="radius_bottom" style="background:#7e786f; border:2px solid #7e786f" |<div style="font-size:0">[[image:pix.gif|15px]]</div> | ||
Line 463: | Line 446: | ||
<div style="padding:10px; text-align: justify;"> | <div style="padding:10px; text-align: justify;"> | ||
<font color="#000000"><font size ="2px"> | <font color="#000000"><font size ="2px"> | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
+ | '''User Testing (Attendee)'''<br> | ||
+ | [[Image:Clickdl-brown.png|70px]] [[Media:Attendee changes final.docx|<span style="color: #000000; padding: 10px 15px 0 15px; font-size: 12px;">View list of changes</span>]] | ||
+ | '''User Testing (Admin/Organizer/Speaker)'''<br> | ||
+ | [[Image:Clickdl-brown.png|70px]] [[Media:User changes.docx|<span style="color: #000000; padding: 10px 15px 0 15px; font-size: 12px;">View list of changes</span>]] | ||
+ | '''User Testing (with sponsors)'''<br> | ||
+ | [[Image:Clickdl-brown.png|70px]] [[Media:Client changes.docx|<span style="color: #000000; padding: 10px 15px 0 15px; font-size: 12px;">View list of changes</span>]]<br> | ||
|- | |- | ||
| colspan="2" class="radius_bottom" style="background:#7e786f; border:2px solid #7e786f" |<div style="font-size:0">[[image:pix.gif|15px]]</div> | | colspan="2" class="radius_bottom" style="background:#7e786f; border:2px solid #7e786f" |<div style="font-size:0">[[image:pix.gif|15px]]</div> |
Latest revision as of 12:40, 23 April 2012
"Lack of documentation is becoming a problem for acceptance."
Home | Team / Stakeholders | Project Overview | Project Documentation | Project Management | Resource & Reference | LOMS & Reflections |
Diagrams | Prototypes | Minutes of Meeting | User Testing Materials | Presentation Materials |
User Testing (Midterm)
User Testing (Midterm) Methodology
Participants
|
Participants A pilot test will be conducted internally before the actual test begins. We will select a total for 30 participants for the actual user test: The participants' responsibilities will be to attempt to complete a set of representative task scenarios presented to them in as efficient and timely a manner as possible, and to provide feedback regarding the usability and acceptability of the user interface. The participants will be directed to provide honest opinions regarding the usability of the application, and to participate in post-session subjective questionnaires and debriefing. The best candidates to be engaged are: | |
Procedure
|
Participants will be divided down into 3 categories. These categories are namely IMAPAC staff, SMU professors and fellow students. IMAPAC staff will take on the role of Admin users. Our user tests will be held at Seminar Room 2-2 in Singapore Management University over 2 days. A computer with the web application will be loaded in the testing environment. The facilitator(s) seated in the same room will monitor the participant’s interaction with the web application. Note takers and data logger will monitor the sessions in a corner of the room to record quantitative and qualitative data. The test sessions will be photographed.
The debrief session will be taken very seriously as it allows participants to convey their exact feelings about the web application. After the debrief session, the testers will thank the participants for their involvement. | |
Materials/Documents
|
The following materials have been developed for record and evaluation purposes. Consent Form User Manual Questionnaires Test Cases | |
User Testing (Midterm) Analysis
Data Collected
|
Data Collected
Results Analysis We derived at the following information from the data that we collected:
| |
Post-Midterm UT Changes
Changes Made
|
Design
| |
User Testing(Final)
User Testing (Final) Methodology
Participants
|
Participants A pilot test was conducted internally before the actual test began. We selected a total of 30 participants for the actual user test: The participants' responsibilities was to attempt to complete a set of representative task scenarios presented to them in as efficient and timely a manner as possible. Their test results were recorded by the facilitators who were seated next to them. The best candidates to be engaged are: | |
Procedure
|
Participants were divided down into 3 categories. These categories were namely IMAPAC staff, SMU professors and fellow students. Our user tests were held at Singapore Management University over 3 days. A computer with the web application was loaded in the testing environment. The facilitator(s) seated in the same room monitored the participant’s interaction with the web application. Note takers and data logger will monitor the sessions in a corner of the room to record quantitative and qualitative data.
| |
Materials/Documents
|
The following materials have been developed for record and evaluation purposes. Consent Form User Manuals Test Cases | |
User Testing (Final) Analysis
Data Collected
|
Data Collected Data we collected during the user testing sessions included:
Results Analysis We derived at the following information from the quantitative data that we collected:
| |
Post-Final UT Changes
Changes Made
|
User Testing (Attendee) User Testing (Admin/Organizer/Speaker) User Testing (with sponsors) | |