HeaderSIS.jpg

Difference between revisions of "IS480 Team wiki: 2015T2 6Sigma final wiki"

From IS480
Jump to navigation Jump to search
 
(87 intermediate revisions by 2 users not shown)
Line 46: Line 46:
 
<!--------------------------- Content Starts here ------------------>
 
<!--------------------------- Content Starts here ------------------>
 
==Documents==
 
==Documents==
 
+
[https://www.dropbox.com/s/u3qc8xlriblp5mo/6SIGMA%20-%20IS480%20Final%20Presentation.pdf?dl=0 Final Presentation Slides] <br>
==In A Nutshell==
+
[https://www.dropbox.com/s/he4xf97hjvm7wc1/6Sigma%20-%20List%20of%20APIs%20for%20IS480%20System%202.pdf?dl=0 IS480 Management System II List of APIs]
  
 
==Project Management==
 
==Project Management==
 
===Functions achieved===
 
===Functions achieved===
 +
<div align="center">Functionalities highlighted in red are not achieved.</div>
 +
[[File:6Sigma Priority Circle.jpg | 800px | center]]
 +
 
<br>
 
<br>
 
{| class="wikitable" style="margin: auto;width:80%; text-align:center"
 
{| class="wikitable" style="margin: auto;width:80%; text-align:center"
Line 155: Line 158:
 
|-
 
|-
 
| Team Module
 
| Team Module
| CRUD Proposal/Mid- Term/Final Review
+
| CRUD Acceptance/Mid- Term/Final Review
 
| Allows course coordinators, supervisors, reviewers and sponsors to add their proposal/mid-term/final reviews
 
| Allows course coordinators, supervisors, reviewers and sponsors to add their proposal/mid-term/final reviews
 
| 1
 
| 1
Line 220: Line 223:
 
|}
 
|}
  
 
+
<br>
 +
<br>
 +
[[File:6Sigma Functions Achieved.png | 800px | center]]
  
  
 
<br>
 
<br>
  
===Planned Schedule===
+
===Planned Schedule during Mid Terms===
  
 
[[File:6Sigma Midterm Schedule.png | center | 800px ]]
 
[[File:6Sigma Midterm Schedule.png | center | 800px ]]
Line 232: Line 237:
 
[[File:6Sigma Timeline.png | center | 800px ]]
 
[[File:6Sigma Timeline.png | center | 800px ]]
  
===Issues Faced And Reasons for Change in Schedule===
+
===Reasons for Change in Schedule & Functionalities===
<br>
 
<div style="text-align: center;"><font size="4" color=green> ''' Iteration 5 : 8 December 2014 to 23 December 2014'''</font></div>
 
{| class="wikitable" style="margin: auto;width:90%; text-align:center"
 
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width 400px" | Issues
 
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width: 400px" | Solution
 
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width: 300px" | Mitigation towards schedule
 
|-
 
| Previously, we had resource files (File containing end points) which separated the different roles from their business functions (Eg: Faculty's business functions on teams were classified into FacultyTeamResource). <b>this did not conform the the REST API standards</b>, and resulted in repeated codes.
 
This resulted in many different resource files, and it was difficult to trace code and fix bugs, resulting in delay. This has also frustrated a lot of teammates. <br>
 
<b>Our team foresaw that maintainability was going to be a problem in the long run. </b>
 
|  It was decided for all business functions to be grouped together under a same resource file, with different roles executing different tasks through the use of If-Else blocks. Eg: Functionalities involving teams would be classifed under TeamResource.
 
| rowspan="2" | <b> This resulted in a delay of 2 days.</b> As we previously had an increase in buffer day from iteration 3, <b>this resulted in a deduction of 2 days from our buffer.</b>
 
|-
 
| Our team was still new to the technology of JPA, and we developed our business logics in the resource files. <br> This resulted in repeated codes.
 
| After researching on the best practices online, we discovered that business logics were best to be encapsulated in the service files, in order to achieve a Service Oriented Architecture. This resulted in refactoring of business logics into the service files.
 
|}
 
 
 
<br>
 
<br>
 
<div style="text-align: center;"><font size="4" color=green> ''' Iteration 7 : 9 Jan 2015 to 24 Jan 2015'''</font></div>
 
{| class="wikitable" style="margin: auto;width:90%; text-align:center"
 
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width 400px" | Issues
 
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width: 400px" | Solution
 
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width: 300px" | Mitigation towards schedule
 
|-
 
| Although we have finished the main functionalities required according to our sponsor’s requirements, our sponsors suggested many usability improvements which we tried to cater to. As a result, we found ourselves working harder and harder. This is a significant signal of a creep of scope. Moreover, our supervisor Prof. Chris Boesch reminded us that the ultimate aim of the system should be to benefit all stakeholders. <br><br> After much discussion with sponsors during the sponsor test on 16th Jan 2015, Fiona has suggested to the team that it would be advisable for the team to remove functions which are of less priority and without much value add such that more can be achieved by focusing on the core functionalities / functionalities with the most value add.
 
|  In order to find out whether this would be much of a value add to our main target users (Students), our team has actually conducted a survey and we’ve summarized the results in the surveys page [[IS480 Team wiki: 2015T2 6Sigma User Surveys | HERE]]
 
| 6Sigma has decided to remove CRUD Comments, iterations from our project scope,
 
|}
 
 
 
 
 
<br>
 
<br>
 
<div style="text-align: center;"><font size="4" color=green> ''' Iteration 8 : 24 Jan 2015 to 7 Feb 2015'''</font></div>
 
{| class="wikitable" style="margin: auto;width:90%; text-align:center"
 
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width 400px" | Issues
 
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width: 400px" | Solution
 
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width: 300px" | Mitigation towards schedule
 
|-
 
| After a series of user tests (Student and Faculty) conducted on 28 - 30th Jan, our team realized that usability was a huge issue to all of our stakeholders. Most of our users found that our fonts and colors were difficult to read, functionalities were difficult to find as they were not intuitively displayed.
 
|  While our team understand that it would not be feasible to develop a UI which suits the needs of all members in the IS480 Community, we decided to research more about the best practices of UI Designing. Hence, we discovered Angular Material, a library built for AngularJS, which implements Google's material design principles. Angular Material provides a set of reusable, well-tested, and accessible UI components based on the Material Design system. We decided to adopt this because the elements in Material Design is based on Google's empirical research on the best design principles.  <br><br> <b>Google Material Design</b> is a comprehensive guide for visual, motion, and interaction design across platforms and devices.To find out more about Material Design, click [http://www.google.com/design/spec/material-design/introduction.html here]
 
| 6Sigma has decided to replace the previously scheduled Integrated Grid View for faculty. Also, our team has finished the "send email notifications to sponsors upon creation" function. However, the rest of the notifications functions and view notifications would be pushed to iteration 11. If need be, our team would use our Iteration 12 buffer to handle this functionality.
 
|}
 
 
 
===User Tests===
 
<div style="text-align: center;"><font size="4" color=green> ''' User Tests'''</font></div>
 
{| class="wikitable" style="margin: auto;width:50%; text-align:center"
 
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width 100px" | Event
 
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width: 100px" | Previous Date
 
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width: 100px" | Re-scheduled to
 
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width: 100px" | Re-scheduled date
 
|-
 
| Sponsor Test 1
 
| 2 Jan 2015
 
| Faculty Test
 
| 30 Jan 2015
 
|-
 
| Student Test 1
 
| 12 Jan 2015
 
| Student Test
 
| 30 Jan 2015
 
|-
 
| Student Test 2
 
| 2 Feb 2015
 
| rowspan="2" | User Test 2
 
| rowspan="2" | 12 Feb 2015
 
|-
 
| Faculty Test 2
 
| 9th Feb 2015
 
|}
 
  
==Changes in Scope==
 
 
===Addition===
 
 
{| class="wikitable" style="margin: auto;width:80%; text-align:center"
 
{| class="wikitable" style="margin: auto;width:80%; text-align:center"
 
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width 400px" | Function
 
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width 400px" | Function
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width: 100px" | Scheduled into:
 
 
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width: 300px" | Mitigation towards schedule
 
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width: 300px" | Mitigation towards schedule
 
|-
 
|-
| <strike><b>Integrated Grid View for grading </b>. <br><br>After gathering feedback from Prof. Jason Woodard, he suggested that an integrated grid view for grading would be useful for the faculty members. </strike> <br><br> <b> UPDATE: </b> The grading view is pushed back after mid term and was replaced by Google (Angular) Material Design due to usability concerns which was one of the reason why the previous project failed. Angular Material Design provides a set of reusable, well-tested, and accessible UI components based on the Material Design system, which is a researched and well tested visual language for users that synthesizes the classic principles of good design. This is done so as to benefit all stakeholders through a more usable application based on best UI design practices.
+
| <b>Manipulate Statistics</b> <br> After our mid term presentation, our sponsors suggested that more emphasis should be placed on the grading function as the faculty members would be using the grading functionality for grading in the final presentations. <br><br> Due to our tight schedule, we felt that more time should be allocated to the enhancement and fixing of the grading module such that it is well tested and usable enough for faculty members to adopt the IS480 Management System II for grading.  
 
+
| rowspan="3" | During our 11, 12 and 13th iteration, extensive user tests would be conducted on the faculty members and the grading module would be improved as well.
 
+
|-
| Iteration 8
+
| <b>Notifications (Changed to mailing list)</b> <br> After our mid term presentation, our sponsors suggested that more emphasis should be placed on the grading function as the faculty members would be using the grading functionality for grading in the final presentations. <br><br> Due to our tight schedule, we felt that more time should be allocated to the enhancement and fixing of the grading module such that it is well tested and usable enough for faculty members to adopt the IS480 Management System II for grading.
| Send notifications to users and View Notifications would be scheduled to iteration 11. If need be, we would use the 2 weeks buffer (Iteration 12), which was previously allocated for bug fixing before full time rollout
 
|}
 
 
 
===Removal===
 
{| class="wikitable" style="margin: auto;width:80%; text-align:center"
 
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width 400px" | Function
 
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width: 300px" | Mitigation towards schedule
 
 
|-
 
|-
| <b>Sponsor Comments</b> <br><br> We conducted a [[IS480 Team wiki: 2015T2 6Sigma User Surveys | survey]] to find out whether sponsor comments would be beneficial to the student community. The results of the survey suggests that majority of the students preferred communicating with the sponsors on a face to face basis, and they were also worried about the privacy of their comments. Hence, we decided not to proceed with the development of this function. <br><br> [[IS480 Team wiki: 2015T2 6Sigma User Surveys | More information about the survey]]
+
| <b>Audit (Changed to logging) </b> <br><br> It was technically unfeasible due to the requiring of multiple databases. Moreover, our database structure was already decided before this function was requested by the sponsor. Hence, we've scaled down the Audit functionality to logging. On the other hand, we scaled this functionality down to cater more time for extensive testing and enhancement of the grading functionality. The logging function would only log important tasks performed by users for accountability purposes.  
| The time allocated for developing this function would be allocated into Integrated Grid View and Analytics module
 
|-
 
| <b>Iterations</b> <br><br> We conducted a [[IS480 Team wiki: 2015T2 6Sigma User Surveys | survey]] to find out whether students would prefer to upload their iteration information onto the system such that analysis results could be provided to them for better project management. The results of the survey suggests that majority of the students did not welcome this as they felt that the increase in workload outweighs the added advantage it brings.<br><br> [[IS480 Team wiki: 2015T2 6Sigma User Surveys | More information about the survey]]
 
| The time allocated for developing this function would be allocated into Integrated Grid View and Analytics module.
 
 
|}
 
|}
 
==Project Metrics==
 
  
 
===Schedule Metrics===
 
===Schedule Metrics===
<div style="text-align: center;"><font size="4"> We've adopted the following schedule metrics guidelines to help us track the progress of our project.</font></div>
+
<div style="text-align: center;"><font size="4"> After our mid term presentation, our reviewer, Prof Alan, advised us that we should be tracking planned vs actual no. of hours if we were using time boxing. </font></div>
 
 
[[File:6Sigma Schedule Metrics.png|center| 800px]]
 
<br>
 
[[File:6 Sigma Schedule Metrics timeline.png | center | 900px ]]
 
  
 +
[[File:6Sigma Total hours.png | center|800px]]
 +
<br>
 
{| class="wikitable" style="margin: auto;width:70%; text-align:center"
 
{| class="wikitable" style="margin: auto;width:70%; text-align:center"
 
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width:50px" | Iteration
 
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width:50px" | Iteration
Line 400: Line 316:
 
| 100%
 
| 100%
 
| None. We are on schedule. However, it is important to note that there were still many bugs in this iteration, which is due to the Angular Material UI change
 
| None. We are on schedule. However, it is important to note that there were still many bugs in this iteration, which is due to the Angular Material UI change
 +
|-
 +
| 10
 +
| 14
 +
| 14
 +
| 100%
 +
| None. We are on schedule.
 +
|-
 +
| 11
 +
| 14
 +
| 14
 +
| 100%
 +
| None. We are on schedule.
 +
|-
 +
| 12
 +
| 14
 +
| 14
 +
| 100%
 +
| None. We are on schedule.
 +
|-
 +
| 13
 +
| 14
 +
| 14
 +
| 100%
 +
| None. We are on schedule.
 +
|-
 +
| 14
 +
| 9
 +
| 9
 +
| 100%
 +
| None. We are on schedule.
 +
|-
 
|}
 
|}
  
 
===Bug Metrics===
 
===Bug Metrics===
  
 +
[[File:6Sigma Bugmetric.png | center |900px]]
 +
<br>
 
[[File:6Sigma Bug Metrics.png | center | 900px]]
 
[[File:6Sigma Bug Metrics.png | center | 900px]]
  
{| class="wikitable" style="margin: auto;width:80%; text-align:center"
+
{| class="wikitable" style="margin: auto;width:70%; text-align:center"
 
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width:50px" | Iteration
 
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width:50px" | Iteration
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width:50px" | Score
+
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width:50px" | Low
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width:300px" | Notable reasons
+
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width:50px" | High
 +
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width:50px" | Critical
 +
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width:100px" | Total Score
 
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width:300px" | Actions Taken
 
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width:300px" | Actions Taken
 
|-
 
|-
 
| 1
 
| 1
 +
| 1
 +
| 5
 +
| 0
 
| 26
 
| 26
| -
 
 
|Stop current development and resolve the bug immediately. However, as we completed the functionalities early, we managed to solve the bugs before the end of iteration. Hence, project was not rescheduled
 
|Stop current development and resolve the bug immediately. However, as we completed the functionalities early, we managed to solve the bugs before the end of iteration. Hence, project was not rescheduled
 
|-
 
|-
 
| 2
 
| 2
 +
| 1
 +
| 1
 +
| 0
 
| 6
 
| 6
| -
 
 
| Use scheduled debugging time for debugging
 
| Use scheduled debugging time for debugging
 
|-
 
|-
 
| 3
 
| 3
 +
| 2
 +
| 4
 +
| 0
 
| 30
 
| 30
| -
 
 
| Stop current development and resolve the bug immediately. However, as we completed the functionalities early, we managed to solve the bugs before the end of iteration. Hence, project was not rescheduled
 
| Stop current development and resolve the bug immediately. However, as we completed the functionalities early, we managed to solve the bugs before the end of iteration. Hence, project was not rescheduled
 
|-
 
|-
 
| 4
 
| 4
 +
| 0
 +
| 7
 +
| 0
 
| 35
 
| 35
| -
 
 
| Stop current development and resolve the bug immediately.We managed to solve the bugs before the end of iteration. Hence, project was not rescheduled
 
| Stop current development and resolve the bug immediately.We managed to solve the bugs before the end of iteration. Hence, project was not rescheduled
 
|-
 
|-
 
| 5
 
| 5
 +
| 0
 +
| 17
 +
| 0
 
| 95
 
| 95
| Previously, we had resource files (File containing end points) which groups all business functions together, regardless of the role of the executor. Our team foresaw that maintainability was going to be a problem in the long run with the adoption of this structure. <br><br>A major overhaul of codes was done in this iteration, and hence, the high bug count.
 
 
| Stop current development and resolve the bug immediately. We are unable to complete bug fixing on time, and hence, we decided to use up 2 days of our buffer. Currently, we are left with 5 days of buffer after this iteration.
 
| Stop current development and resolve the bug immediately. We are unable to complete bug fixing on time, and hence, we decided to use up 2 days of our buffer. Currently, we are left with 5 days of buffer after this iteration.
 
|-
 
|-
 
| 6
 
| 6
 +
| 0
 +
| 6
 +
| 0
 
| 30
 
| 30
| -
 
 
| Stop current development and resolve the bug immediately. We managed to solve the bugs before the end of iteration. Hence, project was not rescheduled.
 
| Stop current development and resolve the bug immediately. We managed to solve the bugs before the end of iteration. Hence, project was not rescheduled.
 
|-
 
|-
 
| 7
 
| 7
 +
| 2
 +
| 8
 +
| 0
 
| 42
 
| 42
| It was evident that the basic core functionalities of our application was still not up to standard, and the basic functionalities require more work to be added on. Apart from that, the user test was to be conducted 2 weeks’ time, on Jan 30th.
 
 
| Stop current development and resolve the bug immediately. As too many funactionalties was breaking at this point in time, we decided to reschedule "Manipulate Statistics" to later iterations.  Click [[IS480 Team wiki: 2015T2 6Sigma Issues Faced| HERE]] to find out more
 
| Stop current development and resolve the bug immediately. As too many funactionalties was breaking at this point in time, we decided to reschedule "Manipulate Statistics" to later iterations.  Click [[IS480 Team wiki: 2015T2 6Sigma Issues Faced| HERE]] to find out more
 
|-
 
|-
 
| 8
 
| 8
 +
| 0
 +
| 9
 +
| 3
 
| 75
 
| 75
| We decided to revamp our UI to adopt the best practices and principles of design through the use of Angular Material. As such, there were many UI bugs which were discovered with the re-designing of our UI.
+
| Stop current development and resolve the bug immediately. Click [[IS480 Team wiki: 2015T2 6Sigma Issues Faced| HERE]] to find out more about the change management plan to Angular Material
| Stop current development and resolve the bug immediately. Click [[IS480 Team wiki: 2015T2 6Sigma Issues Faced| HERE]] to find out more about the change management plan for Angular Material
 
 
|-
 
|-
 
| 9
 
| 9
 +
| 0
 +
| 16
 +
| 0
 
| 80
 
| 80
| We decided to revamp our UI to adopt the best practices and principles of design through the use of Angular Material. As such, there were many UI bugs which were discovered with the re-designing of our UI.
+
| Stop current development and resolve the bug immediately. Click [[IS480 Team wiki: 2015T2 6Sigma Issues Faced| HERE]] to find out more about the change management plan to Angular Material
| Stop current development and resolve the bug immediately. Click [[IS480 Team wiki: 2015T2 6Sigma Issues Faced| HERE]] to find out more about the change management plan for Angular Material
+
|-
 +
| 10
 +
| 4
 +
| 1
 +
| 1
 +
| 39
 +
| The critical bug observed here was due to the SSO Service. We realized that not all smu user groups were captured in our system and hence, one faculty member could not log in. We seek technical help from IITS to retrieve all smu user groups in order to prevent this from happening again.
 +
|-
 +
| 11
 +
| 12
 +
| 3
 +
| 0
 +
| 27
 +
| Use the scheduled debugging time for debugging.
 +
|-
 +
| 12
 +
| 9
 +
| 1
 +
| 0
 +
| 14
 +
| Use the scheduled debugging time for debugging.
 +
|-
 +
| 13
 +
| 4
 +
| 1
 +
| 0
 +
| 9
 +
| Use the scheduled debugging time for debugging.
 +
|-
 +
| 14
 +
| 0
 +
| 1
 +
| 0
 +
| 5
 +
| Use the scheduled debugging time for debugging
 +
|-
 
|}
 
|}
  
== Team's Metrics ==
+
==User Tests & Feedback==
  
===Coding Workload===
+
<div style="text-align: center;"><font size="4" color=green> ''' User Tests'''</font></div>
[[File:6Sigma Coding workload.png | center | 1000px]]
+
{| class="wikitable" style="margin: auto;width:80%; text-align:center"
 +
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width 300px" | Event
 +
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width: 100px" | Date
 +
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width: 200px" | No of Particpants
 +
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width: 200px" | Results
 +
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width: 200px" | Raw Data
 +
|-
 +
| Student Test
 +
| 28th Jan 2015
 +
| 32 Students
 +
| [https://wiki.smu.edu.sg/is480/IS480_Team_wiki%3A_2015T2_6Sigma_User_Surveys#Student_Test Click HERE for detailed results]
 +
| [https://www.dropbox.com/s/gl180ncfd4lfc4m/Student%20Test%201%20Raw%20Data.xlsx?dl=0 Click HERE for raw data]
 +
|-
 +
| rowspan="2" | Faculty Test 1 <br> (Session 1 and Session 2)
 +
| Session 1 <br> (30th Jan 2015)
 +
| Prof Jason <br> Prof Benjamin Gan <br> Instructor Fiona Lee
 +
| [https://wiki.smu.edu.sg/is480/IS480_Team_wiki%3A_2015T2_6Sigma_User_Surveys#Faculty_Test  Click HERE for detailed results]
 +
| [https://www.dropbox.com/s/rny14poi3ucavrr/Raw%20Data.xlsx?dl=0 Click Here for raw Data]
 +
|-
 +
| Session 2 <br> (12th-13th Feb 2015)
 +
| Prof Benjamin Gan <br> Instructor Fiona Lee
 +
| [https://wiki.smu.edu.sg/is480/IS480_Team_wiki%3A_2015T2_6Sigma_User_Surveys#Session_2_-_Benjamin_Gan_.26_Fiona_Lee  Click HERE for detailed results]
 +
| [https://www.dropbox.com/s/rny14poi3ucavrr/Raw%20Data.xlsx?dl=0 Click Here for raw Data]
 +
|-
 +
| Student Test 2
 +
| 14th Feb 2015
 +
| 20 Students
 +
| [https://wiki.smu.edu.sg/is480/IS480_Team_wiki%3A_2015T2_6Sigma_User_Surveys#Student_Test_2  Click HERE for detailed results]
 +
| [https://www.dropbox.com/s/a2fhnig7qk1ogsd/Student%20Test%202%20Raw%20Data.xlsx?dl=0 Click HERE for raw Data]
 +
|-
 +
| Faculty Test 2
 +
| 18th - 25th March 2015
 +
| Prof Jason <br> Prof Benjamin Gan <br> Prof Debin Gao <br> Prof Chris Boesch <br> Prof Li Yingjiu <br> Prof Cheok Lai Tee <br> Prof Patrick Thng <br> Prof David Lo <br> Prof Alan Megargel
 +
| [https://wiki.smu.edu.sg/is480/IS480_Team_wiki%3A_2015T2_6Sigma_User_Surveys#Faculty_Test_3  Click HERE for detailed results]
 +
| [https://www.dropbox.com/s/rbrn01bu74qckwg/Faculty%20Test%203%20Raw%20Data.xlsx?dl=0 Click HERE for Raw Data ]
 +
|}
  
===Coding Hours===
+
<br>
[[File:6Sigma Coding Hours.png | center | 1000px]]
 
  
===Non Coding Hours===
+
<div style="text-align: center;"><font size="4" color=green> ''' User Feedback'''</font></div>
[[File:6Sigma Non Coding Hours.png | center | 1000px]]
+
{| class="wikitable" style="margin: auto;width:80%; text-align:center"
 +
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width 300px" | Event
 +
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width: 100px" | Date
 +
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width: 200px" | No of Particpants
 +
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width: 200px" | Results
 +
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width: 200px" | Raw Data
 +
|-
 +
| Student Survey
 +
| 23rd Jan 2015
 +
| 61 Students
 +
| [https://wiki.smu.edu.sg/is480/IS480_Team_wiki%3A_2015T2_6Sigma_User_Surveys#Survey_1 Click HERE for detailed survey results]
 +
| [https://www.dropbox.com/s/5366jec4i3hokjs/Student%20Survey.xlsx?dl=0 Click HERE for Raw Data]
 +
|-
 +
| 6Sigma IS480 User Feedback Session With GongCha Giveaway
 +
| 23rd March - 25th March 2015
 +
| 150 unique students
 +
| [https://wiki.smu.edu.sg/is480/IS480_Team_wiki%3A_2015T2_6Sigma_User_Surveys#User_Feedback_Session_With_GongCha_Giveaway Click HERE for detailed survey results]
 +
| [https://docs.google.com/spreadsheets/d/1DzPBhwDGGSsfZp1sYeckvNbHoOqQrs615CYQroZ4idw/edit#gid=438504620 Click HERE for Raw Data]
 +
|-
 +
| Gathering of Faculty's Visioning
 +
| 23rd - 31st March 2015
 +
| Prof Chris Boesch <br> Prof Cheok Lai Tee <br> Prof Patrick Thng <br> Prof Debin Gao
 +
| [https://wiki.smu.edu.sg/is480/IS480_Team_wiki%3A_2015T2_6Sigma_User_Surveys#Gathering_of_Faculty.27s_Vision Click HERE for detailed survey results]
 +
| [https://www.dropbox.com/s/h1for7nxeo26208/Faculty%20Visioning.xlsx?dl=0 Click HERE for Raw Data]
 +
|-
 +
| <b> SUMMARY OF TOTAL Feedback implemented vs gathered from all feedback sessions </b>
 +
| -
 +
| -
 +
| -
 +
| [https://www.dropbox.com/s/ajbm1dnb3b9c0nx/Comments%20implemented%20vs%20not%20implemented.xlsx?dl=0 Click HERE for Raw Data]
 +
|}
  
===Total Hours===
+
==Load Testing==
[[File:6Sigma Total Hours.png | center | 1000px]]
+
===Test Configuration===
 +
{| class="wikitable" style="margin: auto;width:80%; text-align:center"
 +
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width 100px" | Max User Load
 +
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width: 100px" | Duration (Minutes)
 +
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width: 100px" | Threads
 +
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width: 100px" | Rampup
 +
|-
 +
| 50
 +
| 20
 +
| 50
 +
| 300
 +
|-
 +
|}
  
===Punch Card (GIT)===
+
===Faculty & Admin Functions===
[[File:6Sigma Punch card.png | center | 1000px]]
+
====Results====
 +
[[https://wiki.smu.edu.sg/is480/File%3AFaculty_Hazelcast_Report.pdf Hazelcast Report]]
 +
[[https://wiki.smu.edu.sg/is480/File%3AFaculty_Reference_Report.pdf Reference Report]]
 +
{| class="wikitable" style="margin: auto;width:80%; text-align:center"
 +
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width 300px" | Result Type
 +
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width: 100px" | Graph
 +
|-
 +
| User Load
 +
| [[File:Usersf.png|600px]]
 +
|-
 +
| Response Time
 +
| [[File:Responsef.png|600px]]
 +
|-
 +
| Latency
 +
| [[File:Latencyf.png|600px]]
 +
|-
 +
| Bandwidth in KB/s
 +
| [[File:Bandwidthf.png|600px]]
 +
|-
 +
| Hits/s
 +
| [[File:Hitsf.png|600px]]
 +
|-
 +
| Errors
 +
| [[File:Errorsf.png|600px]]
 +
|}
 +
 
 +
===Student Functions===
 +
====Results====
 +
[[https://wiki.smu.edu.sg/is480/File%3AStudent_Hazelcast_Report.pdf Hazelcast Report]]
 +
[[https://wiki.smu.edu.sg/is480/File%3AStudent_Reference_Report.pdf Reference Report]]
 +
 
 +
{| class="wikitable" style="margin: auto;width:80%; text-align:center"
 +
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width 300px" | Result Type
 +
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width: 100px" | Graph
 +
|-
 +
| User Load
 +
| [[File:Users.png|600px]]
 +
|-
 +
| Response Time
 +
| [[File:Response.png|600px]]
 +
|-
 +
| Latency
 +
| [[File:Latency.png|600px]]
 +
|-
 +
| Bandwidth in KB/s
 +
| [[File:Bandwidth.png|600px]]
 +
|-
 +
| Hits/s
 +
| [[File:Hits.png|600px]]
 +
|-
 +
| Errors
 +
| [[File:Errors.png|600px]]
 +
|}
 +
 
 +
===Conclusion===
 +
<div style="text-align: center;"><font size="4">
 +
For faculty and admin test cases, average throughput increased from 1 hit/s to 12 hits/s. The average page response time decreased from 76495ms to 2739ms, a 28 fold improvement.
 +
At the 90th percentile, page response time dropped from 110285ms to 6569ms, a 16 fold improvement.
 +
 
 +
 
 +
For student test cases, average throughput increased from 1 hit/s to 8 hits/s. The average page response time decreased from 68700ms to 4133ms, a 16 fold improvement.
 +
At the 90th percentile, page response time dropped from 103455ms to 9355ms, a 11 fold improvement.
 +
 
 +
 
 +
As the results were stable (0 errors), the testing environment controlled and the test run on the similar deployment environment, the overall improvements provided by Hazelcast significantly improved the performance of the application.
 +
 
 +
</font></div>
  
 
==Risks==
 
==Risks==
  
===Risks===
+
 
 
[[File:6Sigma Risk Table.png |center| 800px]]
 
[[File:6Sigma Risk Table.png |center| 800px]]
  
Line 490: Line 637:
 
<div style="text-align: center;"><font size="4" color="red"> '''Top 3 risks identified''' </font></div>
 
<div style="text-align: center;"><font size="4" color="red"> '''Top 3 risks identified''' </font></div>
  
{| class="wikitable" style="margin: auto;width:90%; text-align:center"
+
{| class="wikitable" style="margin: auto;width:95%; text-align:center"
 
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width:150px" | Risk Type
 
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width:150px" | Risk Type
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width:200px" | Risk Event
+
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width:300px" | Risk Event
 
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width:70px" | Occurrence
 
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width:70px" | Occurrence
 
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width:70px" | Impact
 
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width:70px" | Impact
 
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width:100px" | Category
 
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width:100px" | Category
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width:400px" | Mitigation
+
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width:300px" | Preventive measures
 +
! style="font-weight: bold;background: #16a085;color:#ecf0f1; width:400px" | Reactive Measures
 
|-
 
|-
|Adoption Risk
+
|Adoption Risk
|Our system might not be well received by the IS480 Community for usage and efforts to capture user's data for analytics might fail
+
|We might run into the risk of not understanding how good it had to be for the client to adopt it. We might run into the cycle of constantly trying to be good enough, hoping its good enough, without actually knowing what is good enough.
|High
+
|High
|High
+
|High
 
|style="color:red;" | '''A'''  
 
|style="color:red;" | '''A'''  
|We intend to roll the system out in week 7, such that the next IS480 Batch can start using our system. Also, our team would be conducting numerous user tests and survey to improve usability.
+
|Our team would conduct early user test and aim an early deployment for faculty members to explore the system.
 +
|Our team gathered feedback from the faculty members personally to meet their requirements of the system. Also, we also addressed the concerns which faculty members might have regarding our application before the final presentation starts. Moreover, we got a sensing of how comfortable the faculty were with using our application.
 
|-
 
|-
|Regulatory Risk
+
|Live deployment risk
|Analytics and statistics used in this project may require the use of personal data and may violate the Personal Data Protection Act (PDPA).
+
|Our applications might face issues in the production environment when it is used for actual grading.
 
|High
 
|High
 
|High
 
|High
 
|style="color:red;" | '''A'''  
 
|style="color:red;" | '''A'''  
|The use of any personal data will be declared to the sponsor and all data will be protected from unauthorized access/use.
+
|Our testing environment would be replicated as closely as possible to the deployment environment.
 +
|During the final presentation period, we would schedule ourselves to be on standby for assistance in case of technical difficulties faced by faculty members.
 
|-
 
|-
|Technology Risk
+
|Integration Risk
|As mediawiki is highly unfavorable to crawling, crawling efforts might fail
+
|During the integration of our application with the SSO service provided by IITS, there might be data compatibility issues.
 
|High
 
|High
 
|High
 
|High
 
|style="color:red;" | '''A'''  
 
|style="color:red;" | '''A'''  
|The development for learning analytics is kept to the end so as to cater time to manually extract data in the event the wiki is uncrawlable.
+
|Use available data on LDAP (Lightweight Directory Access Protocol) in hopes of a full compatibility.
 +
|When there were login issues, we communicated with IITS and conducted an investigation with regards to the smu_groups fields that the SSO application returns to resolve it
 
|}
 
|}
 
 
===Mitigation===
 
 
[[File:6Sigma Risk mitigation.png | center | 900px]]
 
  
 
==Technical Complexity==
 
==Technical Complexity==
 +
[[File:6Sigma Technical Complexity.png | 800px | center]]
  
[[File:6Sigma Technical Complexity.png | center | 800px]]
+
== X Factor==
 +
[[File:X-factors-6sigma.png|centre|800px]]
  
 
==Learning Outcomes==
 
==Learning Outcomes==
 +
===Individual===
 +
[[File:Learing Outcomes 2.png | center | 800px]]
  
Link to our Knowledge Based Contribution :[https://wiki.smu.edu.sg/is480/Knowledge_base#Angular_JS here]
+
===What we have learnt and what we would have done better===
[[File:6Sigma Learning outcomes.png | center | 700px]]
+
[[File:6Sigma Learning outcome2.png | center | 700px]]

Latest revision as of 20:27, 21 April 2015

6Sigma Logo.png

Home

About Us

IS480 Management System

Project Management

Project Documentation

Home Midterm Finals

Documents

Final Presentation Slides
IS480 Management System II List of APIs

Project Management

Functions achieved

Functionalities highlighted in red are not achieved.
6Sigma Priority Circle.jpg


Iteration Module Function Description Confidence Comments
1 Student Module CRUD Student Profile Allows students to CRUD their profiles. 1 -
Team Module CRUD Team Profile Allows a student to CRUD a team profile. 1 -
CRUD Team member Allows a student to CRUD team members. 1 -
General Module SSO Allows all users with SMU domain credentials to login to the system. 1 Some faculty members were still unable to SSO into our application due to the different user groups assigned to each faculty. We quickly resolved this by liaising with IITS to obtain all SIS faculty user groups
2 Faculty Module CRUD Faculty Allows faculty to CRUD their profiles. 1 -
Sponsor Module CRUD Accounts Allows new sponsors to request an account and propose a project. 1 -
Login Allows external sponsors to login as they do not have SSO accounts. 1 -
3 Project Module CRUD Projects Allows sponsors to CRUD projects for students to take 1 -
4 Project Module CRUD Project Teams Allows sponsor to approve or reject teams for their projects. 1 -
Team Module CRUD Project Status Allows the course coordinator/supervisor to update the status of the team’s project. (i.e. accepted or rejected) 1 -
Coordinator Module CRUD Term Allows the Course Coordinator to CRUD terms in the system. 1 -
5 Faculty Module Assign Supervisor/Reviewer to Team Allows Course Coordinator to a specific supervisor and assign him to a specific team manually. 1 -
Coordinator Module Suspend Team Suspend the team users so that they will not be a nuisance to the other users. Team will be unsuspended once the Course Coordinator deemed its ok. There is a user status attached to all the users (active/ suspended) which is only viewable by the admin. 1 -
Coordinator Module Assign Sponsor Role Course coordinator can assign sponsor roles to faculty. 1 -
6 Project Module Request Project Approval Allows students to request the sponsor to accept the team for the project. 1 -
Team Module CRUD Acceptance/Mid- Term/Final Review Allows course coordinators, supervisors, reviewers and sponsors to add their proposal/mid-term/final reviews 1 -
Project Module CRUD Project Details/Documents Allows team to use the website as a replacement for wiki for their project. 1 -
7 General Module Download statistics User must be able to download IS480 statistics. 1 -
Search Module Search Student / Faculty / Project / Sponsor / Team Search for Student / Faculty / Project / Sponsor / Team based on what user entered into the search bar so that he can retrieve the information 1 -
8 General Module Generation of Email to Sponsors Generate an email to sponsor upon signing up for the system 1 -
9 Analytics Module Cosine Similarity Index Use of algorithm to find out the similar teams in IS480 based on Technologies Used, Industries and Project Type. 1 -
10 General Module View Statistics Displaying of statistics through the use of NVD3 Charts 1 -
Logging Logging of important information for accountability purposes 1 -
11 Scheduling Module Generating of CSV File format to suit scheduling system. Generating of CSV File format to suit scheduling system for integration purposes. 1 -
General Module Send notifications to users Generates a list of email via the system when there are important announcements to make. 1 -



6Sigma Functions Achieved.png



Planned Schedule during Mid Terms

6Sigma Midterm Schedule.png

Actual Schedule

6Sigma Timeline.png

Reasons for Change in Schedule & Functionalities

Function Mitigation towards schedule
Manipulate Statistics
After our mid term presentation, our sponsors suggested that more emphasis should be placed on the grading function as the faculty members would be using the grading functionality for grading in the final presentations.

Due to our tight schedule, we felt that more time should be allocated to the enhancement and fixing of the grading module such that it is well tested and usable enough for faculty members to adopt the IS480 Management System II for grading.
During our 11, 12 and 13th iteration, extensive user tests would be conducted on the faculty members and the grading module would be improved as well.
Notifications (Changed to mailing list)
After our mid term presentation, our sponsors suggested that more emphasis should be placed on the grading function as the faculty members would be using the grading functionality for grading in the final presentations.

Due to our tight schedule, we felt that more time should be allocated to the enhancement and fixing of the grading module such that it is well tested and usable enough for faculty members to adopt the IS480 Management System II for grading.
Audit (Changed to logging)

It was technically unfeasible due to the requiring of multiple databases. Moreover, our database structure was already decided before this function was requested by the sponsor. Hence, we've scaled down the Audit functionality to logging. On the other hand, we scaled this functionality down to cater more time for extensive testing and enhancement of the grading functionality. The logging function would only log important tasks performed by users for accountability purposes.

Schedule Metrics

After our mid term presentation, our reviewer, Prof Alan, advised us that we should be tracking planned vs actual no. of hours if we were using time boxing.
6Sigma Total hours.png


Iteration Planned Days Actual Days Score Actions Taken
1 14 14 100% None. We are on schedule
2 14 14 100% None. We are on schedule
3 14 13 107% Add one day to buffer
4 14 14 100% None. We are on schedule
5 14 16 87.5% As we previously had an increase in buffer day from iteration 3, this resulted in a deduction of 1 day from our buffer. For more information on the problems and solutions for this iteration, refer to the "Issues Faced" tab.
6 14 14 100% None. We are on schedule
7 14 14 100% Decided to push Manipulate Statistics to further iterations. For more information on the problems and solutions for this iteration, refer to the "Issues Faced" tab.
8 14 14 100% We completed the sending of email notification to sponsors. However, other notifications would be pushed to further iterations such that we can focus on Angular Material. For more information on the problems and solutions for this iteration, refer to the "Issues Faced" tab.
9 14 14 100% None. We are on schedule. However, it is important to note that there were still many bugs in this iteration, which is due to the Angular Material UI change
10 14 14 100% None. We are on schedule.
11 14 14 100% None. We are on schedule.
12 14 14 100% None. We are on schedule.
13 14 14 100% None. We are on schedule.
14 9 9 100% None. We are on schedule.

Bug Metrics

6Sigma Bugmetric.png


6Sigma Bug Metrics.png
Iteration Low High Critical Total Score Actions Taken
1 1 5 0 26 Stop current development and resolve the bug immediately. However, as we completed the functionalities early, we managed to solve the bugs before the end of iteration. Hence, project was not rescheduled
2 1 1 0 6 Use scheduled debugging time for debugging
3 2 4 0 30 Stop current development and resolve the bug immediately. However, as we completed the functionalities early, we managed to solve the bugs before the end of iteration. Hence, project was not rescheduled
4 0 7 0 35 Stop current development and resolve the bug immediately.We managed to solve the bugs before the end of iteration. Hence, project was not rescheduled
5 0 17 0 95 Stop current development and resolve the bug immediately. We are unable to complete bug fixing on time, and hence, we decided to use up 2 days of our buffer. Currently, we are left with 5 days of buffer after this iteration.
6 0 6 0 30 Stop current development and resolve the bug immediately. We managed to solve the bugs before the end of iteration. Hence, project was not rescheduled.
7 2 8 0 42 Stop current development and resolve the bug immediately. As too many funactionalties was breaking at this point in time, we decided to reschedule "Manipulate Statistics" to later iterations. Click HERE to find out more
8 0 9 3 75 Stop current development and resolve the bug immediately. Click HERE to find out more about the change management plan to Angular Material
9 0 16 0 80 Stop current development and resolve the bug immediately. Click HERE to find out more about the change management plan to Angular Material
10 4 1 1 39 The critical bug observed here was due to the SSO Service. We realized that not all smu user groups were captured in our system and hence, one faculty member could not log in. We seek technical help from IITS to retrieve all smu user groups in order to prevent this from happening again.
11 12 3 0 27 Use the scheduled debugging time for debugging.
12 9 1 0 14 Use the scheduled debugging time for debugging.
13 4 1 0 9 Use the scheduled debugging time for debugging.
14 0 1 0 5 Use the scheduled debugging time for debugging

User Tests & Feedback

User Tests
Event Date No of Particpants Results Raw Data
Student Test 28th Jan 2015 32 Students Click HERE for detailed results Click HERE for raw data
Faculty Test 1
(Session 1 and Session 2)
Session 1
(30th Jan 2015)
Prof Jason
Prof Benjamin Gan
Instructor Fiona Lee
Click HERE for detailed results Click Here for raw Data
Session 2
(12th-13th Feb 2015)
Prof Benjamin Gan
Instructor Fiona Lee
Click HERE for detailed results Click Here for raw Data
Student Test 2 14th Feb 2015 20 Students Click HERE for detailed results Click HERE for raw Data
Faculty Test 2 18th - 25th March 2015 Prof Jason
Prof Benjamin Gan
Prof Debin Gao
Prof Chris Boesch
Prof Li Yingjiu
Prof Cheok Lai Tee
Prof Patrick Thng
Prof David Lo
Prof Alan Megargel
Click HERE for detailed results Click HERE for Raw Data


User Feedback
Event Date No of Particpants Results Raw Data
Student Survey 23rd Jan 2015 61 Students Click HERE for detailed survey results Click HERE for Raw Data
6Sigma IS480 User Feedback Session With GongCha Giveaway 23rd March - 25th March 2015 150 unique students Click HERE for detailed survey results Click HERE for Raw Data
Gathering of Faculty's Visioning 23rd - 31st March 2015 Prof Chris Boesch
Prof Cheok Lai Tee
Prof Patrick Thng
Prof Debin Gao
Click HERE for detailed survey results Click HERE for Raw Data
SUMMARY OF TOTAL Feedback implemented vs gathered from all feedback sessions - - - Click HERE for Raw Data

Load Testing

Test Configuration

Max User Load Duration (Minutes) Threads Rampup
50 20 50 300

Faculty & Admin Functions

Results

[Hazelcast Report] [Reference Report]

Result Type Graph
User Load Usersf.png
Response Time Responsef.png
Latency Latencyf.png
Bandwidth in KB/s Bandwidthf.png
Hits/s Hitsf.png
Errors Errorsf.png

Student Functions

Results

[Hazelcast Report] [Reference Report]

Result Type Graph
User Load Users.png
Response Time Response.png
Latency Latency.png
Bandwidth in KB/s Bandwidth.png
Hits/s Hits.png
Errors Errors.png

Conclusion

For faculty and admin test cases, average throughput increased from 1 hit/s to 12 hits/s. The average page response time decreased from 76495ms to 2739ms, a 28 fold improvement. At the 90th percentile, page response time dropped from 110285ms to 6569ms, a 16 fold improvement.


For student test cases, average throughput increased from 1 hit/s to 8 hits/s. The average page response time decreased from 68700ms to 4133ms, a 16 fold improvement. At the 90th percentile, page response time dropped from 103455ms to 9355ms, a 11 fold improvement.


As the results were stable (0 errors), the testing environment controlled and the test run on the similar deployment environment, the overall improvements provided by Hazelcast significantly improved the performance of the application.

Risks

6Sigma Risk Table.png


We've classified the risks we've identified into 3 categories: A, B and C.

'A' risks need the most attention and most well developed mitigation or recovery strategies,

'C' risks can occur but deserve the least amount of planning.


Top 3 risks identified
Risk Type Risk Event Occurrence Impact Category Preventive measures Reactive Measures
Adoption Risk We might run into the risk of not understanding how good it had to be for the client to adopt it. We might run into the cycle of constantly trying to be good enough, hoping its good enough, without actually knowing what is good enough. High High A Our team would conduct early user test and aim an early deployment for faculty members to explore the system. Our team gathered feedback from the faculty members personally to meet their requirements of the system. Also, we also addressed the concerns which faculty members might have regarding our application before the final presentation starts. Moreover, we got a sensing of how comfortable the faculty were with using our application.
Live deployment risk Our applications might face issues in the production environment when it is used for actual grading. High High A Our testing environment would be replicated as closely as possible to the deployment environment. During the final presentation period, we would schedule ourselves to be on standby for assistance in case of technical difficulties faced by faculty members.
Integration Risk During the integration of our application with the SSO service provided by IITS, there might be data compatibility issues. High High A Use available data on LDAP (Lightweight Directory Access Protocol) in hopes of a full compatibility. When there were login issues, we communicated with IITS and conducted an investigation with regards to the smu_groups fields that the SSO application returns to resolve it

Technical Complexity

6Sigma Technical Complexity.png

X Factor

X-factors-6sigma.png

Learning Outcomes

Individual

Learing Outcomes 2.png

What we have learnt and what we would have done better

6Sigma Learning outcome2.png