HeaderSIS.jpg

IS480 Team wiki: 2012T1 M.O.O.T/Final Wiki

From IS480
Jump to navigation Jump to search

UAT.jpg


Project Progress Summary

Download our Final Presentation Slides for your reference during our Final Presentation(0930, Monday 3 December).

Project Highlights

  • Inability to implement unsupervised learning
    • Percentage of men picking male hat and vice versa during UT2 was only 27% - not reliable
  • Cumbersome preparation for User Testing at Plaza Singapura
  • Tricky gesture choices for interactive flow requiring numerous times of gesture changes

There are also other midterm project highlights.

Project Challenges

Balancing Intuitiveness and Technical Competence

Natural User Interface poses a challenge as something intuitive for us is often not easily translated into something that machine can understand. For example, it is natural for us to point at something to indicate that we have chosen the thing, in our case, a door. However, Kinect does not detect finger, hence we need to think of other ways to indicate choice that is not awkward, yet still implementable. We usually come up with a list of possible gestures and intentions, and eliminate them systematically. The same applies for other gestures implemented in AlterSense.

Securing User Testing Site at Plaza Singapura

As CapitaMalls Asia is a large Trans National Company, each employee's job scope is very specific, hence there is a myriad of people we have to liaise with just to secure the testing site. It was difficult to communicate with so many parties, and the assumption that messages would be passed on to relevant people often did not materialize. We received a long list of requirements for "renting" a site a week before the scheduled User Testing (UT). Since none of us had any experience of renting commercial space for roadshow, we struggled in understanding and attempting to meet the requirements, as well as in coming up with contingency plans. Fortunately, through tighter communication with relevant key people and a lot of persistence, we managed to hold the UT as scheduled.

Project Achievements

  • Tested by real shoppers of diverse demographics
  • Possibility to be further developed in phases by CapitaMalls Asia
  • More than 80% gender prediction accuracy in controlled environment
  • More than 70% gender prediction accuracy in uncontrolled environment
  • Developed a readily deploy-able motion sensing gender predicting system from scratch

Project Management

Project Schedule

Our weekly progress table helps in tracking weekly progress, complementing the planned schedule summary that provides us with a bird's eye view of the project.

The two aforementioned documentations also aid us in establishing actual vs planned schedule, which comes in handy in monitoring outstanding tasks and to inspect whether the delays inflict much impact on the iterations that they follow.

Overall, we were able to follow our planned schedule quite closely. Major challenges were encountered in Iteration 1, as described in our midterm project summary. Post midterm, the most challenging period resulting in relatively significant delay was in Iteration 6. Preparation for User Testing at Plaza Singapura took development time away from Russell Goh Zhi Wei,who had to liaise with electrical contractor to pull power out of the insulator and set up cabling setup that fulfills required regulation. As a result, all Advertisement Management System-related features and other tasks assigned to Russell were delayed.

Project Metrics

Schedule & Bug Metrics

We keep track of schedule and bug metrics, and the metrics have been useful external cues in directing the project.

Overall, velocity of 17 is our average, stable velocity - indicating that in an iteration, team can be reliably expected to complete tasks of worth 17 points. The points were assigned based on perceived complexity or time required to complete the tasks. There was a schedule slip in Iteration 6 due to the unexpected amount of extra effort required in securing a site for User Testing in Plaza Singapura. Time was hence channeled from completing development task to settling the requirements for User Testing. Nonetheless, team caught up and recovered to the stable velocity of 17 in the following iteration by using the buffer period (weekend between iterations).

MOOTBugMetric.jpg

Generally, all bugs found were resolved by the end of the iteration in which the bugs were found. An exception was iteration 2, as the focus of the iteration was to come up with a working alpha version of gender recognition. Moreover, the bug was found at the end of the iteration.Iteration 3 displayed a high total bug score as numerous medium severity bugs were found in this iteration, requiring us to push "saving learning state" functionality to Iteration 6 to accommodate more time for bug squashing. The team, however, eventually decided to drop the mentioned functionality as effort required seemed to surpass the possible benefit; taking into account feedback received during Midterm Presentation as well. A critical bug was encountered in Iteration 5, requiring immediate action and halting the development - resulting in a delay of the development of two functionality. Our highest bug score was observed in Iteration 6, resulting in the delay of the development of a number of functionality; for some members needed to spend time debugging before proceeding with scheduled features.

Gender Recognition Metrics

Our gender recognition metrics can be accessed here.

We used leave-one-out cross-validation to find the optimum training set size that yields the highest gender prediction accuracy. Several rounds of testing shows that the minimum amount of training sets needed should be at least 60. More data is required to achieve an accuracy above 90%.

A round of testing of the integration of the hair and skirt parameters as biased inputs into the Neural Network has shown that the hair and skirt parameters were affecting the accuracy of gender prediction due to their relatively low accuracy of determining the presence of hair and skirt respectively. As a biased input has a much higher weightage than a non-biased parameter in influencing the final outcome of gender prediction, the final accuracy of gender prediction has thus dropped significantly.
Since gender prediction works fine with just the three non-biased parameters (height, shoulder & hip width) as shown during our UT1 - with the accuracy reaching 87%, we decided not to introduce any biased parameter to influence the final outcome of gender prediction.

Technical Complexity

Front End Development

The front end development of AlterSense involves developing a scene manager that ensures every scene can be developed without manipulating the properties of various graphical user elements in the previous or upcoming scenes. We had to implement Boolean variables and methods that could manipulate the properties of a graphical element. For instance, the Windows Presentation Foundation layer framework offers a storyboard class that enables animations such as the fade effect of the Speech Bubble in AlterSense. We also had to develop other methods that would control the visibility, position, size and other properties to aid front-end development.

Front-end development requires a lot of debugging and testing due to the nature of timing, sequence and aesthetics. Considerations such as whether the object appeared at the right moment, is large enough for users to see, yet it should not obstruct other elements in the process is a balancing act altogether. There has to be careful sequencing when it comes to invoking various methods, and ultimately, the overall result should not promote a bad user experience.

Preventing Object from Going Out of Frame

With regards to user experience, we encountered an issue that graphical objects would go out of the sight if the user moves to the edges of the Kinect Camera view. We identified this to be a bad user design, and created methods to ensure that AlterSense would know when a graphical object is out of frame, and re-adjust the parameters accordingly so that it will always remain on the screen.

To do this, we have to dynamically identify the height and width of the object as well as retrieve the boundaries of the canvas. Next, we identified the 4 corners of an object and monitor the positions of these 4 points in real-time. When the shopper moves to the edge of the camera view, one or more of the object’s positions will collide with the edges of the canvas object, and AlterSense would stop adjusting the positions of the object, and set the positions such that it would be viewable. When the shopper moves back into an acceptable boundary, AlterSense would ensure that the graphical object would continue to move within the acceptable boundaries. The overall effect of this implementation ensures that instructions and images will always be viewable within the screen, regardless of how the shopper positions himself in front of the Kinect.

Linking Systems

Content Access

The AlterSense solution comprises of several systems that are interconnected. Deciding which method to link these systems required numerous design considerations. Initially the access of tags via the AMS (in providing the content required for targeted advertising in AlterSense) was done online, but problems were encountered as it was slow and unresponsive at times.

Hence, we had to switch to a local storage of the tag images so that the system does not need to send a web request every time a user wants to download or view tag. This implementation allows faster access (view and download) of tags without having to send a web request to the Microsoft Tag server. Furthermore, this new arrangement allows user to access the tags even if the Microsoft Tag SOAP server is down.

Enabling Dynamic Connections

The Microsoft Tag SOAP web service is an external service provided by Microsoft, where AMS serves as an interface to manage all ads and data associated with each tag. The AMS has to be designed to enable dynamic change of end-point connected to the account associated. The current token used by AMS is under an academic account. Associated token had to be configured to be easily changeable to facilitate switching of token after CMA purchases commercial tag creation licence. The configuration required a lot of point of reference settings.

Quality of Product

Project Deliverable

Stage Specification Modules
Project Management Metrics Schedule & Bug Metrics, Bug Log
Minutes Client Meeting,Supervisor Meeting
Content Interaction flow chart Flow Chart
Narrative flow Interactive Content Flow
Requirement Functionality list Functionalities
Design Use Case Use Case
Architectural diagram Architectural Diagram
Testing User Testing 2 documentation UT2, UT2 Supporting Documents
Handover Manual Deployment Manual

Quality

  • Scalability:
    • AlterSense is a stand alone application that is designed to be deployed to many shopping malls in Singapore
    • Storing of content in Advertisement Management System data, refer to dynamic content in AMS
  • Flexibility:
    • Gender-targeted content is retrieved from AMS, which offers an easy interface to manage (create, update, delete) content online
    • Scenes and gestures in AlterSense storyboard are decoupled, allowing amendment of scene sequence or interaction to be implemented without inflicting drastic changes to other scenes, as mentioned in Front End Development technical complexity
  • Performance:

Using hexadecimal value instead of the color function (in the Drawing Library) to color pixel for background and outline improved performance as it is a more direct method, resolving the issue of lags we encountered in the initial stage

  • Configurability:
    • The use of Neural Network facilitates machine learning - client can attempt to further improve the accuracy of gender recognition by acquiring more training data, although our data analysis has shown that more than 60 training data is able to produce more than 70% accuracy
    • MySQL credentials and Microsoft Tag token is stored in separate files to enable easy configuration
    • A switch has been created for Supervised Learning scene - client can choose to configure (activate it or replace the scene with a more reliable one) it accordingly
  • Usability:
    • Numerous front end development features such as dynamic callout bubble, preventing interaction object from going out of frame,as explained in Technical Complexity allows real shoppers to use AlterSense with considerable ease - as observed during User Testing 2
    • Automatic refreshing of scene, error controls, and instructions allow shoppers to interact with AlterSense even without guidance
  • Maintainability:

Code is organized into controller classes based on functions, allowing straight forward editing and updating of code

  • Fault-tolerance:

Decoupling certain functions of AMS from the Microsoft Tag web service improves fault-tolerance. Even if the Microsoft Tag SOAP service is down, downloading of tag will still be available, just that creation and editing of new tags will be affected

Deployment

We handed over a readily deploy-able system to CapitaMalls Asia (CMA). The deliverable delivered for User Acceptance are:

  • Code executable
  • Pre-requisites software and framework
  • Deployment manual
  • Pitch video to show an overview of scenes and story board
  • Project documentation - content from Wiki in pdf format, as requested by client

Hardware used for development are CMA properties and will be returned accordingly, and can hence be used for actual deployment as well.

Testing

Description Details
Date 5 November 2012 from 12pm to 6pm
Objectives
  • Verify if users can react to AlterSense without prompting
  • Test the intuitiveness of our user interface (photo taking) and hand gestures
  • Gather user opinion about using AlterSense in a mall setting
  • Test the accuracy of gender recognition in a mall setting
Testers

53 testers - 29 females & 24 males
Details of testers can be found in Testers' Profile

Execution flow

1. Ask shopper if they have a smartphone and to download Microsoft Tag if they have one. If they are unwilling to download, we lend them our phones
2. Let user use AlterSense by himself/herself
3. Observe user if he/she has any difficulties responding to AlterSense
4. Measure the time taken for user to raise his/her right hand and to scan the Microsoft tag
5. Let user complete a survey form

Results

Findings from observation form
Results of survey form

Reflection

"Team M.O.O.T has shown remarkable tenacity and creativity in coming up with brilliant ideas to help the client
and have performed to the best of their abilities to make things happen."

Ethan Quek
Manager (Information Technology)
CapitaMalls Asia Limited

Team Reflection

We simply cannot believe that we have made it this far. The very initial idea of coming up with an interactive mall directory was turned down by CMA at the blink of an eye, rendering weeks of brainstorming and preparation to pitch the idea in vain. Nonetheless, if Edwin (IT head of CMA) did not urge us to look for something with a WOW factor, something novel, something that would excite shoppers; we would not have come up with AlterSense - and might indeed have ended up with a project that we ourselves might not be proud of.

AlterSense is a dream comes true for the team - we were inspired as we watched a video of wall display displaying intriguing visualizations as a person walked past the wall. However, there were a myriad of challenges in materializing our desired interactive wall display: none of us has any background in visual design, what was the business value, what kind of story board could be crafted out of it; not to mention the challenge of picking up C# and venturing into hardware calibration as well.

The success of the team in completing the project would not have been possible without the support of two people: our supervisor, Jason, and our main point of contact with CMA, Ethan. Jason has assisted the team to great extent in trimming down the scope and in crafting a holistic business solution. Jason encouraged the team to go back to the drawing board often to come up with a more cohesive story board - it might seem tedious, but the results were apparent. Without so doing, AlterSense could have ended up as a pool of disparate technological exploration, instead of a complete solution with interactive narrative flow that it is now. Ethan has been a great support to the team - our invaluable mediator who was willing to stay back after office hour to listen to our ideas and updates, who tried to voice our concerns to the other parties concerned in CMA, who readily assisted us with logistics procurement, and supported us from the project inception to its completion. Definitely, along the course of project, we have received help and encouragement from other sources too. Professors whom we consulted on Neural Network and data mining, Professor Ben who linked us up with CMA after finding out about our mall directory idea, as well as other faculty members and CMA employees who have assisted us in one way and another.

Generally, we are satisfied with how far AlterSense has come to. Nonetheless, if we could do things differently, we would like to try using decision tree in determining gender. Doing so will give us more control and insights into how gender is determined by AlterSense. Although Neural Network (NN) was chosen due to the fuzziness of the classification required as well as its ability to do machine learning, NN is a black box - nobody knows how exactly the inputs are processed to determine gender (output). Although it still works to a level of decent accuracy and is indeed suitable for our client's needs, knowing how exactly it works and having more direct control over the prediction process may allow further expansion of the gender recognition feature.

We have picked up new technical skills from this project, no doubt. Nevertheless, most importantly, we have learnt to be resilient and persistent - in pitching idea, in defining our scope, in materializing something that we would like to see. We perceive it as a life skill - we would certainly face difficulties in our future endeavor, may we then be able to look at Project AlterSense on hindsight and recall that we can achieve our goals if we persevere: learn if we do not know how to, seek if we do not have, ask if we are not given.

Individual Reflection


Clarissa.png

This journey has been a challenging, yet fulfilling experience for me. As our project focuses on shopper's experience, it is essential to always keep the shopper in mind while designing the storyboard. Whenever there was a change in requirements or we needed to add something new into the storyboard, we had to take into consideration many factors such as whether the addition would be too abrupt and destroy the flow of the storyboard or whether the experience would be intuitive enough without being too straight in the face. Ensuring that all the various sections are integrated together with a consistent storyline and common theme while keeping the fun factor was definitely a very challenging task. In addition, from our User Testing 2, I realized that the perspective of being a tester of an application is very different from being the creator of the application. Certain things that we thought would be natural and intuitive to a person did not turn out to be so. Hence there was a need to change our perspective and start thinking of the shopper as one that would requires a lot of hints.

Through this project, I have also picked up valuable soft skills such as working with people with different working styles. Having a diversified team with each team member having different strengths certainly helped us in leveraging on each other's strengths. In addition, learning to understand one another's weaknesses and think of ways to adapt and work around it instead of expecting others to change definitely was one of the learning point.

Personally, I am very grateful to be part of this exciting project with this special bunch of people and am very glad that our team have managed to come this far. It is definitely a memorable experience which would help guide me through my future.


RussellZhiWei.png

Project AlterSense trained me not only to be well-versed in my development skills, but also soft skills such as logistics and people management. It was evident in our last user testing in Plaza Singapura, where we went through several hurdles such as coordinating with several parties to approve the location for testing (people from the Mall, IT and MarCom Department), to even liaising with technician to prepare power points for our test location. These incidents trained me in areas that couldn’t be taught in an academic setting, equipping me with skills that would be essential in real world setting.

When developing AlterSense’s Kinect Gesture, the thought process behind each gesture was vital in making the user experience enjoyable and fun. Using Kinect challenged us to design and create visual cues for users, to understand what kind of gesture is required for the desired input. Something as simple as choosing a door could be difficult to be understood by a machine. We had to brainstorm, what would be the best method for a user to select a door? Can the user complete an action without any instruction given? Designing a natural user interface for user to interact with was the most challenging part for me. Through our user testing and feedback, I was able to gain more experience and insight in this field and able to put myself in the shoppers’ perspective when developing.

Overall my IS480 journey had been fulfilling, working with CapitaMalls Asia allows me to pick up business acumen that would be valuable for my career in the future. Having an awesome team and friends to share this journey with me, IS480 would definitely be the most memorable project in SMU.


Inez.png

I used to think that schedule planning was more of a perfunctory administrative matter. This project, however, shows me that the project schedule is,in fact, the thing that keeps everyone on the same page. It lets us know where we are, what else are left to do. Iterative methodology allows me to revisit the schedule at the end of iteration, and craft the schedule for upcoming iterations based on our progress in the past iteration. Pivotal Tracker has been a handy tool in determining velocity, which I leverage on as a gauge of how much we can accomplish in iteration.

Many times we were questioned on the feasibility of using Kinect in shopping center, or of using Neural Network instead of regression to determine gender; just because the aforementioned things are not common and put us in a state with very steep learning curve. Equipped with the willingness to learn and an open mind, voila! We have actually come so far! This project instills this realization that I can learn new things and apply them to the context that I want to see them in.

Lastly, I witnessed the pivotal role of business-IT alignment in this project. Balancing narrative flow that provides shopper with satisfying user experience and reasonable features development is an art. We may be tempted to include a feature - just because it is technically reasonable, but have to eventually go back to the drawing board to find alternative way because the initial proposed feature simply does not fit the story line.

All in all, Project AlterSense has provided me with a a glimpse of end-to-end, real world software development. The skills obtained definitely make me feel better equipped to take up other challenging projects in the future.


Kenny.png

The IS480 journey has taken my learning beyond the classroom. Meeting a real client and developing a Kinect project was an exciting and rare opportunity to speak with external parties and obtain a realistic perspective to project management. From a project perspective, it is important to be flexible as changes are bound to happen. This project has gone through several rounds of requirement changes over time, and I believe it’s important to keep the development modules independent so new changes can be implemented more readily. Additionally, I have learnt a lot about developing business requirements as my client offered business domain knowledge as well as meeting other stakeholders from various departments within CapitaMall Asia. Meeting these stakeholders offered valuable insights, and understanding their concerns is a very essential element required to develop the business requirements of AlterSense.

In hard skills, I have gained knowledge in Kinect development, Windows Presentation Foundation layer and various methods to process data streams. In addition, I am more familiar with front-end development as controlling the user interface elements and adjusting them to fit the storyboard proved to be a delicate affair that required vast amounts of testing and careful sequencing of methods. On top of getting things to work, a front-end developer has to place more emphasis in aesthetics, sequence and overall layout.

AlterSense is definitely one of most challenging projects that I will remember. Coming to the end of this project, I am personally satisfied with the outcome. Having everything pieced together to form a working solution that fits CMA’s goals is a fulfilling task. I gained a lot from this project, and believe that these experiences would be useful in my future endeavors.


Stephen.png
I feel grateful for the opportunity to experience this IS480 journey with my fellow team members. At a project level, I learnt a lot about managing the expectations of multiple stakeholders. I felt that it was a rewarding experience because it pushed the boundaries of what our team could achieve, given the time constraints and technological complexities. I also learnt how to appreciate the process of exploring into unknown territory, without having any prior knowledge about a subject matter. At a personal level, I learnt a lot about risk management through juggling multiple projects and prioritising to-do items. Moving forward, I hope to pass on this experience to fellow SMU SIS students who will be embarking on their IS480 journey in the future.