HeaderSIS.jpg

Difference between revisions of "IS480 Team wiki: 2017T1 Ravenous Final Wiki"

From IS480
Jump to navigation Jump to search
 
(44 intermediate revisions by the same user not shown)
Line 35: Line 35:
  
 
[[Image: Ravenous - Midterm Project About.png|720px|center]]<br>
 
[[Image: Ravenous - Midterm Project About.png|720px|center]]<br>
As of 9th October 2017, we have completed 85% of iteration 7 and we have 2 more iterations to go.<br><br>
+
As of 14th November 2017, we have completed 41% of iteration 9 and this is our last iteration.<br><br>
To view midterm slides, click [[Media:Ravenous - MidTermSlides.pdf| here]]
+
To view final slides, click [[Media:Ravenous - finalSlides.pdf| here]]
 
<br>
 
<br>
 
To access EvBot & FaBot, login to Workplace@Facebook instance by clicking [https://psd-test.facebook.com/ here]  
 
To access EvBot & FaBot, login to Workplace@Facebook instance by clicking [https://psd-test.facebook.com/ here]  
 
<br>
 
<br>
To access Dashy, click [https://dashy-production.herokuapp.com/ here]
+
To access Dashy (our instance), click [https://dashy-production.herokuapp.com/ here]<br>
 +
To access Dashy <b>(WOG's live instance)</b>, click [http://dashy.ap-southeast-1.elasticbeanstalk.com/ here]
 +
<br><br>
 +
To view <b>handover documentation</b>, click [https://www.dropbox.com/sh/adqxpkpylslowjv/AACCaJAXvl7KtpYZphDgtH7ma?dl=0 here]
 
<br><br>
 
<br><br>
 
For more information on how to access our applications, please view Quality of Product > Deployment Section
 
For more information on how to access our applications, please view Quality of Product > Deployment Section
 
=== Project Highlights & Snapshots ===
 
=== Project Highlights & Snapshots ===
  
[[Image: Ravenous - Midterm Project Highlights.png|720px|center]]
+
[[Image: Ravenous - Final Project Highlights.png|720px|center]]
  
 
== Project Management ==
 
== Project Management ==
Line 51: Line 54:
 
=== Project Status ===
 
=== Project Status ===
 
The three diagrams below show the progress of our project, a high level overview of completion of project modules and a high level overview of change in scope to our project modules respectively.<br>
 
The three diagrams below show the progress of our project, a high level overview of completion of project modules and a high level overview of change in scope to our project modules respectively.<br>
[[File:Ravenous_-_ProjectStatus.png|center|720px]] <br>
+
[[File:Ravenous_-_FinalProjectStatus.png|center|720px]] <br>
[[File:Ravenous_-_FunctionalitiesSummary.png|center|900px]]<br><br>
+
[[File:Ravenous_-_FinalFunctionalitiesSummary.png|center|900px]]<br><br>
[[File:Ravenous_-_FunctionalitiesChange.png|center|900px]]
+
[[File:Ravenous_-_FinalFunctionalitiesChange1.png|center|900px]]
  
 
==== Scope: Core ====
 
==== Scope: Core ====
Line 243: Line 246:
 
| View word cloud chart
 
| View word cloud chart
 
| No
 
| No
| Fully deployed and 100% Tested; On production server
+
| Delivered
 
| 1.0
 
| 1.0
 
| style="color: #008000;" | Completed
 
| style="color: #008000;" | Completed
Line 304: Line 307:
 
| Analytics Dashboard Module IV
 
| Analytics Dashboard Module IV
 
| Export dashboard to CSV File
 
| Export dashboard to CSV File
| Yes
+
| No
| To be completed in iteration 8
+
| Fully deployed and 100% Tested; On production server
 
| 1.0
 
| 1.0
| style="color: #3498DB;" | To be completed in iteration 8
+
| style="color: #008000;" | Completed
 
|-
 
|-
 
| Analytics Dashboard Module IV
 
| Analytics Dashboard Module IV
| List of activated and deactivated accounts of each agency in past 7 days
+
| Filtering options based on user activity/profile
 
| No
 
| No
| In Progress
+
| Fully deployed and 100% Tested; On production server
 
| 1.0
 
| 1.0
| style="color: #FFA500;" | In Progress
+
| style="color: #008000;" | Completed
 
|-
 
|-
 
| Analytics Dashboard Module IV
 
| Analytics Dashboard Module IV
| Filtering options based on user activity/profile
+
| Dashy additional metrics: Overview, posts/comments of previous day vs last week’s same day
 +
| No
 +
| Fully deployed and 100% Tested; On production server
 +
| 1.0
 +
| style="color: #008000;" | Completed
 +
|-
 +
| Analytics Dashboard Module IV
 +
| Dashy additional metrics: Group, Measure posts, comments and reactions in a given period of time and see the most popular days and times that members engage. (advanced filtering)
 
| No
 
| No
| In Progress
+
| Fully deployed and 100% Tested; On production server
 
| 1.0
 
| 1.0
| style="color: #FFA500;" | In Progress
+
| style="color: #008000;" | Completed
 
|-
 
|-
 
| Analytics Dashboard Module IV
 
| Analytics Dashboard Module IV
| Dashy additional metrics: Overview
+
| Dashy additional metrics: Most seen posts by number of seen per group
 
| Yes
 
| Yes
| In Progress
+
| Fully deployed and 100% Tested; On production server
 
| 1.0
 
| 1.0
| style="color: #FFA500;" | In Progress
+
| style="color: #008000;" | Completed
 
|-
 
|-
 
| Analytics Dashboard Module IV
 
| Analytics Dashboard Module IV
| Dashy additional metrics: Group
+
| Dashy additional metrics: Top contributor by agency
 
| Yes
 
| Yes
| To be completed in iteration 8
+
| Fully deployed and 100% Tested; On production server
 
| 1.0
 
| 1.0
| style="color: #3498DB;" | To be completed in iteration 8
+
| style="color: #008000;" | Completed
 
|-
 
|-
 
| FMS Module II
 
| FMS Module II
 
| Release a booking that has begun
 
| Release a booking that has begun
| Yes
+
| No
| In Progress
+
| Fully deployed and 100% Tested; On production server
 
| 1.0
 
| 1.0
| style="color: #FFA500;" | In Progress
+
| style="color: #008000;" | Completed
 
|-
 
|-
 
| FMS Module II
 
| FMS Module II
| Extend a booking that has begun
+
| Remind and extend a booking that has begun
| Yes
+
| No
| In Progress
+
| Fully deployed and 100% Tested; On production server
 
| 1.0
 
| 1.0
| style="color: #FFA500;" | In Progress
+
| style="color: #008000;" | Completed
 
|-
 
|-
 
| FMS Module II
 
| FMS Module II
 
| Modify make, delete, view and search facilities to replicate GovTech's System
 
| Modify make, delete, view and search facilities to replicate GovTech's System
| Yes
+
| No
| In Progress
+
| Fully deployed and 100% Tested; On production server
 
| 1.0
 
| 1.0
| style="color: #FFA500;" | In Progress
+
| style="color: #008000;" | Completed
 
|-
 
|-
 
| Facility Booking ChatBot Module II
 
| Facility Booking ChatBot Module II
 
| Advanced NLP
 
| Advanced NLP
| Yes
+
| No
 
| Fully deployed and 100% Tested; On production server
 
| Fully deployed and 100% Tested; On production server
 
| 1.0
 
| 1.0
Line 374: Line 384:
 
| Facility Booking ChatBot Module II
 
| Facility Booking ChatBot Module II
 
| Re-prompt user with other room to book in case of clash in booking
 
| Re-prompt user with other room to book in case of clash in booking
| Yes
+
| No
 
| Fully deployed and 100% Tested; On production server
 
| Fully deployed and 100% Tested; On production server
 
| 1.0
 
| 1.0
Line 381: Line 391:
 
| Analytics Dashboard and Bots Integration Module
 
| Analytics Dashboard and Bots Integration Module
 
| Expose EvBot Event report
 
| Expose EvBot Event report
| Yes
+
| No
 
| Fully deployed and 100% Tested; On production server
 
| Fully deployed and 100% Tested; On production server
 
| 1.0
 
| 1.0
Line 388: Line 398:
 
| Analytics Dashboard and Bots Integration Module
 
| Analytics Dashboard and Bots Integration Module
 
| Dashy displays event statistics for event organisers
 
| Dashy displays event statistics for event organisers
| Yes
+
| No
 
| Fully deployed and 100% Tested; On production server
 
| Fully deployed and 100% Tested; On production server
 
| 1.0
 
| 1.0
Line 395: Line 405:
 
| Analytics Dashboard and Bots Integration Module
 
| Analytics Dashboard and Bots Integration Module
 
| Expose FaBot usage metrics
 
| Expose FaBot usage metrics
| Yes
+
| No
| In Progress
+
| Fully deployed and 100% Tested; On production server
 
| 1.0
 
| 1.0
| style="color: #FFA500;" | In Progress
+
| style="color: #008000;" | Completed
 
|-
 
|-
 
| Analytics Dashboard and Bots Integration Module
 
| Analytics Dashboard and Bots Integration Module
 
| Expose EvBot usage metrics
 
| Expose EvBot usage metrics
| Yes
+
| No
 
| Fully deployed and 100% Tested; On production server
 
| Fully deployed and 100% Tested; On production server
 
| 1.0
 
| 1.0
Line 409: Line 419:
 
| Analytics Dashboard and Bots Integration Module
 
| Analytics Dashboard and Bots Integration Module
 
| View FaBot usage metrics for workplace managers
 
| View FaBot usage metrics for workplace managers
| Yes
+
| No
| In Progress
+
| Fully deployed and 100% Tested; On production server
 
| 1.0
 
| 1.0
| style="color: #FFA500;" | In Progress
+
| style="color: #008000;" | Completed
 
|-
 
|-
 
| Analytics Dashboard and Bots Integration Module
 
| Analytics Dashboard and Bots Integration Module
 
| View EvBot usage metrics for workplace managers
 
| View EvBot usage metrics for workplace managers
| Yes
+
| No
 
| Fully deployed and 100% Tested; On production server
 
| Fully deployed and 100% Tested; On production server
 
| 1.0
 
| 1.0
Line 437: Line 447:
 
| Event ChatBot (Organisers) II
 
| Event ChatBot (Organisers) II
 
| Organisers trigger prompt for participants to check in
 
| Organisers trigger prompt for participants to check in
| Yes
+
| No
 
| Fully deployed and 100% Tested; On production server
 
| Fully deployed and 100% Tested; On production server
 
| 1.0
 
| 1.0
Line 444: Line 454:
 
| Event ChatBot (Organisers) II
 
| Event ChatBot (Organisers) II
 
| Display attendees yet to complete survey
 
| Display attendees yet to complete survey
| Yes
+
| No
 
| Fully deployed and 100% Tested; On production server
 
| Fully deployed and 100% Tested; On production server
 
| 1.0
 
| 1.0
Line 451: Line 461:
 
| Analytics Dashboard Mobile Responsiveness
 
| Analytics Dashboard Mobile Responsiveness
 
| Optimize web pages
 
| Optimize web pages
| Yes
+
| No
 
| Fully deployed and 100% Tested; On production server
 
| Fully deployed and 100% Tested; On production server
 
| 1.0
 
| 1.0
Line 471: Line 481:
 
| Database Optimization Module
 
| Database Optimization Module
 
| Optimize syncing of new data from Workplace@FB with Crunchy's database
 
| Optimize syncing of new data from Workplace@FB with Crunchy's database
| Yes
+
| No
| To be completed in iteration 8
+
| Fully deployed and 100% Tested; On production server
 
| 1.0
 
| 1.0
| style="color: #3498DB;" | To be completed in iteration 8
+
| style="color: #008000;" | Completed
|-
 
| Database Optimization Module
 
| Separate Database into transactional and analytical for all applications
 
| Yes
 
| To be completed in iteration 8
 
| 1.0
 
| style="color: #3498DB;" | To be completed in iteration 8
 
|-
 
| Event ChatBot (Organisers) III
 
| Word cloud on survey results
 
| Yes
 
| To be completed in iteration 8
 
| 1.0
 
| style="color: #3498DB;" | To be completed in iteration 8
 
 
|-
 
|-
 
| Event ChatBot (Organisers) III
 
| Event ChatBot (Organisers) III
 
| Export Event Report as CSV
 
| Export Event Report as CSV
| Yes
+
| No
| To be completed in iteration 8
+
| Fully deployed and 100% Tested; On production server
 
| 1.0
 
| 1.0
| style="color: #3498DB;" | To be completed in iteration 8
+
| style="color: #008000;" | Completed
 
|-
 
|-
 
|}
 
|}
Line 501: Line 497:
  
 
=== Project Schedule ===
 
=== Project Schedule ===
This section shows a low level overview of our tasks in each iteration before acceptance and current point (mid term)
+
This section shows a low level overview of our tasks in each iteration before midterm and current point (final)
  
 
==== Planned ====
 
==== Planned ====
[[File:Ravenous - Acceptance Schedule.png|900px|center]]
+
[[File:Ravenous - Midterm for final Schedule.png|900px|center]]
  
 
==== Actual ====
 
==== Actual ====
[[File:Ravenous - Midterm Schedule.png|900px|center]]
+
[[File:Ravenous - final Schedule.png|900px|center]]
  
 
=== Project Metrics ===
 
=== Project Metrics ===
 
The chart shows the number of completed tasks against number of assigned tasks for all our iterations so far. The bigger the circle, the higher the percentage of task metrics<br><br>
 
The chart shows the number of completed tasks against number of assigned tasks for all our iterations so far. The bigger the circle, the higher the percentage of task metrics<br><br>
  
[[File:Ravenous - Midterm TM Summary.png|900px|center]]<br>
+
[[File:Ravenous - Final TM Summary.png|900px|center]]<br>
  
From Iteration 0 to 5, we keep track of task assignment date, expected due date and actual completion date. From there, we carry out task completion date analysis, individual task assignment analysis and group task assignment analysis. However, from iteration 6 onward, we made an improvement to our project management and keep track of expected time spent per task as well as actual time spent per task.<br><br>
 
<center>
 
[[Image: Ravenous - Iteration analysis 5.png |720px|Ravenous Iteration analysis 5]]
 
[[Image: Ravenous - Iteration analysis 6.png |720px|Ravenous Iteration analysis 6]]
 
</center>
 
 
<br>
 
<br>
 
The charts below show Total Bug Score per iteration, Total Number of Bugs per iteration and Average Bug Score per Bug per iteration respectively.<br><br>
 
The charts below show Total Bug Score per iteration, Total Number of Bugs per iteration and Average Bug Score per Bug per iteration respectively.<br><br>
Line 525: Line 516:
 
[[File:Ravenous - Total Bug Score.png|720px]]
 
[[File:Ravenous - Total Bug Score.png|720px]]
 
[[File:Ravenous - Total Bug Num.png|720px]]
 
[[File:Ravenous - Total Bug Num.png|720px]]
[[File:Ravenous - Avg Bug Score.png|720px]]
 
 
</center>
 
</center>
  
Line 532: Line 522:
 
[[File:Ravenousrm.png|720px|center]]<br>
 
[[File:Ravenousrm.png|720px|center]]<br>
  
From the period of Acceptance (18/8/2016) to Mid-Term presentation, we would like to highlight these two risks as area of concerns.
+
From the period of Mid Term (9/10/2017) to Final presentation, we would like to highlight these two risks as area of concerns.
  
 
{| class="wikitable" style="background-color:#FFFFFF; width:70%;margin-left:15%; text-align: center;"
 
{| class="wikitable" style="background-color:#FFFFFF; width:70%;margin-left:15%; text-align: center;"
Line 554: Line 544:
 
| Our team is part of the Multi-Company Group with PSD, MOE and Facebook. We are in touch with Rohan who we can approach if we have any problem with the Workplace instance
 
| Our team is part of the Multi-Company Group with PSD, MOE and Facebook. We are in touch with Rohan who we can approach if we have any problem with the Workplace instance
 
|-
 
|-
| 10
+
| 11
| Project Management Risk
+
| Technical Risk
| GovTech is keen to share their API with us for us to explore the potential of our applications on government platform. However, the API from GovTech is only available in late Oct/early Nov
+
| Team is given a relatively small server to deploy on the government instance. Thus, code has to be optimized in order the application to work.
 
| High
 
| High
 
| High
 
| High
 
| A
 
| A
 
| Mitigation
 
| Mitigation
| We are intending to replicate some of the functions that GovTech applications have. This can not only prepare us for possible integration but also tap on their knowledge to create functions that users want.
+
| Developers to learn and understand how to optimize code through online resources
 
|-
 
|-
 
|}
 
|}
  
 
<br>
 
<br>
From the period of Acceptance (18/8/2016) to Mid-Term presentation, we have a total of 29 change requests. Out of the 29 change requests, 13 changes proposed by the team based on results from UT2 and majority of them are UX/UI changes. We would like to list 3 interesting/impactful change requests.
+
From the period of Mid Term (9/10/2017) to Final presentation, we have a total of 3 change requests.
 
<br>
 
<br>
  
Line 581: Line 571:
 
| style="background-color:#28B463; color: #000000; font-weight: bold;" | Status of Request
 
| style="background-color:#28B463; color: #000000; font-weight: bold;" | Status of Request
 
|-
 
|-
| 11
+
| 38
 
| Dashy, Sponsor
 
| Dashy, Sponsor
| Mobile Responsiveness
+
| Top Contributor in Agency
 
| 2
 
| 2
 
| 2
 
| 2
| High, Allow people to use it on the go
+
| High, Allows Workplace Managers to quickly know who are the top contributor in their agency. Currently, workplace default dashboard only allow them to view top contributor of whole of government
 
| 4, High
 
| 4, High
 
| Accept change request and team discuss the most appropriate iteration to implement the change request
 
| Accept change request and team discuss the most appropriate iteration to implement the change request
 
| Closed
 
| Closed
 
|-
 
|-
| 25
+
| 39
| FaBot, Team Ravenous
+
| Dashy, Sponsor
| Input format for date/time too rigid. Require advanced NLP.
+
| Average seen rate of all groups in agency
 
| 2
 
| 2
 
| 2
 
| 2
| High, This will improve user experience by giving users more options to type
+
| High, It allows Workplace Managers to quickly know how many posts are being viewed by how many their users in their agency, hence this supplements them in knowing activity rate in their agency
 
| 4, High
 
| 4, High
 
| Accept change request and team discuss the most appropriate iteration to implement the change request
 
| Accept change request and team discuss the most appropriate iteration to implement the change request
 
| Closed
 
| Closed
 
|-
 
|-
| 36
+
| 40
| EvBot, Team Ravenous
+
| Dashy, Sponsor
| EvBot will automatically prompt user to link event the moment it detected user has created an event
+
| Deploy on WOG AWS server
| 2
+
| 3
| 2
+
| 3
| High, It will guide the users in using EvBot to link events
+
| High, Allow the team to deploy for live usage
| 4, High
+
| 6, High
 
| Accept change request and team discuss the most appropriate iteration to implement the change request
 
| Accept change request and team discuss the most appropriate iteration to implement the change request
 
| Closed
 
| Closed
Line 615: Line 605:
 
=== Technical Complexity ===
 
=== Technical Complexity ===
  
==== Asynchronous Nature of NodeJS ====
+
==== Technical complexity 1 - Deployment Server Memory  ====
<br><b>SetTimeout</b><br>
+
We had to work with an Amazon EC2 T2.micro instance to host our analytics processing server. The T2.Micro instance only provides 1gb memory and the processing server is expected to load an expected volume of 1 million data from the Whole Of Government Workplace instance.<br><br>
To stop the asynchronous calls initially, setTimeout was used. This was used via trial and error basis to see how long a previous request call would take, and run the next block of codes after the elapsed time.<br><br>
+
 
This is easy to implement however, this comes with many disadvantages. For one, this is a highly unreliable method. Although a request call to Graph/Messenger API would usually take less than second, there could be instances where FB’s server slow down and exceed a second. This would cause our app to crash since there would still be many variables not instantiated yet. This method is essentially hardcoding.<br><br>
+
The initial task of populating our database was a challenge as we constantly reached our memory limit while trying to retrieve and store the data from Facebook Workplace, into our database. Upgrading the instance to a better one was an option but we decided to optimize the memory usage of our code first.<br><br>
Secondly, it causes the code to be very hard to read for other developers. With each setTimeout, the next block of codes has to be indented in. With the nature of our code, we will be affected by asynchronous calls many times, causing our codes to be unnecessarily heavily indented to the right. <br><br>
+
 
 +
<b>(i) Bluebird Promises</b><br>
 +
During our mid-term presentation, we mentioned that we used the Native Promise library to approach our asynchronous issues. Bluebird promises work the same say as Node’s native Promise however, it is less memory intensive. The benchmark results can be found on bluebird’s documentation but we decided to run some tests ourselves. Here is the score from the tests we ran using Node’s process.memoryUsage() function:<br><br>
 +
 
 +
[[File: Ravenous - ftc1a.png|480px|center]]<br>
 +
[[File: Ravenous - ftc1b.png|480px|center]]<br>
 +
 
 +
Through our results, the heapUsed for the bluebird library is lesser than the native Promise, confirming the benchmark scores found online. Thus, we refactored the promises that were used by memory intensive functions to make use of the bluebird library<br><br>
 +
 
 +
 
 +
<b>(ii) Node Garbage Collector</b><br>
 +
To free up memory within the code, we had to make use of Node’s garbage collector which runs in the background. The garbage collector frees up memory when the objects have no reference to any other objects. In order to reduce the memory usage of our code, we explicitly force node’s garbage collector to remove objects that are unused by assigning a null value to the object before ending the function.<br><br>
 +
 
 +
[[File: Ravenous - ftc1c.png|480px|center]]<br>
 +
[[File: Ravenous - ftc1d.png|480px|center]]<br>
 +
 
  
[[File: Ravenous - tc1.png|720px|center]]<br>
+
<b>(iii) Mongo Cursor and Buffer (Managing Memory Usage)</b><br>
[[File: Ravenous - tc2.png|720px|center]]<br>
+
The usage of cursor is similar to streaming data. Instead of fetching all the data from the database and consuming it all at once, cursor streams the data to our server. <br><br>
  
<br><b>Callbacks</b><br>
+
[[File: Ravenous - ftc1e.png|480px|center]]<br>
Callback was a better method than hardcoding timeouts. This guarantees a request call is completed before proceeding to the next block of codes for most cases. Callbacks can solve most problems but not all. For example, getting multiple values from request calls first, before proceeding and making multiple request calls in order.<br><br>
 
Another drawback to using callbacks, is that the indentation problem is not solved. With every callback function, the next block of code is indented, making it difficult for other developers to read.<br><br>
 
  
[[File: Ravenous - tc3.png|720px|center]]<br>
+
After streaming the data, we manually implemented our own Buffer to batch process the records that were streamed. With the combination of Mongo’s cursor and our buffer, it helped us to manage the memory usage of our application.
[[File: Ravenous - tc4.png|720px|center]]<br>
 
[[File: Ravenous - tc5.png|720px|center]]<br>
 
  
<br><b>Recursion</b><br>
 
To make multiple calls in order, Promises was also explored but it did not work. This could also be due to the lack of understanding the Promise concept at that period. Since timeouts, callbacks and promises weren’t working, we explored the use of Recursion to send messages in guaranteed order to users.<br><br>
 
  
[[File: Ravenous - tc6.png|720px|center]]<br>
+
<b>(iv) EcmaScript6 (ES6)’s let and CONST declaration</b><br>
 +
We realized that with the new declarations of let and CONST, it uses less memory than the old global declaration ‘var’. We refactored our codes to make use of only ES6’s let and CONST declaration.<br><br><br>
  
<br><b>Promises</b><br>
+
==== Technical complexity 2 - Code Optimization to improve Runtime ====
Promises required some reading up and researching before actual implementation as this is a new asynchronous concept with little examples online as compared to callbacks and setTimeout. Promises solves all of our problems. It guarantees the request is done before moving on, it can make multiple request calls and storing the values synchronously and lastly, do all these while maintaining clean code.<br><br>
+
Since we had to process approximately a million data, our codes have to run quickly. One of the reasons why Node.js was chosen as the language for this task, is for this very purpose.<br><br>
  
[[File: Ravenous - tc7.png|720px|center]]<br>
+
<b>(i) Async Parallel</b><br>
  
This is a really clean way of sending multiple messages and performing other calls within the function. Imagine using callbacks or timeouts, after each sendMessage and delay function, all subsequent codes has to be indented. For the example, the code would have had to be indented more than 10 times!<br><br>
+
With the new version of Node.js (v7.6 and above), came the native async library. We took advantage of this library by making asynchronous queries to our database so that database queries can run in parallel.<br><br>
  
[[File: Ravenous - tc8.png|720px|center]]<br>
+
[[File: Ravenous - ftc2a.png|480px|center]]<br>
  
We used Promise.all to collect all contents of the resolved body from the promise methods. In the above example, a request is made to Graph API to retrieve the Event Image URL for a list of Events. Thereafter, we use a Promise.all to retrieve all the URLs obtained previously. This is done synchronously. <br><br>
+
In the simple example above, two functions (calls) are made to the database. However, with the async.parallel function, they are made at the same time. Only when both queries have finished executing, will the code continue with the results of both queries. This helps to improve the runtime processing speed when widely used across the code.<br><br>
  
[[File: Ravenous - tc9.png|720px|center]]<br>
 
  
Async/Await – Node Version >= 7.6.0<br>
+
<b>(ii) Native Mongo Queries</b><br>
Async/await has recently been shipped with official support in Node 7.6 (released in early 2017). It is built on top of promises and needs to be used in conjunction with them. While Promises keeps code cleaner in comparison to using callbacks, async/await is even more concise.<br><br>
 
  
[[File: Ravenous - tc10.png|720px|center]]<br>
+
Although the library that we used to interact with MongoDB, named Mongoose, has certain functions to aid developers in using a MongoDB, it is limited. The general find() or findOneAndUpdate() queries are insufficient for more complex queries. Therefore, an easy but inefficient solution is to find all records, and sift out the records we need by iterating through the records. This is extremely inefficient as compared to using the native queries.<br><br>
  
In this example, we used the await keyword before calling asynchronous functions (that must return promises). If this were written with standard promises, there would need to be a chain of then blocks and return statements. Async/await makes code look synchronous and improves readability significantly.<br><br>
+
Therefore, we had to learn Mongo Queries to improve the speed of our application. We made use of Mongoose’s aggregate function include in Mongo’s operators to perform some of the more complex queries.<br><br>
  
==== ChatBot Sessions ====
+
Some of the operators that we used include:$match, $group, $project, $sort, $count<br><br>
  
Messenger platform, the platform the team is developing the bots on, does not support sessions out of the box. <br>
+
[[File: Ravenous - ftc2b.png|480px|center]]<br><br><br>
E.g. <br>
 
Bot: Can I take your order?<br>
 
User: I want a Hamburger.<br>
 
Bot: What? Why?<br><br>
 
Sessions in our chatbots is crucial as our use case requires many user inputs. Even so, user input alone is not enough. Our chatbots will have to take in multiple user inputs in order, and perform validation on those user inputs. When the inputs are invalid, the bot needs to recognize and re-prompt the user for a valid input.<br><br>
 
To overcome this problem, we implemented our own ‘sessions framework’. Firstly, we made use of a database to store a user’s session.<br><br>
 
  
[[File: Ravenous - tc11.png|720px|center]]<br>
+
==== Technical complexity 3 - Making Large Number of API Calls to Workplace Graph API ====
 +
As of 1st November 2017, there are a little more than 150,000 public servants on the Whole of Government Workplace instance. Retrieving every detail on these accounts would result in a few hundreds of thousand requests to Workplace’s API. <br><br>
  
We also implemented a set of codes in a session.js file to manipulate and access the session variables. The way the code works, is that it’ll query the database with every user call to the bot to check for any existing session (e.g. bot is in the middle of prompting Booking Date). If an existing session exists, it needs to listen to the relevant call. E.g. If bot is prompting for a booking date, a bot must expect a booking date, and not a booking time.<br><br>
+
This was an issue initially as we were met with lots of rate limiting and throttle error that were set in Workplace’s Graph API. This limits us to about 200 calls per hour per user in aggregate. We adopted some techniques to overcome this issue: <br><br>
  
[[File: Ravenous - tc12.png|720px|center]]<br>
+
<b>(i) Bluebird Concurrency Control (Promise.Map)</b><br>
[[File: Ravenous - tc13.png|720px|center]]<br>
+
Bluebird’s Promise.Map allowed us to manage the number of requests we made to Workplace API at any moment.<br><br>
[[File: Ravenous - tc14.png|720px|center]]<br>
 
  
The advantage of a self-implemented session framework, is that we can control validations tightly. When an invalid input is entered, the bot will recognize it, and immediately prompt for a valid input. <br><br>
+
[[File: Ravenous - ftc3a.png|480px|center]]<br>
However, there are disadvantages to it as well. Firstly, with every user message to the bot, a query to the DB is made to check for any existing session. Even when a user is ‘chatting’ with the bot saying ‘Hello!’, a DB query is made. This will slow the DB down significantly when multiple users are using the bot at the same time, or when a user spams the bot with messages. This caused numerous bugs when the bot did not reply on the first input. When the user enters his input the second time, the bot responds.<br><br>
 
Secondly, to have instant validation, users must input in their date and time in a specific format. Any deviations from the format will render it an invalid input. The rigidity was not liked by most users.<br><br>
 
  
<center>
+
The code example above limits us to make only 10 concurrent calls to the ‘populatePostsForGroup’ function. In this function, requests are made to Workplace Graph API, instead of just shooting thousands of requests to Workplace Graph API in a matter of seconds.<br><br>
[[File: Ravenous - tc15.png|240px]]
+
 
[[File: Ravenous - tc16.png|240px]]<br>
+
Furthermore, the concurrency control in bluebird is able continuously make calls at the specified rate of 10. For example, when two requests are done, the queued requests will send an additional two to Graph API.<br><br>
</center>
+
 
 +
In essence, the concurrency control balances between fast API calls and API rate limit errors. <br><br>
 +
 
 +
<b>(ii) Request Retry</b><br>
 +
Since hundreds of thousand API calls have to be made, there are bound to be failures in some of the calls. During our test phase, we encountered some of failures with minimal explanation. After recommendation from our Sponsor’s Tech team, we concluded that the best way to solve this error, is to just try sending the request again. <br><br>
 +
 
 +
To do this, we made use of the requestretry library on NPM. <br><br>
 +
 
 +
[[File: Ravenous - ftc3b.png|480px|center]]<br>
 +
 
 +
The library helped us to easily resend the request to whenever an error is returned in the response. From the above options set, the request will keep trying at a maximum attempt of 5 times. After the 5th attempt is exhausted, it will pause the retry for five seconds before continuing to try. <br><br>
  
To overcome the UX and performance issue, we made use of an external API (apiai) on top of our codes. We used apiai to parse a user’s message into intents and entities. For example, 11/12/2017, 11-12-17 and even 11 Dec 2017 are all recognized as date formats. This is the same for time periods. <br><br>
+
This helped us to overcome the unknown Workplace API errors and allowed us to continue retrieve data from Workplace reliably.<br><br><br>
However, this is lacking in terms of validation. If a user enters a date that is before today e.g. 11/01/2001, APIAI will accept as it is still a ‘valid’ date to them. For the bot’s use case, this date is invalid as there is no way a user can book a facility before today. Similarly, if a user types a booking time of 3pm to 1pm, it should be invalid.<br><br>
 
To overcome this, we added our set of validation codes. <br><br>
 
  
[[File: Ravenous - tc17.png|720px|center]]<br>
+
==== Technical complexity 4 - Real-Time Data (New) ====
 +
With the new Webhook subscription feature offered by Workplace Graph API, it allows us to update our database data in real-time. <br><br>
  
Another challenge we faced while using apiai is to allow the user to only fill up the invalid fields e.g. User only needs to fill up booking date instead of booking date + booking time + booking room again in an event of an initial invalid input.<br><br>
+
In order to listen to these events from Workplace, we had to set up our own webhooks. The webhooks were set up using Workplace’s interface.<br><br>
This was difficult as apiai does not allow us to reset the session context on the fly as this is tightly controlled by them. For example, as soon as a booking date is invalid, we tried to send a POST request to apiai to reset the context, causing it to re-prompt user for a booking date. But apiai rejected the call, and will continue asking for the remaining prompts.<br><br>
 
A workaround we found, is to manually store all invalid input a user has. With these errors, we’ll make a new call to the API, leaving out those errors. Once these fields are left out, APIAI will only prompt those fields.<br><br>
 
  
[[File: Ravenous - tc18.png|720px|center]]<br>
+
The topics our webhooks are subscribed to are: Posts (New and Modified), Comments (New and Modified), Users (New and Deleted), Events (New)<br><br>
[[File: Ravenous - tc19.png|360px|center]]<br>
 
  
Finally, since session is not supported, it was difficult to pass variables around in our code. We were used to storing variables in a session, and retrieving the variables from the session in another file. Since this was not supported, we coded out a custom function such that we were able to pass variables around using the messenger platform’s Postback/payload system, following the URL query parameters format.<br><br>
+
[[File: Ravenous - ftc4a.png|720px|center]]<br>
  
[[File: Ravenous - tc20.png|480px|center]]<br>
+
Every time a new event occurs, for example a user just posted a new comment, Workplace will send us the payload to the subscribed webhook accordingly. We will then handle the events accordingly.<br><br><br>
[[File: Ravenous - tc21.png|480px|center]]<br>
 
[[File: Ravenous - tc22.png|480px|center]]<br>
 
[[File: Ravenous - tc23.png|480px|center]]<br>
 
  
 
== Quality of Product ==
 
== Quality of Product ==
Line 733: Line 726:
 
|-
 
|-
  
|rowspan="4"| Testing
+
|rowspan="5"| Testing
 
|| [https://www.dropbox.com/s/ca2rqe23w4px2lv/Bug%20Log.xlsx?dl=0 Bug Log]  
 
|| [https://www.dropbox.com/s/ca2rqe23w4px2lv/Bug%20Log.xlsx?dl=0 Bug Log]  
 
|-  
 
|-  
Line 741: Line 734:
 
|-
 
|-
 
|| [https://wiki.smu.edu.sg/is480/IS480_Team_wiki%3A_2017T1_Ravenous_User_Testing_2 UT2]
 
|| [https://wiki.smu.edu.sg/is480/IS480_Team_wiki%3A_2017T1_Ravenous_User_Testing_2 UT2]
 +
|-
 +
|| [https://wiki.smu.edu.sg/is480/IS480_Team_wiki%3A_2017T1_Ravenous_User_Testing_3 UT3]
 
|-
 
|-
 
|}
 
|}
Line 748: Line 743:
 
The deployment links are as follow:
 
The deployment links are as follow:
  
We have 2 versions of Dashy for different purposes. Staging refers to application in the development phase and production refers to applications are that production ready. Only these two applications have a user interface.<br>
+
We have 3 versions of Dashy for different purposes. Staging refers to application in the development phase and production refers to applications are that production ready. WOG Live instance is the one we deploy on WOG's server. Only these three applications have a user interface.<br>
 
<ul>
 
<ul>
 
<li>[https://dashy-staging.herokuapp.com Dashy (Analytics Dashboard, staging)]</li>
 
<li>[https://dashy-staging.herokuapp.com Dashy (Analytics Dashboard, staging)]</li>
 
<li>[https://dashy-production.herokuapp.com Dashy (Analytics Dashboard, production)]</li>
 
<li>[https://dashy-production.herokuapp.com Dashy (Analytics Dashboard, production)]</li>
 +
<li>[http://dashy.ap-southeast-1.elasticbeanstalk.com/ (Analytics Dashboard, <b>WOG Live Instance</b>)]</li>
 
</ul>
 
</ul>
 
<br>
 
<br>
Line 771: Line 767:
  
 
=== Testing ===
 
=== Testing ===
Team Ravenous has conducted a total of 2 User Testings.<br>
+
Team Ravenous has conducted a total of 3 User Testings.<br>
 
The first user testing placed emphasize on whether users know how to use our applications, whether they understand the flow of tasks and observe how they interact with our ChatBots. <br>
 
The first user testing placed emphasize on whether users know how to use our applications, whether they understand the flow of tasks and observe how they interact with our ChatBots. <br>
 
The second user test is about the usability improvement that we have made since user testing 1.<br>
 
The second user test is about the usability improvement that we have made since user testing 1.<br>
 +
The third user test is about the usability improvement that we have made since user testing 2.<br>
  
 
The results of UT1 can be found [https://wiki.smu.edu.sg/is480/IS480_Team_wiki%3A_2017T1_Ravenous_User_Testing_1 here]<br>
 
The results of UT1 can be found [https://wiki.smu.edu.sg/is480/IS480_Team_wiki%3A_2017T1_Ravenous_User_Testing_1 here]<br>
The results of UT2 can be found [https://wiki.smu.edu.sg/is480/IS480_Team_wiki%3A_2017T1_Ravenous_User_Testing_2 here]<br><br>
+
The results of UT2 can be found [https://wiki.smu.edu.sg/is480/IS480_Team_wiki%3A_2017T1_Ravenous_User_Testing_2 here]<br>
 +
The results of UT3 can be found [https://wiki.smu.edu.sg/is480/IS480_Team_wiki%3A_2017T1_Ravenous_User_Testing_3 here]<br><br>
  
 
Details of internal testing can be found in the deliverable table above.
 
Details of internal testing can be found in the deliverable table above.
Line 787: Line 785:
 
=== Individual Reflection ===
 
=== Individual Reflection ===
  
[[File: Ravenous - Midterm learning 1.png|720px|center]]
+
[[File: Ravenous - final learning 1.png|720px|center]]
[[File: Ravenous - Midterm learning 2.png|720px|center]]
+
 
[[File: Ravenous - Midterm learning 3.png|720px|center]]
 
 
<!--/Content-->
 
<!--/Content-->

Latest revision as of 14:53, 22 November 2017

Ravenouslogo.png


Iconh.png
Iconau1.png
Iconpo1.png
Iconpm1.png
Icond1.png




Project Progress Summary

Ravenous - Midterm Project About.png


As of 14th November 2017, we have completed 41% of iteration 9 and this is our last iteration.

To view final slides, click here
To access EvBot & FaBot, login to Workplace@Facebook instance by clicking here
To access Dashy (our instance), click here
To access Dashy (WOG's live instance), click here

To view handover documentation, click here

For more information on how to access our applications, please view Quality of Product > Deployment Section

Project Highlights & Snapshots

Ravenous - Final Project Highlights.png

Project Management

Project Status

The three diagrams below show the progress of our project, a high level overview of completion of project modules and a high level overview of change in scope to our project modules respectively.

Ravenous - FinalProjectStatus.png


Ravenous - FinalFunctionalitiesSummary.png



Ravenous - FinalFunctionalitiesChange1.png

Scope: Core

Module Task New Feature? Status Confidence level Comments
Event ChatBot (Organiser) Register an event No Fully deployed and 100% Tested; On production server 1.0 Completed
Event ChatBot (Organiser) Remove event No Fully deployed and 100% Tested; On production server 1.0 Completed
Event ChatBot (Organiser) Close event No Fully deployed and 100% Tested; On production server 1.0 Completed
Event ChatBot (Organiser) Add & Remove questions No Fully deployed and 100% Tested; On production server 1.0 Completed
Event ChatBot (Organiser) Send Survey Questions No Fully deployed and 100% Tested; On production server 1.0 Completed
Event ChatBot (Organiser) Send Survey Reminder to participants No Fully deployed and 100% Tested; On production server 1.0 Completed
Event ChatBot (Organiser) View Survey Questions No Fully deployed and 100% Tested; On production server 1.0 Completed
Event ChatBot (Organiser) View Event Snapshot No Fully deployed and 100% Tested; On production server 1.0 Completed
Event ChatBot (Organiser) About EvBot No Fully deployed and 100% Tested; On production server 1.0 Completed
Event ChatBot (Participants) Check-in Event Attendance No Fully deployed and 100% Tested; On production server 1.0 Completed
Event ChatBot (Participants) View Surveys No Fully deployed and 100% Tested; On production server 1.0 Completed
Event ChatBot (Participants) About EvBot No Fully deployed and 100% Tested; On production server 1.0 Completed
Facility Booking ChatBot & FMS Module I Search available facilities No Fully deployed and 100% Tested; On production server 1.0 Completed
Facility Booking ChatBot & FMS Module I Book available facilities No Fully deployed and 100% Tested; On production server 1.0 Completed
Facility Booking ChatBot & FMS Module I View booked facilities No Fully deployed and 100% Tested; On production server 1.0 Completed
Facility Booking ChatBot & FMS Module I Cancel booked facilities No Fully deployed and 100% Tested; On production server 1.0 Completed
Analytics Dashboard Module I View claim rate vs active rate for all agencies and specific agency chart No Fully deployed and 100% Tested; On production server 1.0 Completed
Analytics Dashboard Module I View aggregated engagement scores of groups within an agency No Fully deployed and 100% Tested; On production server 1.0 Completed
Analytics Dashboard Module I View breakdown of engagement scores of groups within an agency No Fully deployed and 100% Tested; On production server 1.0 Completed
Analytics Dashboard Module I View Pc vs Mobile Graph No Delivered 1.0 Completed
Analytics Dashboard Module II View group privacy setting across specific group charts No Fully deployed and 100% Tested; On production server 1.0 Completed
Analytics Dashboard Module II View group activity charts No Fully deployed and 100% Tested; On production server 1.0 Completed
Analytics Dashboard Module II View interaction analysis of specific agency No Fully deployed and 100% Tested; On production server 1.0 Completed
Analytics Dashboard Module III View number of active users charts No Fully deployed and 100% Tested; On production server 1.0 Completed
Analytics Dashboard Module III View content on workplace chart No Fully deployed and 100% Tested; On production server 1.0 Completed
Analytics Dashboard Module III View word cloud chart No Delivered 1.0 Completed
Analytics Dashboard Module III View post-time & comment-time chart No Fully deployed and 100% Tested; On production server 1.0 Completed
Authentication & Security Module Register account on Dashy OTP Bot No Fully deployed and 100% Tested; On production server 1.0 Completed
Authentication & Security Module OTP Bot No Fully deployed and 100% Tested; On production server 1.0 Completed
Authentication & Security Module Login/Logout on Dashy No Fully deployed and 100% Tested; On production server 1.0 Completed
Authentication & Security Module Forget password No Fully deployed and 100% Tested; On production server 1.0 Completed
Authentication & Security Module Secure all API Calls with JWT Token No Fully deployed and 100% Tested; On production server 1.0 Completed

Scope: Secondary

Module Task New Feature? Status Confidence level Comments
Analytics Dashboard Module IV Export dashboard to CSV File No Fully deployed and 100% Tested; On production server 1.0 Completed
Analytics Dashboard Module IV Filtering options based on user activity/profile No Fully deployed and 100% Tested; On production server 1.0 Completed
Analytics Dashboard Module IV Dashy additional metrics: Overview, posts/comments of previous day vs last week’s same day No Fully deployed and 100% Tested; On production server 1.0 Completed
Analytics Dashboard Module IV Dashy additional metrics: Group, Measure posts, comments and reactions in a given period of time and see the most popular days and times that members engage. (advanced filtering) No Fully deployed and 100% Tested; On production server 1.0 Completed
Analytics Dashboard Module IV Dashy additional metrics: Most seen posts by number of seen per group Yes Fully deployed and 100% Tested; On production server 1.0 Completed
Analytics Dashboard Module IV Dashy additional metrics: Top contributor by agency Yes Fully deployed and 100% Tested; On production server 1.0 Completed
FMS Module II Release a booking that has begun No Fully deployed and 100% Tested; On production server 1.0 Completed
FMS Module II Remind and extend a booking that has begun No Fully deployed and 100% Tested; On production server 1.0 Completed
FMS Module II Modify make, delete, view and search facilities to replicate GovTech's System No Fully deployed and 100% Tested; On production server 1.0 Completed
Facility Booking ChatBot Module II Advanced NLP No Fully deployed and 100% Tested; On production server 1.0 Completed
Facility Booking ChatBot Module II View room host details on workplace and have the ability to contact host No Fully deployed and 100% Tested; On production server 1.0 Completed
Facility Booking ChatBot Module II Re-prompt user with other room to book in case of clash in booking No Fully deployed and 100% Tested; On production server 1.0 Completed
Analytics Dashboard and Bots Integration Module Expose EvBot Event report No Fully deployed and 100% Tested; On production server 1.0 Completed
Analytics Dashboard and Bots Integration Module Dashy displays event statistics for event organisers No Fully deployed and 100% Tested; On production server 1.0 Completed
Analytics Dashboard and Bots Integration Module Expose FaBot usage metrics No Fully deployed and 100% Tested; On production server 1.0 Completed
Analytics Dashboard and Bots Integration Module Expose EvBot usage metrics No Fully deployed and 100% Tested; On production server 1.0 Completed
Analytics Dashboard and Bots Integration Module View FaBot usage metrics for workplace managers No Fully deployed and 100% Tested; On production server 1.0 Completed
Analytics Dashboard and Bots Integration Module View EvBot usage metrics for workplace managers No Fully deployed and 100% Tested; On production server 1.0 Completed
Event ChatBot (Organisers) II Display attendee that didn't turn up No Fully deployed and 100% Tested; On production server 1.0 Completed
Event ChatBot (Organisers) II Organisers message broadcast No Fully deployed and 100% Tested; On production server 1.0 Completed
Event ChatBot (Organisers) II Organisers trigger prompt for participants to check in No Fully deployed and 100% Tested; On production server 1.0 Completed
Event ChatBot (Organisers) II Display attendees yet to complete survey No Fully deployed and 100% Tested; On production server 1.0 Completed
Analytics Dashboard Mobile Responsiveness Optimize web pages No Fully deployed and 100% Tested; On production server 1.0 Completed

Scope: Good to have

Module Task New Feature? Status Confidence level Comments
Database Optimization Module Optimize syncing of new data from Workplace@FB with Crunchy's database No Fully deployed and 100% Tested; On production server 1.0 Completed
Event ChatBot (Organisers) III Export Event Report as CSV No Fully deployed and 100% Tested; On production server 1.0 Completed


Project Schedule

This section shows a low level overview of our tasks in each iteration before midterm and current point (final)

Planned

Ravenous - Midterm for final Schedule.png

Actual

Ravenous - final Schedule.png

Project Metrics

The chart shows the number of completed tasks against number of assigned tasks for all our iterations so far. The bigger the circle, the higher the percentage of task metrics

Ravenous - Final TM Summary.png



The charts below show Total Bug Score per iteration, Total Number of Bugs per iteration and Average Bug Score per Bug per iteration respectively.

Ravenous - Total Bug Score.png Ravenous - Total Bug Num.png

Project Risks

Ravenousrm.png


From the period of Mid Term (9/10/2017) to Final presentation, we would like to highlight these two risks as area of concerns.

S/N Risk Type Risk Description Likelihood Impact Level Risk Grade Strategy Adopted Action
8 Technical Risk Our project is heavily dependent on Facebook API that we have no control over. High High A Mitigation Our team is part of the Multi-Company Group with PSD, MOE and Facebook. We are in touch with Rohan who we can approach if we have any problem with the Workplace instance
11 Technical Risk Team is given a relatively small server to deploy on the government instance. Thus, code has to be optimized in order the application to work. High High A Mitigation Developers to learn and understand how to optimize code through online resources


From the period of Mid Term (9/10/2017) to Final presentation, we have a total of 3 change requests.

S/N Application, Requested by Change description Impact on Schedule Technical Complexity Business Value/Reason for Request Score Action Taken Status of Request
38 Dashy, Sponsor Top Contributor in Agency 2 2 High, Allows Workplace Managers to quickly know who are the top contributor in their agency. Currently, workplace default dashboard only allow them to view top contributor of whole of government 4, High Accept change request and team discuss the most appropriate iteration to implement the change request Closed
39 Dashy, Sponsor Average seen rate of all groups in agency 2 2 High, It allows Workplace Managers to quickly know how many posts are being viewed by how many their users in their agency, hence this supplements them in knowing activity rate in their agency 4, High Accept change request and team discuss the most appropriate iteration to implement the change request Closed
40 Dashy, Sponsor Deploy on WOG AWS server 3 3 High, Allow the team to deploy for live usage 6, High Accept change request and team discuss the most appropriate iteration to implement the change request Closed

Technical Complexity

Technical complexity 1 - Deployment Server Memory

We had to work with an Amazon EC2 T2.micro instance to host our analytics processing server. The T2.Micro instance only provides 1gb memory and the processing server is expected to load an expected volume of 1 million data from the Whole Of Government Workplace instance.

The initial task of populating our database was a challenge as we constantly reached our memory limit while trying to retrieve and store the data from Facebook Workplace, into our database. Upgrading the instance to a better one was an option but we decided to optimize the memory usage of our code first.

(i) Bluebird Promises
During our mid-term presentation, we mentioned that we used the Native Promise library to approach our asynchronous issues. Bluebird promises work the same say as Node’s native Promise however, it is less memory intensive. The benchmark results can be found on bluebird’s documentation but we decided to run some tests ourselves. Here is the score from the tests we ran using Node’s process.memoryUsage() function:

Ravenous - ftc1a.png


Ravenous - ftc1b.png


Through our results, the heapUsed for the bluebird library is lesser than the native Promise, confirming the benchmark scores found online. Thus, we refactored the promises that were used by memory intensive functions to make use of the bluebird library


(ii) Node Garbage Collector
To free up memory within the code, we had to make use of Node’s garbage collector which runs in the background. The garbage collector frees up memory when the objects have no reference to any other objects. In order to reduce the memory usage of our code, we explicitly force node’s garbage collector to remove objects that are unused by assigning a null value to the object before ending the function.

Ravenous - ftc1c.png


Ravenous - ftc1d.png



(iii) Mongo Cursor and Buffer (Managing Memory Usage)
The usage of cursor is similar to streaming data. Instead of fetching all the data from the database and consuming it all at once, cursor streams the data to our server.

Ravenous - ftc1e.png


After streaming the data, we manually implemented our own Buffer to batch process the records that were streamed. With the combination of Mongo’s cursor and our buffer, it helped us to manage the memory usage of our application.


(iv) EcmaScript6 (ES6)’s let and CONST declaration
We realized that with the new declarations of let and CONST, it uses less memory than the old global declaration ‘var’. We refactored our codes to make use of only ES6’s let and CONST declaration.


Technical complexity 2 - Code Optimization to improve Runtime

Since we had to process approximately a million data, our codes have to run quickly. One of the reasons why Node.js was chosen as the language for this task, is for this very purpose.

(i) Async Parallel

With the new version of Node.js (v7.6 and above), came the native async library. We took advantage of this library by making asynchronous queries to our database so that database queries can run in parallel.

Ravenous - ftc2a.png


In the simple example above, two functions (calls) are made to the database. However, with the async.parallel function, they are made at the same time. Only when both queries have finished executing, will the code continue with the results of both queries. This helps to improve the runtime processing speed when widely used across the code.


(ii) Native Mongo Queries

Although the library that we used to interact with MongoDB, named Mongoose, has certain functions to aid developers in using a MongoDB, it is limited. The general find() or findOneAndUpdate() queries are insufficient for more complex queries. Therefore, an easy but inefficient solution is to find all records, and sift out the records we need by iterating through the records. This is extremely inefficient as compared to using the native queries.

Therefore, we had to learn Mongo Queries to improve the speed of our application. We made use of Mongoose’s aggregate function include in Mongo’s operators to perform some of the more complex queries.

Some of the operators that we used include:$match, $group, $project, $sort, $count

Ravenous - ftc2b.png




Technical complexity 3 - Making Large Number of API Calls to Workplace Graph API

As of 1st November 2017, there are a little more than 150,000 public servants on the Whole of Government Workplace instance. Retrieving every detail on these accounts would result in a few hundreds of thousand requests to Workplace’s API.

This was an issue initially as we were met with lots of rate limiting and throttle error that were set in Workplace’s Graph API. This limits us to about 200 calls per hour per user in aggregate. We adopted some techniques to overcome this issue:

(i) Bluebird Concurrency Control (Promise.Map)
Bluebird’s Promise.Map allowed us to manage the number of requests we made to Workplace API at any moment.

Ravenous - ftc3a.png


The code example above limits us to make only 10 concurrent calls to the ‘populatePostsForGroup’ function. In this function, requests are made to Workplace Graph API, instead of just shooting thousands of requests to Workplace Graph API in a matter of seconds.

Furthermore, the concurrency control in bluebird is able continuously make calls at the specified rate of 10. For example, when two requests are done, the queued requests will send an additional two to Graph API.

In essence, the concurrency control balances between fast API calls and API rate limit errors.

(ii) Request Retry
Since hundreds of thousand API calls have to be made, there are bound to be failures in some of the calls. During our test phase, we encountered some of failures with minimal explanation. After recommendation from our Sponsor’s Tech team, we concluded that the best way to solve this error, is to just try sending the request again.

To do this, we made use of the requestretry library on NPM.

Ravenous - ftc3b.png


The library helped us to easily resend the request to whenever an error is returned in the response. From the above options set, the request will keep trying at a maximum attempt of 5 times. After the 5th attempt is exhausted, it will pause the retry for five seconds before continuing to try.

This helped us to overcome the unknown Workplace API errors and allowed us to continue retrieve data from Workplace reliably.


Technical complexity 4 - Real-Time Data (New)

With the new Webhook subscription feature offered by Workplace Graph API, it allows us to update our database data in real-time.

In order to listen to these events from Workplace, we had to set up our own webhooks. The webhooks were set up using Workplace’s interface.

The topics our webhooks are subscribed to are: Posts (New and Modified), Comments (New and Modified), Users (New and Deleted), Events (New)

Ravenous - ftc4a.png


Every time a new event occurs, for example a user just posted a new comment, Workplace will send us the payload to the subscribed webhook accordingly. We will then handle the events accordingly.


Quality of Product

Intermediate Deliverables

Topic of Interest Link
Project Management Schedule & Functionalities
Metrics
Risk Management
Change Management
Minutes
Project Documentation Technical Diagrams
Prototype
Persona & Scenario
Testing Bug Log
Internal Testing & Test Cases
UT1
UT2
UT3

Deployment

The deployment links are as follow:

We have 3 versions of Dashy for different purposes. Staging refers to application in the development phase and production refers to applications are that production ready. WOG Live instance is the one we deploy on WOG's server. Only these three applications have a user interface.


These are the backend servers that support our applications on Workplace@FB. They do not have a user interface.


Our Workplace@FB instance can be found here: https://psd-test.facebook.com/

- Please note that by logging into our Workplace@FB instance, you will have access to EvBot and FaBot.
- You will need to use the production link above to access Dashy.

You may login to our Workplace@FB instance using the credentials below. You may also login to Dashy using the same credentials
Email login id: wiki_user@test.sis.smu.edu.sg
Password: wiki_user123

Testing

Team Ravenous has conducted a total of 3 User Testings.
The first user testing placed emphasize on whether users know how to use our applications, whether they understand the flow of tasks and observe how they interact with our ChatBots.
The second user test is about the usability improvement that we have made since user testing 1.
The third user test is about the usability improvement that we have made since user testing 2.

The results of UT1 can be found here
The results of UT2 can be found here
The results of UT3 can be found here

Details of internal testing can be found in the deliverable table above.

Reflection

Team Reflection

This project was an exciting and enriching journey for Team Ravenous. Collectively, we not only gain new knowledge and skills about our roles, we also get to expand our professional network. Through this project, we get to meet and interact with PSD Technical Team, GovTech Team as well as business & IT professionals from 6 different government agencies. Through the interactions, we know why are certain processes carried out this way, why they are doing this and so on.

Individual Reflection

Ravenous - final learning 1.png