Difference between revisions of "IS480 Team wiki: 2017T1 Ravenous Final Wiki"
(15 intermediate revisions by the same user not shown) | |||
Line 40: | Line 40: | ||
To access EvBot & FaBot, login to Workplace@Facebook instance by clicking [https://psd-test.facebook.com/ here] | To access EvBot & FaBot, login to Workplace@Facebook instance by clicking [https://psd-test.facebook.com/ here] | ||
<br> | <br> | ||
− | To access Dashy, click [https://dashy-production.herokuapp.com/ here] | + | To access Dashy (our instance), click [https://dashy-production.herokuapp.com/ here]<br> |
+ | To access Dashy <b>(WOG's live instance)</b>, click [http://dashy.ap-southeast-1.elasticbeanstalk.com/ here] | ||
+ | <br><br> | ||
+ | To view <b>handover documentation</b>, click [https://www.dropbox.com/sh/adqxpkpylslowjv/AACCaJAXvl7KtpYZphDgtH7ma?dl=0 here] | ||
<br><br> | <br><br> | ||
For more information on how to access our applications, please view Quality of Product > Deployment Section | For more information on how to access our applications, please view Quality of Product > Deployment Section | ||
Line 243: | Line 246: | ||
| View word cloud chart | | View word cloud chart | ||
| No | | No | ||
− | | | + | | Delivered |
| 1.0 | | 1.0 | ||
| style="color: #008000;" | Completed | | style="color: #008000;" | Completed | ||
Line 310: | Line 313: | ||
|- | |- | ||
| Analytics Dashboard Module IV | | Analytics Dashboard Module IV | ||
− | | | + | | Filtering options based on user activity/profile |
| No | | No | ||
| Fully deployed and 100% Tested; On production server | | Fully deployed and 100% Tested; On production server | ||
Line 317: | Line 320: | ||
|- | |- | ||
| Analytics Dashboard Module IV | | Analytics Dashboard Module IV | ||
− | | | + | | Dashy additional metrics: Overview, posts/comments of previous day vs last week’s same day |
| No | | No | ||
| Fully deployed and 100% Tested; On production server | | Fully deployed and 100% Tested; On production server | ||
Line 324: | Line 327: | ||
|- | |- | ||
| Analytics Dashboard Module IV | | Analytics Dashboard Module IV | ||
− | | Dashy additional metrics: | + | | Dashy additional metrics: Group, Measure posts, comments and reactions in a given period of time and see the most popular days and times that members engage. (advanced filtering) |
| No | | No | ||
| Fully deployed and 100% Tested; On production server | | Fully deployed and 100% Tested; On production server | ||
Line 331: | Line 334: | ||
|- | |- | ||
| Analytics Dashboard Module IV | | Analytics Dashboard Module IV | ||
− | | Dashy additional metrics: | + | | Dashy additional metrics: Most seen posts by number of seen per group |
− | | | + | | Yes |
| Fully deployed and 100% Tested; On production server | | Fully deployed and 100% Tested; On production server | ||
| 1.0 | | 1.0 | ||
Line 338: | Line 341: | ||
|- | |- | ||
| Analytics Dashboard Module IV | | Analytics Dashboard Module IV | ||
− | | Dashy additional metrics: | + | | Dashy additional metrics: Top contributor by agency |
| Yes | | Yes | ||
| Fully deployed and 100% Tested; On production server | | Fully deployed and 100% Tested; On production server | ||
Line 353: | Line 356: | ||
| FMS Module II | | FMS Module II | ||
| Remind and extend a booking that has begun | | Remind and extend a booking that has begun | ||
− | | | + | | No |
| Fully deployed and 100% Tested; On production server | | Fully deployed and 100% Tested; On production server | ||
| 1.0 | | 1.0 | ||
Line 505: | Line 508: | ||
The chart shows the number of completed tasks against number of assigned tasks for all our iterations so far. The bigger the circle, the higher the percentage of task metrics<br><br> | The chart shows the number of completed tasks against number of assigned tasks for all our iterations so far. The bigger the circle, the higher the percentage of task metrics<br><br> | ||
− | [[File:Ravenous - | + | [[File:Ravenous - Final TM Summary.png|900px|center]]<br> |
<br> | <br> | ||
Line 513: | Line 516: | ||
[[File:Ravenous - Total Bug Score.png|720px]] | [[File:Ravenous - Total Bug Score.png|720px]] | ||
[[File:Ravenous - Total Bug Num.png|720px]] | [[File:Ravenous - Total Bug Num.png|720px]] | ||
− | |||
</center> | </center> | ||
Line 533: | Line 535: | ||
| style="background-color:#28B463; color: #000000; font-weight: bold;" | Action | | style="background-color:#28B463; color: #000000; font-weight: bold;" | Action | ||
|- | |- | ||
− | | | + | | 8 |
− | | | + | | Technical Risk |
− | | | + | | Our project is heavily dependent on Facebook API that we have no control over. |
| High | | High | ||
| High | | High | ||
| A | | A | ||
| Mitigation | | Mitigation | ||
− | | | + | | Our team is part of the Multi-Company Group with PSD, MOE and Facebook. We are in touch with Rohan who we can approach if we have any problem with the Workplace instance |
|- | |- | ||
− | | | + | | 11 |
| Technical Risk | | Technical Risk | ||
− | | | + | | Team is given a relatively small server to deploy on the government instance. Thus, code has to be optimized in order the application to work. |
| High | | High | ||
| High | | High | ||
| A | | A | ||
| Mitigation | | Mitigation | ||
− | | | + | | Developers to learn and understand how to optimize code through online resources |
|- | |- | ||
|} | |} | ||
<br> | <br> | ||
− | From the period of Mid Term (9/10/2017) to Final presentation, we have a total of | + | From the period of Mid Term (9/10/2017) to Final presentation, we have a total of 3 change requests. |
<br> | <br> | ||
Line 586: | Line 588: | ||
| High, It allows Workplace Managers to quickly know how many posts are being viewed by how many their users in their agency, hence this supplements them in knowing activity rate in their agency | | High, It allows Workplace Managers to quickly know how many posts are being viewed by how many their users in their agency, hence this supplements them in knowing activity rate in their agency | ||
| 4, High | | 4, High | ||
+ | | Accept change request and team discuss the most appropriate iteration to implement the change request | ||
+ | | Closed | ||
+ | |- | ||
+ | | 40 | ||
+ | | Dashy, Sponsor | ||
+ | | Deploy on WOG AWS server | ||
+ | | 3 | ||
+ | | 3 | ||
+ | | High, Allow the team to deploy for live usage | ||
+ | | 6, High | ||
| Accept change request and team discuss the most appropriate iteration to implement the change request | | Accept change request and team discuss the most appropriate iteration to implement the change request | ||
| Closed | | Closed | ||
Line 598: | Line 610: | ||
The initial task of populating our database was a challenge as we constantly reached our memory limit while trying to retrieve and store the data from Facebook Workplace, into our database. Upgrading the instance to a better one was an option but we decided to optimize the memory usage of our code first.<br><br> | The initial task of populating our database was a challenge as we constantly reached our memory limit while trying to retrieve and store the data from Facebook Workplace, into our database. Upgrading the instance to a better one was an option but we decided to optimize the memory usage of our code first.<br><br> | ||
− | <b>Bluebird Promises</b><br> | + | <b>(i) Bluebird Promises</b><br> |
During our mid-term presentation, we mentioned that we used the Native Promise library to approach our asynchronous issues. Bluebird promises work the same say as Node’s native Promise however, it is less memory intensive. The benchmark results can be found on bluebird’s documentation but we decided to run some tests ourselves. Here is the score from the tests we ran using Node’s process.memoryUsage() function:<br><br> | During our mid-term presentation, we mentioned that we used the Native Promise library to approach our asynchronous issues. Bluebird promises work the same say as Node’s native Promise however, it is less memory intensive. The benchmark results can be found on bluebird’s documentation but we decided to run some tests ourselves. Here is the score from the tests we ran using Node’s process.memoryUsage() function:<br><br> | ||
Line 607: | Line 619: | ||
− | <b>Node Garbage Collector</b><br> | + | <b>(ii) Node Garbage Collector</b><br> |
To free up memory within the code, we had to make use of Node’s garbage collector which runs in the background. The garbage collector frees up memory when the objects have no reference to any other objects. In order to reduce the memory usage of our code, we explicitly force node’s garbage collector to remove objects that are unused by assigning a null value to the object before ending the function.<br><br> | To free up memory within the code, we had to make use of Node’s garbage collector which runs in the background. The garbage collector frees up memory when the objects have no reference to any other objects. In order to reduce the memory usage of our code, we explicitly force node’s garbage collector to remove objects that are unused by assigning a null value to the object before ending the function.<br><br> | ||
Line 614: | Line 626: | ||
− | <b>Mongo Cursor and Buffer (Managing Memory Usage)</b><br> | + | <b>(iii) Mongo Cursor and Buffer (Managing Memory Usage)</b><br> |
The usage of cursor is similar to streaming data. Instead of fetching all the data from the database and consuming it all at once, cursor streams the data to our server. <br><br> | The usage of cursor is similar to streaming data. Instead of fetching all the data from the database and consuming it all at once, cursor streams the data to our server. <br><br> | ||
Line 622: | Line 634: | ||
− | <b>EcmaScript6 (ES6)’s let and CONST declaration</b><br> | + | <b>(iv) EcmaScript6 (ES6)’s let and CONST declaration</b><br> |
We realized that with the new declarations of let and CONST, it uses less memory than the old global declaration ‘var’. We refactored our codes to make use of only ES6’s let and CONST declaration.<br><br><br> | We realized that with the new declarations of let and CONST, it uses less memory than the old global declaration ‘var’. We refactored our codes to make use of only ES6’s let and CONST declaration.<br><br><br> | ||
Line 628: | Line 640: | ||
Since we had to process approximately a million data, our codes have to run quickly. One of the reasons why Node.js was chosen as the language for this task, is for this very purpose.<br><br> | Since we had to process approximately a million data, our codes have to run quickly. One of the reasons why Node.js was chosen as the language for this task, is for this very purpose.<br><br> | ||
− | <b>Async Parallel</b><br> | + | <b>(i) Async Parallel</b><br> |
With the new version of Node.js (v7.6 and above), came the native async library. We took advantage of this library by making asynchronous queries to our database so that database queries can run in parallel.<br><br> | With the new version of Node.js (v7.6 and above), came the native async library. We took advantage of this library by making asynchronous queries to our database so that database queries can run in parallel.<br><br> | ||
Line 637: | Line 649: | ||
− | <b>Native Mongo Queries</b><br> | + | <b>(ii) Native Mongo Queries</b><br> |
Although the library that we used to interact with MongoDB, named Mongoose, has certain functions to aid developers in using a MongoDB, it is limited. The general find() or findOneAndUpdate() queries are insufficient for more complex queries. Therefore, an easy but inefficient solution is to find all records, and sift out the records we need by iterating through the records. This is extremely inefficient as compared to using the native queries.<br><br> | Although the library that we used to interact with MongoDB, named Mongoose, has certain functions to aid developers in using a MongoDB, it is limited. The general find() or findOneAndUpdate() queries are insufficient for more complex queries. Therefore, an easy but inefficient solution is to find all records, and sift out the records we need by iterating through the records. This is extremely inefficient as compared to using the native queries.<br><br> | ||
Line 652: | Line 664: | ||
This was an issue initially as we were met with lots of rate limiting and throttle error that were set in Workplace’s Graph API. This limits us to about 200 calls per hour per user in aggregate. We adopted some techniques to overcome this issue: <br><br> | This was an issue initially as we were met with lots of rate limiting and throttle error that were set in Workplace’s Graph API. This limits us to about 200 calls per hour per user in aggregate. We adopted some techniques to overcome this issue: <br><br> | ||
− | <b>Bluebird Concurrency Control (Promise.Map)</b><br> | + | <b>(i) Bluebird Concurrency Control (Promise.Map)</b><br> |
Bluebird’s Promise.Map allowed us to manage the number of requests we made to Workplace API at any moment.<br><br> | Bluebird’s Promise.Map allowed us to manage the number of requests we made to Workplace API at any moment.<br><br> | ||
Line 663: | Line 675: | ||
In essence, the concurrency control balances between fast API calls and API rate limit errors. <br><br> | In essence, the concurrency control balances between fast API calls and API rate limit errors. <br><br> | ||
− | <b>Request Retry</b><br> | + | <b>(ii) Request Retry</b><br> |
Since hundreds of thousand API calls have to be made, there are bound to be failures in some of the calls. During our test phase, we encountered some of failures with minimal explanation. After recommendation from our Sponsor’s Tech team, we concluded that the best way to solve this error, is to just try sending the request again. <br><br> | Since hundreds of thousand API calls have to be made, there are bound to be failures in some of the calls. During our test phase, we encountered some of failures with minimal explanation. After recommendation from our Sponsor’s Tech team, we concluded that the best way to solve this error, is to just try sending the request again. <br><br> | ||
Line 684: | Line 696: | ||
Every time a new event occurs, for example a user just posted a new comment, Workplace will send us the payload to the subscribed webhook accordingly. We will then handle the events accordingly.<br><br><br> | Every time a new event occurs, for example a user just posted a new comment, Workplace will send us the payload to the subscribed webhook accordingly. We will then handle the events accordingly.<br><br><br> | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
== Quality of Product == | == Quality of Product == | ||
Line 742: | Line 743: | ||
The deployment links are as follow: | The deployment links are as follow: | ||
− | We have | + | We have 3 versions of Dashy for different purposes. Staging refers to application in the development phase and production refers to applications are that production ready. WOG Live instance is the one we deploy on WOG's server. Only these three applications have a user interface.<br> |
<ul> | <ul> | ||
<li>[https://dashy-staging.herokuapp.com Dashy (Analytics Dashboard, staging)]</li> | <li>[https://dashy-staging.herokuapp.com Dashy (Analytics Dashboard, staging)]</li> | ||
<li>[https://dashy-production.herokuapp.com Dashy (Analytics Dashboard, production)]</li> | <li>[https://dashy-production.herokuapp.com Dashy (Analytics Dashboard, production)]</li> | ||
+ | <li>[http://dashy.ap-southeast-1.elasticbeanstalk.com/ (Analytics Dashboard, <b>WOG Live Instance</b>)]</li> | ||
</ul> | </ul> | ||
<br> | <br> | ||
Line 782: | Line 784: | ||
=== Individual Reflection === | === Individual Reflection === | ||
+ | |||
+ | [[File: Ravenous - final learning 1.png|720px|center]] | ||
<!--/Content--> | <!--/Content--> |
Latest revision as of 14:53, 22 November 2017
Contents
Project Progress Summary
As of 14th November 2017, we have completed 41% of iteration 9 and this is our last iteration.
To view final slides, click here
To access EvBot & FaBot, login to Workplace@Facebook instance by clicking here
To access Dashy (our instance), click here
To access Dashy (WOG's live instance), click here
To view handover documentation, click here
For more information on how to access our applications, please view Quality of Product > Deployment Section
Project Highlights & Snapshots
Project Management
Project Status
The three diagrams below show the progress of our project, a high level overview of completion of project modules and a high level overview of change in scope to our project modules respectively.
Scope: Core
Module | Task | New Feature? | Status | Confidence level | Comments |
Event ChatBot (Organiser) | Register an event | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Event ChatBot (Organiser) | Remove event | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Event ChatBot (Organiser) | Close event | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Event ChatBot (Organiser) | Add & Remove questions | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Event ChatBot (Organiser) | Send Survey Questions | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Event ChatBot (Organiser) | Send Survey Reminder to participants | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Event ChatBot (Organiser) | View Survey Questions | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Event ChatBot (Organiser) | View Event Snapshot | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Event ChatBot (Organiser) | About EvBot | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Event ChatBot (Participants) | Check-in Event Attendance | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Event ChatBot (Participants) | View Surveys | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Event ChatBot (Participants) | About EvBot | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Facility Booking ChatBot & FMS Module I | Search available facilities | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Facility Booking ChatBot & FMS Module I | Book available facilities | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Facility Booking ChatBot & FMS Module I | View booked facilities | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Facility Booking ChatBot & FMS Module I | Cancel booked facilities | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Analytics Dashboard Module I | View claim rate vs active rate for all agencies and specific agency chart | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Analytics Dashboard Module I | View aggregated engagement scores of groups within an agency | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Analytics Dashboard Module I | View breakdown of engagement scores of groups within an agency | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Analytics Dashboard Module I | View Pc vs Mobile Graph | No | Delivered | 1.0 | Completed |
Analytics Dashboard Module II | View group privacy setting across specific group charts | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Analytics Dashboard Module II | View group activity charts | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Analytics Dashboard Module II | View interaction analysis of specific agency | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Analytics Dashboard Module III | View number of active users charts | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Analytics Dashboard Module III | View content on workplace chart | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Analytics Dashboard Module III | View word cloud chart | No | Delivered | 1.0 | Completed |
Analytics Dashboard Module III | View post-time & comment-time chart | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Authentication & Security Module | Register account on Dashy OTP Bot | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Authentication & Security Module | OTP Bot | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Authentication & Security Module | Login/Logout on Dashy | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Authentication & Security Module | Forget password | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Authentication & Security Module | Secure all API Calls with JWT Token | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Scope: Secondary
Module | Task | New Feature? | Status | Confidence level | Comments |
Analytics Dashboard Module IV | Export dashboard to CSV File | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Analytics Dashboard Module IV | Filtering options based on user activity/profile | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Analytics Dashboard Module IV | Dashy additional metrics: Overview, posts/comments of previous day vs last week’s same day | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Analytics Dashboard Module IV | Dashy additional metrics: Group, Measure posts, comments and reactions in a given period of time and see the most popular days and times that members engage. (advanced filtering) | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Analytics Dashboard Module IV | Dashy additional metrics: Most seen posts by number of seen per group | Yes | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Analytics Dashboard Module IV | Dashy additional metrics: Top contributor by agency | Yes | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
FMS Module II | Release a booking that has begun | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
FMS Module II | Remind and extend a booking that has begun | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
FMS Module II | Modify make, delete, view and search facilities to replicate GovTech's System | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Facility Booking ChatBot Module II | Advanced NLP | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Facility Booking ChatBot Module II | View room host details on workplace and have the ability to contact host | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Facility Booking ChatBot Module II | Re-prompt user with other room to book in case of clash in booking | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Analytics Dashboard and Bots Integration Module | Expose EvBot Event report | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Analytics Dashboard and Bots Integration Module | Dashy displays event statistics for event organisers | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Analytics Dashboard and Bots Integration Module | Expose FaBot usage metrics | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Analytics Dashboard and Bots Integration Module | Expose EvBot usage metrics | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Analytics Dashboard and Bots Integration Module | View FaBot usage metrics for workplace managers | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Analytics Dashboard and Bots Integration Module | View EvBot usage metrics for workplace managers | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Event ChatBot (Organisers) II | Display attendee that didn't turn up | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Event ChatBot (Organisers) II | Organisers message broadcast | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Event ChatBot (Organisers) II | Organisers trigger prompt for participants to check in | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Event ChatBot (Organisers) II | Display attendees yet to complete survey | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Analytics Dashboard Mobile Responsiveness | Optimize web pages | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Scope: Good to have
Module | Task | New Feature? | Status | Confidence level | Comments |
Database Optimization Module | Optimize syncing of new data from Workplace@FB with Crunchy's database | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Event ChatBot (Organisers) III | Export Event Report as CSV | No | Fully deployed and 100% Tested; On production server | 1.0 | Completed |
Project Schedule
This section shows a low level overview of our tasks in each iteration before midterm and current point (final)
Planned
Actual
Project Metrics
The chart shows the number of completed tasks against number of assigned tasks for all our iterations so far. The bigger the circle, the higher the percentage of task metrics
The charts below show Total Bug Score per iteration, Total Number of Bugs per iteration and Average Bug Score per Bug per iteration respectively.
Project Risks
From the period of Mid Term (9/10/2017) to Final presentation, we would like to highlight these two risks as area of concerns.
S/N | Risk Type | Risk Description | Likelihood | Impact Level | Risk Grade | Strategy Adopted | Action |
8 | Technical Risk | Our project is heavily dependent on Facebook API that we have no control over. | High | High | A | Mitigation | Our team is part of the Multi-Company Group with PSD, MOE and Facebook. We are in touch with Rohan who we can approach if we have any problem with the Workplace instance |
11 | Technical Risk | Team is given a relatively small server to deploy on the government instance. Thus, code has to be optimized in order the application to work. | High | High | A | Mitigation | Developers to learn and understand how to optimize code through online resources |
From the period of Mid Term (9/10/2017) to Final presentation, we have a total of 3 change requests.
S/N | Application, Requested by | Change description | Impact on Schedule | Technical Complexity | Business Value/Reason for Request | Score | Action Taken | Status of Request |
38 | Dashy, Sponsor | Top Contributor in Agency | 2 | 2 | High, Allows Workplace Managers to quickly know who are the top contributor in their agency. Currently, workplace default dashboard only allow them to view top contributor of whole of government | 4, High | Accept change request and team discuss the most appropriate iteration to implement the change request | Closed |
39 | Dashy, Sponsor | Average seen rate of all groups in agency | 2 | 2 | High, It allows Workplace Managers to quickly know how many posts are being viewed by how many their users in their agency, hence this supplements them in knowing activity rate in their agency | 4, High | Accept change request and team discuss the most appropriate iteration to implement the change request | Closed |
40 | Dashy, Sponsor | Deploy on WOG AWS server | 3 | 3 | High, Allow the team to deploy for live usage | 6, High | Accept change request and team discuss the most appropriate iteration to implement the change request | Closed |
Technical Complexity
Technical complexity 1 - Deployment Server Memory
We had to work with an Amazon EC2 T2.micro instance to host our analytics processing server. The T2.Micro instance only provides 1gb memory and the processing server is expected to load an expected volume of 1 million data from the Whole Of Government Workplace instance.
The initial task of populating our database was a challenge as we constantly reached our memory limit while trying to retrieve and store the data from Facebook Workplace, into our database. Upgrading the instance to a better one was an option but we decided to optimize the memory usage of our code first.
(i) Bluebird Promises
During our mid-term presentation, we mentioned that we used the Native Promise library to approach our asynchronous issues. Bluebird promises work the same say as Node’s native Promise however, it is less memory intensive. The benchmark results can be found on bluebird’s documentation but we decided to run some tests ourselves. Here is the score from the tests we ran using Node’s process.memoryUsage() function:
Through our results, the heapUsed for the bluebird library is lesser than the native Promise, confirming the benchmark scores found online. Thus, we refactored the promises that were used by memory intensive functions to make use of the bluebird library
(ii) Node Garbage Collector
To free up memory within the code, we had to make use of Node’s garbage collector which runs in the background. The garbage collector frees up memory when the objects have no reference to any other objects. In order to reduce the memory usage of our code, we explicitly force node’s garbage collector to remove objects that are unused by assigning a null value to the object before ending the function.
(iii) Mongo Cursor and Buffer (Managing Memory Usage)
The usage of cursor is similar to streaming data. Instead of fetching all the data from the database and consuming it all at once, cursor streams the data to our server.
After streaming the data, we manually implemented our own Buffer to batch process the records that were streamed. With the combination of Mongo’s cursor and our buffer, it helped us to manage the memory usage of our application.
(iv) EcmaScript6 (ES6)’s let and CONST declaration
We realized that with the new declarations of let and CONST, it uses less memory than the old global declaration ‘var’. We refactored our codes to make use of only ES6’s let and CONST declaration.
Technical complexity 2 - Code Optimization to improve Runtime
Since we had to process approximately a million data, our codes have to run quickly. One of the reasons why Node.js was chosen as the language for this task, is for this very purpose.
(i) Async Parallel
With the new version of Node.js (v7.6 and above), came the native async library. We took advantage of this library by making asynchronous queries to our database so that database queries can run in parallel.
In the simple example above, two functions (calls) are made to the database. However, with the async.parallel function, they are made at the same time. Only when both queries have finished executing, will the code continue with the results of both queries. This helps to improve the runtime processing speed when widely used across the code.
(ii) Native Mongo Queries
Although the library that we used to interact with MongoDB, named Mongoose, has certain functions to aid developers in using a MongoDB, it is limited. The general find() or findOneAndUpdate() queries are insufficient for more complex queries. Therefore, an easy but inefficient solution is to find all records, and sift out the records we need by iterating through the records. This is extremely inefficient as compared to using the native queries.
Therefore, we had to learn Mongo Queries to improve the speed of our application. We made use of Mongoose’s aggregate function include in Mongo’s operators to perform some of the more complex queries.
Some of the operators that we used include:$match, $group, $project, $sort, $count
Technical complexity 3 - Making Large Number of API Calls to Workplace Graph API
As of 1st November 2017, there are a little more than 150,000 public servants on the Whole of Government Workplace instance. Retrieving every detail on these accounts would result in a few hundreds of thousand requests to Workplace’s API.
This was an issue initially as we were met with lots of rate limiting and throttle error that were set in Workplace’s Graph API. This limits us to about 200 calls per hour per user in aggregate. We adopted some techniques to overcome this issue:
(i) Bluebird Concurrency Control (Promise.Map)
Bluebird’s Promise.Map allowed us to manage the number of requests we made to Workplace API at any moment.
The code example above limits us to make only 10 concurrent calls to the ‘populatePostsForGroup’ function. In this function, requests are made to Workplace Graph API, instead of just shooting thousands of requests to Workplace Graph API in a matter of seconds.
Furthermore, the concurrency control in bluebird is able continuously make calls at the specified rate of 10. For example, when two requests are done, the queued requests will send an additional two to Graph API.
In essence, the concurrency control balances between fast API calls and API rate limit errors.
(ii) Request Retry
Since hundreds of thousand API calls have to be made, there are bound to be failures in some of the calls. During our test phase, we encountered some of failures with minimal explanation. After recommendation from our Sponsor’s Tech team, we concluded that the best way to solve this error, is to just try sending the request again.
To do this, we made use of the requestretry library on NPM.
The library helped us to easily resend the request to whenever an error is returned in the response. From the above options set, the request will keep trying at a maximum attempt of 5 times. After the 5th attempt is exhausted, it will pause the retry for five seconds before continuing to try.
This helped us to overcome the unknown Workplace API errors and allowed us to continue retrieve data from Workplace reliably.
Technical complexity 4 - Real-Time Data (New)
With the new Webhook subscription feature offered by Workplace Graph API, it allows us to update our database data in real-time.
In order to listen to these events from Workplace, we had to set up our own webhooks. The webhooks were set up using Workplace’s interface.
The topics our webhooks are subscribed to are: Posts (New and Modified), Comments (New and Modified), Users (New and Deleted), Events (New)
Every time a new event occurs, for example a user just posted a new comment, Workplace will send us the payload to the subscribed webhook accordingly. We will then handle the events accordingly.
Quality of Product
Intermediate Deliverables
Topic of Interest | Link |
---|---|
Project Management | Schedule & Functionalities |
Metrics | |
Risk Management | |
Change Management | |
Minutes | |
Project Documentation | Technical Diagrams |
Prototype | |
Persona & Scenario | |
Testing | Bug Log |
Internal Testing & Test Cases | |
UT1 | |
UT2 | |
UT3 |
Deployment
The deployment links are as follow:
We have 3 versions of Dashy for different purposes. Staging refers to application in the development phase and production refers to applications are that production ready. WOG Live instance is the one we deploy on WOG's server. Only these three applications have a user interface.
- Dashy (Analytics Dashboard, staging)
- Dashy (Analytics Dashboard, production)
- (Analytics Dashboard, WOG Live Instance)
These are the backend servers that support our applications on Workplace@FB. They do not have a user interface.
- Crunchy (Data crunching for Dashy)
- EvBot (Event ChatBot, Available on our workplace@FB instance)
- EvBot (Facility ChatBot, Available on our workplace@FB instance)
- OttoBot (OTP ChatBot, Available on our workplace@FB instance)
- FMS (Facility Management System)
Our Workplace@FB instance can be found here: https://psd-test.facebook.com/
- Please note that by logging into our Workplace@FB instance, you will have access to EvBot and FaBot.
- You will need to use the production link above to access Dashy.
You may login to our Workplace@FB instance using the credentials below. You may also login to Dashy using the same credentials
Email login id: wiki_user@test.sis.smu.edu.sg
Password: wiki_user123
Testing
Team Ravenous has conducted a total of 3 User Testings.
The first user testing placed emphasize on whether users know how to use our applications, whether they understand the flow of tasks and observe how they interact with our ChatBots.
The second user test is about the usability improvement that we have made since user testing 1.
The third user test is about the usability improvement that we have made since user testing 2.
The results of UT1 can be found here
The results of UT2 can be found here
The results of UT3 can be found here
Details of internal testing can be found in the deliverable table above.
Reflection
Team Reflection
This project was an exciting and enriching journey for Team Ravenous. Collectively, we not only gain new knowledge and skills about our roles, we also get to expand our professional network. Through this project, we get to meet and interact with PSD Technical Team, GovTech Team as well as business & IT professionals from 6 different government agencies. Through the interactions, we know why are certain processes carried out this way, why they are doing this and so on.
Individual Reflection