Difference between revisions of "GeViz"

From Visual Analytics for Business Intelligence
Jump to navigation Jump to search
Line 175: Line 175:
 
* The interactive filters are year, ministry, agency and category.  
 
* The interactive filters are year, ministry, agency and category.  
 
|-
 
|-
| <p><center>'''Word on Goods and Services Procured''' </center></p>
+
| <p><center>'''Word Cloud on Goods and Services Procured''' </center></p>
[[File:Word cloud storyboard.png|400px|frameless|center]]
+
[[File:Storyboard wordcloud.png|400px|frameless|center]]
<p><center>'''Source''':https://www.r-graph-gallery.com/wordcloud/</center></p>
+
<p><center>'''Source''':http://www.sthda.com/english/wiki/text-mining-and-word-cloud-fundamentals-in-r-5-simple-steps-you-should-know</center></p>
 
  ||  
 
  ||  
 
* Word cloud to show an overview of the tender description for the selected agency and selected category.  
 
* Word cloud to show an overview of the tender description for the selected agency and selected category.  

Revision as of 14:57, 25 November 2018

Geviz.png


Team

 

Proposal

 

Poster

 

Application

 

Research Paper


PROBLEM & MOTIVATION

GeBIZ is a Singapore Government’s one-stop e-procurement portal which facilitates tender activities between Singapore government and local and overseas suppliers. Currently, there is no available tool to aid the public and ministries to understand and gain insights on the procurement made by the government under each ministry. Hence, we are motivated to create an interactive visualisation tool on government's procurement spending to allow the public and ministries to identify spending patterns and gain insights into procurement spending under each ministry.

OBJECTIVES

In this project, we are creating a visualisation that is able to show the following:

  • Gain an overview of procurement spending made by each ministry and agency
  • Identify the relationships between ministries, agencies and suppliers
  • Identify what are the goods and services procured by ministries and agencies under each category

SELECTED DATASETS

The following datasets will be used for analysis , as elaborated below:

Dataset/Source Data Attributes Rationale of Usage
Government Procurement Data (https://data.gov.sg/dataset/government-procurement)
  • Tender No
  • Agency
  • Tender Description
  • Award Date
  • Tender Detail Status
  • Supplier Name
  • Awarded Amount
To gain information on government procurement such as tender description, amount and supplier information
Ministry and Agencies List
  • Ministry
  • Agency
We will be looking through the Singapore Government Directory (https://www.gov.sg/sgdi/ministries) to categorise the agencies into their respective ministries. This will allow us to visualise the procurement spending on a ministry level.


APPROACH

Approach 2.png

Exploratory Data Analytics
We used Tableau to perform EDA to better understand our dataset and to aid us in the conceptualization of our story board.

Data Cleaning and Feature Creation
We used Excel and Python to create a new column showing the Ministry that each agency belongs to by merging with data obtained from the Singapore Government Directory.

Text Classification using Support Vector Classifier (SVC)
One of the key challenges of working with the provided procurement dataset is the absence of categorization of each procurement transaction. Instead of labelling manually, we applied machine learning to classify the tender descriptions into different categories. We firstly scraped the procurement descriptions and categories from GeBiz website using Selenium and BeautifulSoup libraries in Python to be used as the training and validation dataset in our Support Vector Classifier model. We were able to achieve 90% for training accuracy before performing the categorization prediction.

Government Procurement Dataset after Text Classification

Dataset/Source Data Attributes
Government Procurement Data
  • Tender No
  • Agency
  • Tender Description
  • Award Date
  • Tender Detail Status
  • Supplier Name
  • Awarded Amount
  • Category
  • Sub Category


Visualization in R
The web application will be built in R and deployed to Shinyapps.io

BACKGROUND SURVEY OF RELATED WORKS

Some of these visualizations that we draw inspiration from, are as follows:

Reference of Other Interactive Visualization What We Can Learn

Title : Pareto Analysis of Suppliers

Pareto Analysis Reference.png

Source: https://goo.gl/P9RjHk

  • The use of time series chart allows users to view the rise and fall of prices and prevents users from getting overwhelmed by too much cluttered data as compared to using bar charts.

Title : Word Cloud on Procurement Details

Word cloud reference.png

Source: https://linpack-for-tableau.com/data-visualizations/tableau-dashboards/procurement-dashboard/procurement-cockpit/

  • Based on the size of the words, we can identify the keywords quickly. We can learn that by making frequency used keywords stand out can prevent us from overlooking it if it is presented in a tabular format.

Title : Breakdown on UK Government Spending

Treemap ref 2.png

Source: http://www.nickmalleson.co.uk/2014/02/uk-government-spending-treemap.html

  • What we can learn on this project is the use of cross filtering to provide an interactive filtering of data.
  • The charts on the map will zoom into the details based on the user’s filter preference.

Title : Team Budget Breadown

Sankey diagram reference.png

Source:https://acquireprocure.com/spend-analysis-visualisation/3-reasons-procurement-professionals-use-sankey-diagrams

  • Area shading map allows us to quickly see which area has more HDB flats of which type.
  • We can also understand that there are new areas and drawing of boundary changes across the years.
  • There is also the information at a glance at the side for the users to view.

Title : Analyzing Involved Authorities, Tenders and Companies

Graph network references.png

Source: https://linkurio.us/blog/exploring-e1-3-trillion-in-public-contracts-with-graph-visualization/#!prettyPhotooard

  • Data is sorted in descending order, making sure that the viewer will be able to have quick inference.

BRAINSTORMING SESSIONS

Brainstorm2.png

During our brainstorming session, we came out a list of visualization which are able to achieve our objectives and eventually shortlisted 4 visualization - Tree map, Network Graph, Sankey Graph and Word Cloud. After rounds of refinement and consultations with our Professor, the image above is the final draft for our visualization.

PROPOSED STORYBOARD

Our group has proposed the following storyboard in the use of our visual application:

Proposed Layout What We Can Analyse

Overview of Procurement Spending

Storyboard treemap.png

Source:https://www.r-graph-gallery.com/treemap/

  • Treemap to show the spending breakdown for each category of all agencies under the selected ministry.
  • Theinteractive filters are year and ministry.

Relationship between Ministry and Suppliers

Storyboard network.png

Source: https://cran.r-project.org/web/packages/visNetwork/vignettes/Introduction-to-visNetwork.html

  • Network diagram to show the relationship of agencies and suppliers of the selected ministry.
  • The interactive filters are year, ministry and range filter which allows the user to filter the suppliers based on the procurement amount.

Cashflow from Agency to Suppliers

Storyboard sankey.png

Source: https://www.r-graph-gallery.com/sankey-diagram/

  • Sankey diagram to show the cash flow between selected agency and suppliers for the selected category.
  • The interactive filters are year, ministry, agency and category.

Word Cloud on Goods and Services Procured

Storyboard wordcloud.png

Source:http://www.sthda.com/english/wiki/text-mining-and-word-cloud-fundamentals-in-r-5-simple-steps-you-should-know

  • Word cloud to show an overview of the tender description for the selected agency and selected category.
  • The filters are year, ministry, agency and category.
  • We added a searchable table below the word cloud to allow the user to search for keywords and view the exact tender description

TOOLS & TECHNOLOGIES

Tools and technologies

Tools used.png


Data Architecture

Data architecture 2.png


KEY CHALLENGES

The following are some of the key technical challenges that we may face throughout the course of the project:

Key Challenges Mitigation Plan
Unfamiliarity with R and Rshiny Libraries
  • Attend R Shiny Workshop
  • Independent learning via online resources such as Datacamp
  • Ask team mates for help
Unfamiliarity with Libraries for Machine Learning and Web Crawling
  • Clean, transform and analyse data together
  • Independent learning via online resources
Data Cleaning and Transformation
  • Need to crawl data on website to obtain training data for text classification
  • Clean, transform the data together


TIMELINE

Timeline 2.png

REFERENCES

COMMENTS

Feel free to leave us some comments so that we can improve! We dont bite :)

No. Name Date Comments
1. Insert your name here Insert date here Insert comment here
2. Insert your name here Insert date here Insert comment here
3. Insert your name here Insert date here Insert comment here