Difference between revisions of "GeViz"
Mechong.2015 (talk | contribs) |
Mechong.2015 (talk | contribs) |
||
(43 intermediate revisions by the same user not shown) | |||
Line 22: | Line 22: | ||
| style="padding:0.2em; font-size:100%; background-color:#143c67; border-bottom:0px solid #3D9DD7; text-align:center; color:#F5F5F5" width="10%" | | | style="padding:0.2em; font-size:100%; background-color:#143c67; border-bottom:0px solid #3D9DD7; text-align:center; color:#F5F5F5" width="10%" | | ||
[[1718t1is428T7_Research Paper|<font color="#F5F5F5" size=3 face="Lato">Research Paper</font>]] | [[1718t1is428T7_Research Paper|<font color="#F5F5F5" size=3 face="Lato">Research Paper</font>]] | ||
+ | |||
+ | |style="background:none;" width="1%" | | ||
+ | | style="padding:0.2em; font-size:100%; background-color:#143c67; border-bottom:0px solid #3D9DD7; text-align:center; color:#F5F5F5" width="10%" | | ||
+ | [[Project_Groups|<font color="#F5F5F5" size=3 face="Lato">Back to Project Groups</font>]] | ||
|} | |} | ||
</div> | </div> | ||
Line 36: | Line 40: | ||
* Identify the relationships between ministries, agencies and suppliers | * Identify the relationships between ministries, agencies and suppliers | ||
* Identify what are the goods and services procured by ministries and agencies under each category | * Identify what are the goods and services procured by ministries and agencies under each category | ||
− | |||
==<div style="background:#143c67; padding: 15px; font-weight: bold; line-height: 0.3em;letter-spacing:0.5em;font-size:20px"><font color=#fbfcfd face="Lato"><center>SELECTED DATASETS</center></font></div>== | ==<div style="background:#143c67; padding: 15px; font-weight: bold; line-height: 0.3em;letter-spacing:0.5em;font-size:20px"><font color=#fbfcfd face="Lato"><center>SELECTED DATASETS</center></font></div>== | ||
− | + | The following datasets will be used for analysis , as elaborated below: | |
{| class="wikitable" | {| class="wikitable" | ||
|- | |- | ||
Line 65: | Line 68: | ||
<br/> | <br/> | ||
− | ===Text Classification using Support Vector Classifier (SVC) | + | ==<div style="background:#143c67; padding: 15px; font-weight: bold; line-height: 0.3em;letter-spacing:0.5em;font-size:20px"><font color=#fbfcfd face="Lato"><center>APPROACH</center></font></div>== |
+ | [[File:Approach 2.png|800px|frameless|center]] | ||
+ | <b> Exploratory Data Analytics </b></br> | ||
+ | We used Tableau to perform EDA to better understand our dataset and to aid us in the conceptualization of our story board. | ||
+ | <br/> | ||
+ | |||
+ | <b> Data Cleaning and Feature Creation </b></br> | ||
+ | We used Excel and Python to create a new column showing the Ministry that each agency belongs to by merging with data obtained from the Singapore Government Directory. | ||
+ | <br/> | ||
+ | |||
+ | <b> Text Classification using Support Vector Classifier (SVC) </b></br> | ||
One of the key challenges of working with the provided procurement dataset is the absence of categorization of each procurement transaction. Instead of labelling manually, we applied <b>machine learning</b> to classify the tender descriptions into different categories. We firstly scraped the procurement descriptions and categories from GeBiz website using <b> Selenium</b> and <b> BeautifulSoup </b> libraries in Python to be used as the training and validation dataset in our Support Vector Classifier model. We were able to achieve <b> 90% for training accuracy </b> before performing the categorization prediction. <br/><br/> | One of the key challenges of working with the provided procurement dataset is the absence of categorization of each procurement transaction. Instead of labelling manually, we applied <b>machine learning</b> to classify the tender descriptions into different categories. We firstly scraped the procurement descriptions and categories from GeBiz website using <b> Selenium</b> and <b> BeautifulSoup </b> libraries in Python to be used as the training and validation dataset in our Support Vector Classifier model. We were able to achieve <b> 90% for training accuracy </b> before performing the categorization prediction. <br/><br/> | ||
Line 71: | Line 84: | ||
{| class="wikitable" | {| class="wikitable" | ||
|- | |- | ||
− | ! Dataset/Source | + | ! style = "width: 50%;" | Dataset/Source |
− | ! Data Attributes | + | ! style = "width: 50%;" | Data Attributes |
|- | |- | ||
| Government Procurement Data | | Government Procurement Data | ||
− | | | + | | |
* Tender No | * Tender No | ||
* Agency | * Agency | ||
Line 86: | Line 99: | ||
* Sub Category | * Sub Category | ||
|} | |} | ||
+ | |||
+ | <br/> | ||
+ | <b> Visualization in R </b><br/> | ||
+ | The web application will be built in R and deployed to Shinyapps.io | ||
==<div style="background:#143c67; padding:15px; font-weight: bold; line-height: 0.3em;letter-spacing:0.5em;font-size:20px"><font color=#fbfcfd face="Lato"><center>BACKGROUND SURVEY OF RELATED WORKS</center></font></div>== | ==<div style="background:#143c67; padding:15px; font-weight: bold; line-height: 0.3em;letter-spacing:0.5em;font-size:20px"><font color=#fbfcfd face="Lato"><center>BACKGROUND SURVEY OF RELATED WORKS</center></font></div>== | ||
Line 98: | Line 115: | ||
<p><center>'''Source''': https://goo.gl/P9RjHk </center></p> | <p><center>'''Source''': https://goo.gl/P9RjHk </center></p> | ||
|| | || | ||
− | * The use of | + | * The use of pareto chart allows us to identify the component(s) which is/are contributing significantly and how the categorises value sums up cumlatively. |
|- | |- | ||
| <p><center>'''Title : Word Cloud on Procurement Details ''' </center></p> | | <p><center>'''Title : Word Cloud on Procurement Details ''' </center></p> | ||
Line 104: | Line 121: | ||
<p><center>'''Source''': https://linpack-for-tableau.com/data-visualizations/tableau-dashboards/procurement-dashboard/procurement-cockpit/</center></p> | <p><center>'''Source''': https://linpack-for-tableau.com/data-visualizations/tableau-dashboards/procurement-dashboard/procurement-cockpit/</center></p> | ||
|| | || | ||
− | * | + | * Based on the size of the words, we can identify the keywords quickly. We can learn that by making frequency used keywords stand out can prevent us from overlooking it if it is presented in a tabular format. |
− | |||
|- | |- | ||
− | | <p><center>'''Title : Breakdown on Government | + | | <p><center>'''Title : Breakdown on UK Government Spending ''' </center></p> |
− | [[File:Treemap ref.png|400px|frameless|center]] | + | [[File:Treemap ref 2.png|400px|frameless|center]] |
− | <p><center>'''Source''': http://www. | + | <p><center>'''Source''': http://www.nickmalleson.co.uk/2014/02/uk-government-spending-treemap.html</center></p> |
|| | || | ||
− | * | + | * Based on the chart, we can gain an overview of how the spending is breakdown according to the size and colour of the box. |
− | * | + | * From this, we are able to identify outliers or signficant contribution quickly at a glance. |
|- | |- | ||
| <p><center>'''Title : Team Budget Breadown ''' </center></p> | | <p><center>'''Title : Team Budget Breadown ''' </center></p> | ||
Line 118: | Line 134: | ||
<p><center>'''Source''':https://acquireprocure.com/spend-analysis-visualisation/3-reasons-procurement-professionals-use-sankey-diagrams</center></p> | <p><center>'''Source''':https://acquireprocure.com/spend-analysis-visualisation/3-reasons-procurement-professionals-use-sankey-diagrams</center></p> | ||
|| | || | ||
− | * | + | * Based on the path of the sankey chart, we are able to identify the cash flow from one end to the other end. |
− | + | * The size of the path allows us to to identify how signficant it is in terms of value. | |
− | * | ||
|- | |- | ||
| <p><center>'''Title : Analyzing Involved Authorities, Tenders and Companies ''' </center></p> | | <p><center>'''Title : Analyzing Involved Authorities, Tenders and Companies ''' </center></p> | ||
Line 126: | Line 141: | ||
<p><center>'''Source''': https://linkurio.us/blog/exploring-e1-3-trillion-in-public-contracts-with-graph-visualization/#!prettyPhotooard</center></p> | <p><center>'''Source''': https://linkurio.us/blog/exploring-e1-3-trillion-in-public-contracts-with-graph-visualization/#!prettyPhotooard</center></p> | ||
|| | || | ||
− | * | + | * Based on the charts, we are able to identify key relationship between objects quickly based on the nodes and edges. |
− | + | * From this, we can find common nodes as well. | |
|} | |} | ||
==<div style="background:#143c67; padding:15px; font-weight: bold; line-height: 0.3em;letter-spacing:0.5em;font-size:20px"><font color=#fbfcfd face="Lato"><center>BRAINSTORMING SESSIONS</center></font></div>== | ==<div style="background:#143c67; padding:15px; font-weight: bold; line-height: 0.3em;letter-spacing:0.5em;font-size:20px"><font color=#fbfcfd face="Lato"><center>BRAINSTORMING SESSIONS</center></font></div>== | ||
− | + | [[File:Brainstorm2.png|800px|frameless|center]] | |
− | [[File: | + | During our brainstorming session, we came out a list of visualization which are able to achieve our objectives and eventually shortlisted 4 visualization - Tree map, Network Graph, Sankey Graph and Word Cloud. After rounds of refinement and consultations with our Professor, the image above is the final draft for our visualization. |
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
==<div style="background:#143c67; padding:15px; font-weight: bold; line-height: 0.3em;letter-spacing:0.5em;font-size:20px"><font color=#fbfcfd face="Lato"><center>PROPOSED STORYBOARD</center></font></div>== | ==<div style="background:#143c67; padding:15px; font-weight: bold; line-height: 0.3em;letter-spacing:0.5em;font-size:20px"><font color=#fbfcfd face="Lato"><center>PROPOSED STORYBOARD</center></font></div>== | ||
− | < | + | Our group has proposed the following storyboard in the use of our visual application: |
− | + | ||
− | < | + | {| class="wikitable" style="background-color:#FFFFFF;" width="100%" |
+ | |- | ||
+ | ! style="font-weight: bold;width: 50%;" | Proposed Layout | ||
+ | ! style="font-weight: bold;" | What We Can Analyse | ||
+ | |- | ||
+ | | <p><center>'''Overview of Procurement Spending''' </center></p> | ||
+ | [[File:Storyboard treemap.png|400px|frameless|center]] | ||
+ | <p><center>'''Source''':https://www.r-graph-gallery.com/treemap/</center></p> | ||
+ | || | ||
+ | * Treemap to show the spending breakdown for each category of all agencies under the selected ministry. | ||
+ | * The interactive filters are year and ministry. | ||
+ | |- | ||
+ | | <p><center>'''Relationship between Ministry and Suppliers''' </center></p> | ||
+ | [[File:Storyboard network.png|400px|frameless|center]] | ||
+ | <p><center>'''Source''': https://cran.r-project.org/web/packages/visNetwork/vignettes/Introduction-to-visNetwork.html</center></p> | ||
+ | || | ||
+ | * Network diagram to show the relationship of agencies and suppliers of the selected ministry. | ||
+ | * The interactive filters are year, ministry and range filter which allows the user to filter the suppliers based on the procurement amount. | ||
+ | |- | ||
+ | | <p><center>'''Cashflow from Agency to Suppliers ''' </center></p> | ||
+ | [[File:Storyboard sankey.png|400px|frameless|center]] | ||
+ | <p><center>'''Source''': https://www.r-graph-gallery.com/sankey-diagram/</center></p> | ||
+ | || | ||
+ | * Sankey diagram to show the cash flow between selected agency and suppliers for the selected category. | ||
+ | * The interactive filters are year, ministry, agency and category. | ||
+ | |- | ||
+ | | <p><center>'''Word Cloud on Goods and Services Procured''' </center></p> | ||
+ | [[File:Storyboard wordcloud.png|400px|frameless|center]] | ||
+ | <p><center>'''Source''':http://www.sthda.com/english/wiki/text-mining-and-word-cloud-fundamentals-in-r-5-simple-steps-you-should-know</center></p> | ||
+ | || | ||
+ | * Word cloud to show an overview of the tender description for the selected agency and selected category. | ||
+ | * The filters are year, ministry, agency and category. | ||
+ | * We added a searchable table below the word cloud to allow the user to search for keywords and view the exact tender description | ||
+ | |} | ||
− | ==<div style="background:#143c67; padding:15px; font-weight: bold; line-height: 0.3em;letter-spacing:0.5em;font-size:20px"><font color=#fbfcfd face="Lato"><center>TECHNOLOGIES</center></font></div>== | + | ==<div style="background:#143c67; padding:15px; font-weight: bold; line-height: 0.3em;letter-spacing:0.5em;font-size:20px"><font color=#fbfcfd face="Lato"><center>TOOLS & TECHNOLOGIES</center></font></div>== |
<b>Tools and technologies</b> | <b>Tools and technologies</b> | ||
[[File:Tools used.png|800px|frameless|center]] | [[File:Tools used.png|800px|frameless|center]] | ||
Line 163: | Line 199: | ||
The following are some of the key technical challenges that we may face throughout the course of the project: | The following are some of the key technical challenges that we may face throughout the course of the project: | ||
− | + | ||
{| class="wikitable" | {| class="wikitable" | ||
|- | |- | ||
− | ! Key Challenges | + | ! style = "width: 50%;" | Key Challenges |
− | ! Mitigation Plan | + | ! style="width: 50%;"| Mitigation Plan |
|- | |- | ||
− | | Unfamiliarity with | + | | Unfamiliarity with R and Rshiny Libraries |
| | | | ||
* Attend R Shiny Workshop | * Attend R Shiny Workshop | ||
− | * Independent learning via online resources | + | * Independent learning via online resources such as Datacamp |
* Ask team mates for help | * Ask team mates for help | ||
|- | |- | ||
− | | Unfamiliarity with | + | | Unfamiliarity with Libraries for Machine Learning and Web Crawling |
| | | | ||
+ | * Clean, transform and analyse data together | ||
* Independent learning via online resources | * Independent learning via online resources | ||
− | |||
|- | |- | ||
| Data Cleaning and Transformation | | Data Cleaning and Transformation | ||
| | | | ||
− | * Need to crawl data on website to obtain | + | * Need to crawl data on website to obtain training data for text classification |
* Clean, transform the data together | * Clean, transform the data together | ||
− | |||
|} | |} | ||
− | + | ||
<br/> | <br/> | ||
==<div style="background:#143c67; padding:15px; font-weight: bold; line-height: 0.3em;letter-spacing:0.5em;font-size:20px"><font color=#fbfcfd face="Lato"><center>TIMELINE</center></font></div>== | ==<div style="background:#143c67; padding:15px; font-weight: bold; line-height: 0.3em;letter-spacing:0.5em;font-size:20px"><font color=#fbfcfd face="Lato"><center>TIMELINE</center></font></div>== | ||
− | + | [[File:Timeline 2.png|800px|frameless|center]] | |
− | |||
+ | ==<div style="background:#143c67; padding:15px; font-weight: bold; line-height: 0.3em;letter-spacing:0.5em;font-size:20px"><font color=#fbfcfd face="Lato"><center>REFERENCES</center></font></div>== | ||
+ | * https://www.dsta.gov.sg/docs/default-source/dsta-about/dh02200602-gebiz-from-vision-to-reality.pdf?sfvrsn=2 | ||
+ | * https://support.rstudio.com/hc/en-us/articles/201057987-Quick-list-of-useful-R-packages | ||
+ | * http://enhancedatascience.com/2017/07/06/pick-best-r-packages-interactive-plot-visualisation-22/ | ||
+ | * http://www.sthda.com/english/wiki/text-mining-and-word-cloud-fundamentals-in-r-5-simple-steps-you-should-know | ||
+ | * https://www.r-graph-gallery.com/ | ||
+ | * https://rpubs.com/brandonkopp/creating-a-treemap-in-r | ||
+ | * https://cran.r-project.org/web/packages/visNetwork/vignettes/Introduction-to-visNetwork.html | ||
+ | * https://www.displayr.com/sankey-diagrams-r/ | ||
+ | * https://towardsdatascience.com/using-networkd3-in-r-to-create-simple-and-clear-sankey-diagrams-48f8ba8a4ace | ||
+ | * Datacamp Building Web Applications in R with Shiny Course | ||
+ | * https://www.r-graph-gallery.com/the-wordcloud2-library/ | ||
+ | * https://rstudio.github.io/shinythemes/ | ||
+ | * https://rstudio.github.io/DT/shiny.html | ||
+ | * https://rstudio-pubs-static.s3.amazonaws.com/72023_670962b57f444c04999fd1a0a393e113.html | ||
− | |||
==<div style="background:#143c67; padding:15px; font-weight: bold; line-height: 0.3em;letter-spacing:0.5em;font-size:20px"><font color=#fbfcfd face="Lato"><center>COMMENTS</center></font></div>== | ==<div style="background:#143c67; padding:15px; font-weight: bold; line-height: 0.3em;letter-spacing:0.5em;font-size:20px"><font color=#fbfcfd face="Lato"><center>COMMENTS</center></font></div>== | ||
− | + | Feel free to leave us some comments so that we can improve! | |
− | |||
− | Feel free to leave us some comments so that we can improve! | ||
<center> | <center> |
Latest revision as of 15:41, 25 November 2018
Contents
PROBLEM & MOTIVATION
GeBIZ is a Singapore Government’s one-stop e-procurement portal which facilitates tender activities between Singapore government and local and overseas suppliers. Currently, there is no available tool to aid the public and ministries to understand and gain insights on the procurement made by the government under each ministry. Hence, we are motivated to create an interactive visualisation tool on government's procurement spending to allow the public and ministries to identify spending patterns and gain insights into procurement spending under each ministry.
OBJECTIVES
In this project, we are creating a visualisation that is able to show the following:
- Gain an overview of procurement spending made by each ministry and agency
- Identify the relationships between ministries, agencies and suppliers
- Identify what are the goods and services procured by ministries and agencies under each category
SELECTED DATASETS
The following datasets will be used for analysis , as elaborated below:
Dataset/Source | Data Attributes | Rationale of Usage |
---|---|---|
Government Procurement Data (https://data.gov.sg/dataset/government-procurement) |
|
To gain information on government procurement such as tender description, amount and supplier information |
Ministry and Agencies List |
|
We will be looking through the Singapore Government Directory (https://www.gov.sg/sgdi/ministries) to categorise the agencies into their respective ministries. This will allow us to visualise the procurement spending on a ministry level. |
APPROACH
Exploratory Data Analytics
We used Tableau to perform EDA to better understand our dataset and to aid us in the conceptualization of our story board.
Data Cleaning and Feature Creation
We used Excel and Python to create a new column showing the Ministry that each agency belongs to by merging with data obtained from the Singapore Government Directory.
Text Classification using Support Vector Classifier (SVC)
One of the key challenges of working with the provided procurement dataset is the absence of categorization of each procurement transaction. Instead of labelling manually, we applied machine learning to classify the tender descriptions into different categories. We firstly scraped the procurement descriptions and categories from GeBiz website using Selenium and BeautifulSoup libraries in Python to be used as the training and validation dataset in our Support Vector Classifier model. We were able to achieve 90% for training accuracy before performing the categorization prediction.
Government Procurement Dataset after Text Classification
Dataset/Source | Data Attributes |
---|---|
Government Procurement Data |
|
Visualization in R
The web application will be built in R and deployed to Shinyapps.io
BACKGROUND SURVEY OF RELATED WORKS
Some of these visualizations that we draw inspiration from, are as follows:
Reference of Other Interactive Visualization | What We Can Learn |
---|---|
| |
| |
| |
| |
|
BRAINSTORMING SESSIONS
During our brainstorming session, we came out a list of visualization which are able to achieve our objectives and eventually shortlisted 4 visualization - Tree map, Network Graph, Sankey Graph and Word Cloud. After rounds of refinement and consultations with our Professor, the image above is the final draft for our visualization.
PROPOSED STORYBOARD
Our group has proposed the following storyboard in the use of our visual application:
Proposed Layout | What We Can Analyse |
---|---|
| |
| |
| |
|
TOOLS & TECHNOLOGIES
Tools and technologies
Data Architecture
KEY CHALLENGES
The following are some of the key technical challenges that we may face throughout the course of the project:
Key Challenges | Mitigation Plan |
---|---|
Unfamiliarity with R and Rshiny Libraries |
|
Unfamiliarity with Libraries for Machine Learning and Web Crawling |
|
Data Cleaning and Transformation |
|
TIMELINE
REFERENCES
- https://www.dsta.gov.sg/docs/default-source/dsta-about/dh02200602-gebiz-from-vision-to-reality.pdf?sfvrsn=2
- https://support.rstudio.com/hc/en-us/articles/201057987-Quick-list-of-useful-R-packages
- http://enhancedatascience.com/2017/07/06/pick-best-r-packages-interactive-plot-visualisation-22/
- http://www.sthda.com/english/wiki/text-mining-and-word-cloud-fundamentals-in-r-5-simple-steps-you-should-know
- https://www.r-graph-gallery.com/
- https://rpubs.com/brandonkopp/creating-a-treemap-in-r
- https://cran.r-project.org/web/packages/visNetwork/vignettes/Introduction-to-visNetwork.html
- https://www.displayr.com/sankey-diagrams-r/
- https://towardsdatascience.com/using-networkd3-in-r-to-create-simple-and-clear-sankey-diagrams-48f8ba8a4ace
- Datacamp Building Web Applications in R with Shiny Course
- https://www.r-graph-gallery.com/the-wordcloud2-library/
- https://rstudio.github.io/shinythemes/
- https://rstudio.github.io/DT/shiny.html
- https://rstudio-pubs-static.s3.amazonaws.com/72023_670962b57f444c04999fd1a0a393e113.html
COMMENTS
Feel free to leave us some comments so that we can improve!
No. | Name | Date | Comments |
---|---|---|---|
1. | Insert your name here | Insert date here | Insert comment here |
2. | Insert your name here | Insert date here | Insert comment here |
3. | Insert your name here | Insert date here | Insert comment here |