Difference between revisions of "ISSS608 2017-18 T3 Assign Tan Yong Ying Data Overview and Cleaning"
Jump to navigation
Jump to search
Yy.tan.2017 (talk | contribs) |
Yy.tan.2017 (talk | contribs) |
||
(10 intermediate revisions by the same user not shown) | |||
Line 13: | Line 13: | ||
| style="font-family:Century Gothic; font-weight:bold; font-size:100%; text-align:center; background-color:#bdb0ab; width=20%" | | | style="font-family:Century Gothic; font-weight:bold; font-size:100%; text-align:center; background-color:#bdb0ab; width=20%" | | ||
; | ; | ||
− | [[ISSS608_2017- | + | [[ISSS608_2017-18_T3_Assign_Tan_Yong_Ying_Application_Design|<font color="#000000">Application Design</font>]] |
| style="font-family:Century Gothic; font-weight:bold; font-size:100%; text-align:center; background-color:#bdb0ab; width=20%" | | | style="font-family:Century Gothic; font-weight:bold; font-size:100%; text-align:center; background-color:#bdb0ab; width=20%" | | ||
; | ; | ||
− | [[ISSS608_2017- | + | [[ISSS608_2017-18_T3_Assign_Tan_Yong_Ying_Task_1_Insights|<font color="#000000">Task 1 Insights</font>]] |
+ | | style="font-family:Century Gothic; font-weight:bold; font-size:100%; text-align:center; background-color:#bdb0ab; width=20%" | | ||
+ | ; | ||
+ | [[ISSS608_2017-18_T3_Assign_Tan_Yong_Ying_Task_2_Insights|<font color="#000000">Task 2 Insights</font>]] | ||
| style="font-family:Century Gothic; font-weight:bold; font-size:100%; text-align:center; background-color:#bdb0ab; width=20%" | | | style="font-family:Century Gothic; font-weight:bold; font-size:100%; text-align:center; background-color:#bdb0ab; width=20%" | | ||
; | ; | ||
Line 28: | Line 31: | ||
{| class="wikitable" | {| class="wikitable" | ||
|- | |- | ||
− | + | ! Dataset Description !! Screenshot and Metadata | |
|- | |- | ||
− | | 1 | + | | 1. Calls and songs from known birds in the Boonsong Lekagul Wildlife Preserve. They are in MP3 format with varying lengths. The file name contains the species name and an integer that can be referenced with AllBirdsv4.csv for its metadata. || [[Image:AllBirdsFileNames.PNG|center|frame|'''Screenshot of file names provided by Mistford College''']] |
|- | |- | ||
− | | 2 | + | | 2. “AllBirdsv4.csv” contains metadata for the calls and songs from known birds in eight variables. |
|| | || | ||
{| class="wikitable" style="margin: auto;" | {| class="wikitable" style="margin: auto;" | ||
Line 56: | Line 59: | ||
|} <br> [[Image:AllBirdsMetadata.PNG|800px|border]] | |} <br> [[Image:AllBirdsMetadata.PNG|800px|border]] | ||
|- | |- | ||
− | | 3 | + | | 3. Fifteen bird sound files provided by Kasios to support their claim that Rose-crested Blue Pipits are still plentiful across the Preserve. They were recorded over the past few months and are in MP3 format. The file names are integers between 1 and 15, and its metadata can be referenced with Test Bird Locations.csv. || [[Image:TestBirdsFileNames.PNG|center|frame|'''Screenshot of file names provided by Kasios''']] |
|- | |- | ||
− | | 4 | + | | 4. “Test Bird Locations.csv” contains metadata for the Kasios files in three variables. |
|| | || | ||
{| class="wikitable" style="margin: auto;" | {| class="wikitable" style="margin: auto;" | ||
Line 72: | Line 75: | ||
|} <br> [[Image:TestBirdsMetadata.PNG|center|frame|'''Screenshot of Test Birds Locations.csv''']] | |} <br> [[Image:TestBirdsMetadata.PNG|center|frame|'''Screenshot of Test Birds Locations.csv''']] | ||
|- | |- | ||
− | | 5 | + | | 5. “Lekagul Roadways 2018.bmp” is a 200 by 200 pixel map of the Preserve which outlines the roadways through the site. The bottom left of the map has coordinate (0,0) and the top right of the map has coordinate (199,199). Note that the alleged dumping site was centered around coordinate (148, 159) and its extent has not been thoroughly studied. || [[Image:Roadways.jpg|center|frame|'''Map of Lekagul Preserve with roadways''']] |
|} | |} | ||
− | + | <br> | |
+ | <div style="border-style: solid; border-width:0; background: #c8bdb9; padding: 7px; font-weight: bold; text-align:left; line-height: wrap_content; text-indent: 20px; font-size:20px; font-family:Century Gothic;border-bottom:5px solid white; border-top:5px solid black"><font color= #000000>Data Cleaning</font></div> | ||
+ | Out of the 5 pieces of data listed above, only AllBirdsv4.csv requires data cleaning to remove values that cannot be imputed or replaced manually through guessing or inference. The data cleaning outcome for each variable in AllBirdsv4.csv is as follows: | ||
+ | # '''File ID'''<br>This variable has no invalid values, meaning all File IDs are valid integer values.<br>[[File:FileIDSummary.png|400px|none]]<br> | ||
+ | # '''English_name'''<br>This variable has no invalid values. The summary shows we have recordings from 19 unique known species in the Preserve provided to us.<br>[[File:English_nameSummary.png|1000px|none]]<br> | ||
+ | # '''Vocalization_Type'''<br>There is an invalid value of “?” for some rows. In any analysis of bird sounds, it is important to differentiate between songs and calls because they play different roles in the communication of birds. Bird songs are usually used by male birds to establish their territories and attract female birds during the breeding season. On the other hand, bird calls are functional and used to coordinate behavior between pairs or birds in a flock. Thus, any records of unknown vocalization type are excluded from our analysis.<br>[[File:VocalizationScreenshot.png|800px|none]]<br>There are 10 unique values in total (see screenshot below). Since bird sounds are commonly differentiated as a "song" or "call", I reduced the number of level in this variable to the three most common values: "call", "song" and "call,song" (case sensitive). Records that do not contain the words "call" or "song" for Vocalization_type are removed from further analysis.<br>[[File:VocalizationSummary.png|800px|none]]<br> | ||
+ | # '''Quality'''<br>The summary shows there are six levels in this variable: “A”, “B”, “C”, “D”, “E” and “no score”. Although the value “no score” does not indicate the quality of the sound file, it is not a vital piece of information in my analysis because I prioritized files of “A” quality in the comparison of known files against Kasios files. Therefore I did not delete any records based on their Quality value. Instead, the 6 levels can be used as a filter requirement during analysis and application development later on.<br>[[File:QualitySummary.png|600px|none]]<br> | ||
+ | # '''Time'''<br>This variable had a lot of invalid values such as “?” or “?:?”. However, after some exploratory analysis, I decided that it was best to only consider the year of the recording in any analysis that included the temporal factor, because there are not enough data points to consider analyzing the dataset on a deeper level, such as by quarters or months. Therefore, no records were removed based on their time value.<br> | ||
+ | # '''Date'''<br>Through inspection, it can be inferred that the default format for dates in this dataset is “MM/DD/YYYY”. There are records with invalid date values, such as “YYYY-MM-00” or “0000-00-00”. Since my analysis would focus on the change of the species’ spatial distributions by year, it is important to have information on the year the recording was captured, and records without this information would be excluded from further analysis.<br>[[File:DateScreenshot.png|800px|none]]<br> | ||
+ | # '''X'''<br>This variable has no invalid values, meaning all records have integer values between 0 and 199.<br>[[File:XSummary.png|400px|none]]<br> | ||
+ | # '''Y'''<br>Two records have invalid values for the Y variable, and they are removed from further analysis.<br>[[File:YSummary.png|800px|none]] | ||
Banner image credit to: [https://www.flickr.com/photos/23660854@N07/24385545393 Marshal Hedin] | Banner image credit to: [https://www.flickr.com/photos/23660854@N07/24385545393 Marshal Hedin] |
Latest revision as of 22:58, 8 July 2018
|
|
|
|
|
|
Data Overview
For this challenge, we were provided with the following data:
Data Cleaning
Out of the 5 pieces of data listed above, only AllBirdsv4.csv requires data cleaning to remove values that cannot be imputed or replaced manually through guessing or inference. The data cleaning outcome for each variable in AllBirdsv4.csv is as follows:
- File ID
This variable has no invalid values, meaning all File IDs are valid integer values. - English_name
This variable has no invalid values. The summary shows we have recordings from 19 unique known species in the Preserve provided to us. - Vocalization_Type
There is an invalid value of “?” for some rows. In any analysis of bird sounds, it is important to differentiate between songs and calls because they play different roles in the communication of birds. Bird songs are usually used by male birds to establish their territories and attract female birds during the breeding season. On the other hand, bird calls are functional and used to coordinate behavior between pairs or birds in a flock. Thus, any records of unknown vocalization type are excluded from our analysis.
There are 10 unique values in total (see screenshot below). Since bird sounds are commonly differentiated as a "song" or "call", I reduced the number of level in this variable to the three most common values: "call", "song" and "call,song" (case sensitive). Records that do not contain the words "call" or "song" for Vocalization_type are removed from further analysis. - Quality
The summary shows there are six levels in this variable: “A”, “B”, “C”, “D”, “E” and “no score”. Although the value “no score” does not indicate the quality of the sound file, it is not a vital piece of information in my analysis because I prioritized files of “A” quality in the comparison of known files against Kasios files. Therefore I did not delete any records based on their Quality value. Instead, the 6 levels can be used as a filter requirement during analysis and application development later on. - Time
This variable had a lot of invalid values such as “?” or “?:?”. However, after some exploratory analysis, I decided that it was best to only consider the year of the recording in any analysis that included the temporal factor, because there are not enough data points to consider analyzing the dataset on a deeper level, such as by quarters or months. Therefore, no records were removed based on their time value. - Date
Through inspection, it can be inferred that the default format for dates in this dataset is “MM/DD/YYYY”. There are records with invalid date values, such as “YYYY-MM-00” or “0000-00-00”. Since my analysis would focus on the change of the species’ spatial distributions by year, it is important to have information on the year the recording was captured, and records without this information would be excluded from further analysis. - X
This variable has no invalid values, meaning all records have integer values between 0 and 199. - Y
Two records have invalid values for the Y variable, and they are removed from further analysis.
Banner image credit to: Marshal Hedin