ISSS608 2017-18 T3 Assign Tan Yong Ying Data Overview and Cleaning

VAST Challenge 2018: Suspense at the Wildlife Preserve

Overview

Data Overview and Cleaning

Application Design

Task 1 Insights

Task 2 Insights

Conclusion

Data Overview

For this challenge, we were provided with the following data:

Dataset Description	Screenshot and Metadata
1. Calls and songs from known birds in the Boonsong Lekagul Wildlife Preserve. They are in MP3 format with varying lengths. The file name contains the species name and an integer that can be referenced with AllBirdsv4.csv for its metadata.	Screenshot of file names provided by Mistford College
2. “AllBirdsv4.csv” contains metadata for the calls and songs from known birds in eight variables.	Description of each variable in AllBirdsv4.csv Variable Name Description File ID The integer index of the file names for calls and songs from known birds English_name The common English name for the recorded bird Vocalization_type Type of bird sound, typically a “call” or a “song” Quality Quality of the recorded bird sound (A, B, C, D, E or no score). A recording with loud background noise or low bird sound purity has a lower quality. Time Time the recording was captured Date Date the recording was captured X X coordinate of the location the recording was captured at Y Y coordinate of the location the recording was captured at
3. Fifteen bird sound files provided by Kasios to support their claim that Rose-crested Blue Pipits are still plentiful across the Preserve. They were recorded over the past few months and are in MP3 format. The file names are integers between 1 and 15, and its metadata can be referenced with Test Bird Locations.csv.	Screenshot of file names provided by Kasios
4. “Test Bird Locations.csv” contains metadata for the Kasios files in three variables.	Description of each variable in Test Bird Locations.csv Variable Name Description ID The integer index of the file names provided by Kasios X X coordinate of the location the Kasios recording was captured at Y Y coordinate of the location the Kasios recording was captured at Screenshot of Test Birds Locations.csv
5. “Lekagul Roadways 2018.bmp” is a 200 by 200 pixel map of the Preserve which outlines the roadways through the site. The bottom left of the map has coordinate (0,0) and the top right of the map has coordinate (199,199). Note that the alleged dumping site was centered around coordinate (148, 159) and its extent has not been thoroughly studied.	Map of Lekagul Preserve with roadways

Data Cleaning

Out of the 5 pieces of data listed above, only AllBirdsv4.csv requires data cleaning to remove values that cannot be imputed or replaced manually through guessing or inference. The data cleaning outcome for each variable in AllBirdsv4.csv is as follows:

1. File ID
   This variable has no invalid values, meaning all File IDs are valid integer values.
   
   
   
   
2. English_name
   This variable has no invalid values. The summary shows we have recordings from 19 unique known species in the Preserve provided to us.
   
   
   
   
3. Vocalization_Type
   There is an invalid value of “?” for some rows. In any analysis of bird sounds, it is important to differentiate between songs and calls because they play different roles in the communication of birds. Bird songs are usually used by male birds to establish their territories and attract female birds during the breeding season. On the other hand, bird calls are functional and used to coordinate behavior between pairs or birds in a flock. Thus, any records of unknown vocalization type are excluded from our analysis.
   
   
   
   There are 10 unique values in total (see screenshot below). Since bird sounds are commonly differentiated as a "song" or "call", I reduced the number of level in this variable to the three most common values: "call", "song" and "call,song" (case sensitive). Records that do not contain the words "call" or "song" for Vocalization_type are removed from further analysis.
   
   
   
   
4. Quality
   The summary shows there are six levels in this variable: “A”, “B”, “C”, “D”, “E” and “no score”. Although the value “no score” does not indicate the quality of the sound file, it is not a vital piece of information in my analysis because I prioritized files of “A” quality in the comparison of known files against Kasios files. Therefore I did not delete any records based on their Quality value. Instead, the 6 levels can be used as a filter requirement during analysis and application development later on.
   
   
   
   
5. Time
   This variable had a lot of invalid values such as “?” or “?:?”. However, after some exploratory analysis, I decided that it was best to only consider the year of the recording in any analysis that included the temporal factor, because there are not enough data points to consider analyzing the dataset on a deeper level, such as by quarters or months. Therefore, no records were removed based on their time value.
   
6. Date
   Through inspection, it can be inferred that the default format for dates in this dataset is “MM/DD/YYYY”. There are records with invalid date values, such as “YYYY-MM-00” or “0000-00-00”. Since my analysis would focus on the change of the species’ spatial distributions by year, it is important to have information on the year the recording was captured, and records without this information would be excluded from further analysis.
   
   
   
   
7. X
   This variable has no invalid values, meaning all records have integer values between 0 and 199.
   
   
   
   
8. Y
   Two records have invalid values for the Y variable, and they are removed from further analysis.
   
   

Banner image credit to: Marshal Hedin