ISSS608 2017-18 T3 Assign Tan Yong Ying Data Overview and Cleaning
Revision as of 22:58, 8 July 2018 by Yy.tan.2017 (talk | contribs)
|
|
|
|
|
|
Data Overview
For this challenge, we were provided with the following data:
Data Cleaning
Out of the 5 pieces of data listed above, only AllBirdsv4.csv requires data cleaning to remove values that cannot be imputed or replaced manually through guessing or inference. The data cleaning outcome for each variable in AllBirdsv4.csv is as follows:
- File ID
This variable has no invalid values, meaning all File IDs are valid integer values. - English_name
This variable has no invalid values. The summary shows we have recordings from 19 unique known species in the Preserve provided to us. - Vocalization_Type
There is an invalid value of “?” for some rows. In any analysis of bird sounds, it is important to differentiate between songs and calls because they play different roles in the communication of birds. Bird songs are usually used by male birds to establish their territories and attract female birds during the breeding season. On the other hand, bird calls are functional and used to coordinate behavior between pairs or birds in a flock. Thus, any records of unknown vocalization type are excluded from our analysis.
There are 10 unique values in total (see screenshot below). Since bird sounds are commonly differentiated as a "song" or "call", I reduced the number of level in this variable to the three most common values: "call", "song" and "call,song" (case sensitive). Records that do not contain the words "call" or "song" for Vocalization_type are removed from further analysis. - Quality
The summary shows there are six levels in this variable: “A”, “B”, “C”, “D”, “E” and “no score”. Although the value “no score” does not indicate the quality of the sound file, it is not a vital piece of information in my analysis because I prioritized files of “A” quality in the comparison of known files against Kasios files. Therefore I did not delete any records based on their Quality value. Instead, the 6 levels can be used as a filter requirement during analysis and application development later on. - Time
This variable had a lot of invalid values such as “?” or “?:?”. However, after some exploratory analysis, I decided that it was best to only consider the year of the recording in any analysis that included the temporal factor, because there are not enough data points to consider analyzing the dataset on a deeper level, such as by quarters or months. Therefore, no records were removed based on their time value. - Date
Through inspection, it can be inferred that the default format for dates in this dataset is “MM/DD/YYYY”. There are records with invalid date values, such as “YYYY-MM-00” or “0000-00-00”. Since my analysis would focus on the change of the species’ spatial distributions by year, it is important to have information on the year the recording was captured, and records without this information would be excluded from further analysis. - X
This variable has no invalid values, meaning all records have integer values between 0 and 199. - Y
Two records have invalid values for the Y variable, and they are removed from further analysis.
Banner image credit to: Marshal Hedin