Bird Audio – Assigned to Andrei Hushcha and Vanessa Prema; plan to explore use of BirdNet or similar tools to analyze audio data.
Bird Computer Vision (CV) – Assigned to Scott Walters, Bina Patel, and Kaushika Mohan; goal is to build classifiers for bird identification from video footage.
Data Sharing
Pranav has videos ready to share and will describe or provide audio files as needed.
Ben and Pranav will coordinate on making files available via Slack (Bird CV channel).
Project Scope & Goals
Identifying birds in both audio and video data to understand ecological impacts, especially in high-elevation Himalayan environments.
Emphasis on open communication between the ecology perspective (Ben, Pranav) and the machine learning/computer vision perspective (Scott, Bina, Andrei, Vanessa, Kaushika).
Importance of Collaboration
Ben reiterated the need for clear, frequent communication.
Team members should ask questions to ensure accurate understanding of objectives and technical requirements.
Action Items
Pranav will share all relevant data (video files and a sample of audio files) in the designated Slack channel.
Audio Team (Andrei, Vanessa) will begin exploring how to run BirdNet (or similar) on a subset of the audio recordings.
CV Team (Scott, Bina, Kaushika) will begin examining the video samples to plan an approach for image classification/detection.
Ben will post a summary of these action items in Slack, and the team will continue to coordinate in the Bird CV channel.
Andrei and Vanessa introduced themselves as the audio‐classification team, sharing their academic and professional backgrounds.
This was Vanessa’s first research project, and she expressed enthusiasm about learning more.
Marco asked about participants’ backgrounds, and they briefly described their current roles and fields of study.
Audio Classification Project:
Andrei and Vanessa mentioned they had an initial discussion the previous day about how to approach the audio classification problem.
They plan to outline a strategy and next steps for the project, including data gathering and defining a methodology.
Advisor/Coordinator Notes:
Breanna indicated this was the first meeting with the comp advisors and researchers together. She will remain available through Slack, but may not attend all subsequent meetings unless needed.
Going forward, the weekly meetings will focus on progress updates from the researchers (Andrei, Vanessa, others), discussion of any issues, and setting goals for the next week.
Action Items
Audio Team Next Steps:
Andrei and Vanessa will continue refining their plan for the audio classification project and prepare updates for the next meeting.
Progress Updates:
At the next session, the researchers (Andrei, Vanessa, etc.) will provide a summary of what they accomplished since this meeting.
Communication with Breanna:
If the audio team or other researchers need further support, they can reach out to Breanna via Slack or request her presence at a future meeting.
Documentation Exchange:
Marco and Breanna will coordinate on receiving any project documents from the students and sharing them with the advisors as needed.
The team attempted to get an account for Cornell Lab’s bird sound library but could not because they are not Cornell affiliates.
The library in question may have H. bulbar (leaf warbler) recordings, which the team would like to use.
Bird Sound Data Sources & Xeno-Canto
The team currently uses Xeno-Canto for verified bird calls and discussion arose on leveraging BirdNet to confirm species from field recordings.
Field Data vs. Xeno-Canto:
Field data is large (up to 500 GB) but not fully labeled.
Xeno-Canto files are smaller, cleaner, and come with verified labels.
Classification Goals
Primary interest: Distinguish between leaf warbler calls (song vs. buzz) and “not leaf warbler.”
Secondary steps might include investigating other species that have similar calls to improve the model’s generalization.
Discussion on how BirdNet’s performance should be measured and whether confidence thresholds (e.g., 0.9) are justified.
Need to determine the best way to incorporate domain expertise (e.g., suggestions for which other species to include).
Performance Evaluation and Labeling
The group discussed the importance of having a small subset of manually verified labels to measure BirdNet’s accuracy and tune thresholds (precision vs. recall).
The need to split data carefully into training/validation/test sets to avoid overlap from the same audio recordings.
Data Processing Approach
Agreement to use Python (e.g., Librosa or SoundFile libraries) to handle audio slicing and classification rather than manual audio software.
The team will further explore how to automatically trim audio files and assign labels (e.g., 3-second clips with bird calls).
Meeting Summaries and Weekly Reports
Andrei and Vanessa hold frequent meetings and produce summaries; the team can decide which ones to upload to GitHub.
A private GitHub repository will serve as a place to store meeting notes and project files so everyone has access.
Action Items
Evaluate Cornell Lab Options
Determine if there’s an alternative method or contact at Cornell to obtain needed bird calls.
If unsuccessful, seek comparable data sources or rely on existing Xeno-Canto resources.
Small Labeled Subset for Performance Checks
Identify a subset of field data for manual verification.
Use this labeled subset to tune BirdNet thresholds and measure performance.
Python-Based Audio Processing
Explore appropriate Python libraries (e.g., Librosa, SoundFile) to slice audio recordings and manage train/validation/test splits.
Integrate Meeting Summaries on GitHub
Upload relevant meeting notes and weekly reports to the private GitHub repository for archival and reference purposes.
Discuss Inclusion of Additional Bird Species
Determine if adding data from similar species (recommended by domain experts) would help improve model generalization.
Next Steps
Follow up on labeling efforts and threshold selection for BirdNet.
Continue refining the data pipeline, especially the Python-based slicing and classification approach.
Schedule a future meeting to review progress on data labeling, model training, and repository updates.
WEEK 5
Meeting Link (02/07/2025)
Summary of Meeting
Vision Process and Model Accuracy
Pranav asked about the vision process. Andrei mentioned the current use of BirdNet with a confidence level of 0.9 to analyze data.
However, they don’t have model algorithms yet and are working with the data to evaluate its quality.
Building a New Model
Andrei explained that the next step is to create their own model based on known data (buses and calls) and run it to compare results with BirdNet.
Data and Annotations Progress
Scott shared that progress is being made in annotating frames for training data. He had initial issues with a poor annotation rate but found a better nesting site that improved the rate of true positives.
Computational Resources and Data Storage
Scott mentioned the potential need for Amazon S3 for data storage as external hard drives are becoming insufficient for the growing data needs.
Project Updates
Bina was running late but updated that she’s working on data with Yolo models, and plans to sync up with Scott and Kaushika based on Scott’s annotated data.
Action Items
Pranav: Continue posting videos on Dropbox for sharing with the team.
Determine if there’s an alternative method or contact at Cornell to obtain needed bird calls.
If unsuccessful, seek comparable data sources or rely on existing Xeno-Canto resources.
Scott: Keep annotating the frames for training and move towards fine-tuning the models next week with the annotated data. Explore additional data storage solutions, like Amazon S3.
Bina: Sync up with Scott and Kaushika to work with the annotated data for model training.
All: Stay in contact regarding the meeting schedules and resources, and reach out if there are any questions or need for further support.
Determine if adding data from similar species (recommended by domain experts) would help improve model generalization.
Next Steps
The meeting ended with clear action items. The next steps include finalizing the annotated data and exploring storage solutions. The team will continue coordinating their efforts and keep each other updated on their progress.
The group discussed how confidence levels a ect the accuracy of model predictions.
There were concerns about how to interpret a 24% misclassification rate, with opinions that the size of the list matters when judging accuracy.
The group also discussed the trade-o between using a high confidence threshold (fewer but more reliable samples) and a lower threshold (more samples but potentially more noise).
Data Quality and Cleanliness:
The data still requires cleaning and refinement to be useful, as it’s not currently reliable enough for accurate analysis.
Basic analysis has been done, but the data’s current state means the results are not final or useful yet.
Bird Song vs. Call Analysis:
Prema manually went through Xeno Canto files, adding columns to distinguish between bird calls and buzzes. However, further analysis is needed, particularly for files that contain both types of sounds.
Model Evaluation and Metrics:
The team debated whether using absolute numbers or percentages of misclassified species would be more helpful. They agreed that looking at the percentage of misclassified data across di erent thresholds would be more informative.5. Open Source Resources:
The team discussed the open-source nature of BirdNet, but it was noted that the Cornell Merlin ID application is not open-source, which might limit access to its data.
Data Clustering:
The team briefly mentioned clustering analysis but has not made significant progress on this task yet.
Action Items
Investigate Confidence Level Discrepancies:
Further analysis will be done on how confidence levels impact model predictions. A histogram of confidence levels associated with zero labels (misclassifications) will be created to understand the distribution.
Data Cleaning and Preparation:
The team will continue working on cleaning the dataset to ensure it’s ready for analysis. Given the current state, results are not reliable enough to show yet.
Finalize Bird Song vs. Call Data:
Prema will continue working on categorizing bird songs and calls, and the team will later integrate this data into the analysis.
Threshold Analysis for Misclassification:
Focus on analyzing the percentage of misclassified species at different thresholds and comparing them to understand model performance better
Clustering Analysis:
Once data is cleaned, the team will revisit clustering analysis and its application to the bird call data.
Consult Papers and Open Source Data:
Andrei will explore open-source papers and research on BirdNet and related resources to understand better how the data and confidence levels should be interpreted.
Next Steps
Focus on data cleaning and analysis over the next week, with plans to move into the model building phase after completing this foundational work.
The team discussed the current state of the BirdNet model, using it to analyze bird species based on confidence levels and local species lists. They are working to figure out what the confidence levels mean and how to use them effectively for species classification.
Confidence Thresholds:
They compared different confidence levels and observed that as confidence increases, the accuracy of species identification improves. However, there is a trade-off between losing more data and achieving higher confidence.
Species Misclassification:
Vanessa raised concerns about the misclassification of birds, particularly with species like the Pacific Ren, which is not native to the area but is detected with high confidence.
Audio File Analysis:
The audio files are processed in 15-minute chunks, with various species detected throughout. They discussed how to handle misclassified sounds, with one option being to only accept species detections with high confidence.5. Elevation Data: They discussed the importance of high-elevation data, which is still being collected. Pranav noted that the species distribution will change drastically at higher elevations, with fewer species detected.
Sound Pattern Analysis:
Andrei asked whether sounds from other birds around the Humes warbler could be useful. Pranav suggested looking for sound partitioning patterns, as multiple species might sing in overlapping time windows.
Action Items
Set Confidence Threshold:
The team agreed on using a confidence threshold between 0.8 and 0.9 for species identification, balancing data accuracy with completeness.
Handle Misclassifications:
Species not native to the area, like the Pacific Ren, should be excluded from the dataset.
Continue Data Collection:
Pranav will work on gathering high-elevation data, which will provide a more complete picture of species distribution.
Review Audio Windows:
Andrei will experiment with different audio window sizes (e.g., 3, 6, and 9 seconds) to detect patterns in bird sounds and determine if interactions between species can be observed.
Build Model for Humes Warbler:
Once the data is cleaned and the confidence threshold is set, the team will build a model based on Humes warbler sounds, running it through the collected dataset and evaluating its performance.
Slack Communication:
Pranav will be intermittently available on Slack due to an upcoming exam but will continue monitoring updates and providing feedback.
The team is working on extracting the top 3 species probabilities (highest, second, and third) from the BirdNet model. If they can’t implement this in Python, they’ll proceed with the 0.8 confidence threshold in the BirdNet app.
Accuracy Estimation:
Annan suggested estimating the model’s true accuracy using assumptions about misclassification probability. This will help adjust thresholds and provide better accuracy estimates.
Data Labeling & Model Training:
Annan clarified that removing “unexpected species” from training won’t harm the model, but may a ect accuracy reporting.
Weekly Reports:
There were issues accessing weekly updates on Teams, which are being addressed.
Bird Behavior Insights:
Vanessa shared that birds do not overlap in certain areas, which simplifies data filtering.
Action Items
Andrei & Vanessa:
Finalize the BirdNet model analysis, and if possible, implement top 3 probabilities in Python.
Continue adjusting thresholds and cleaning data based on Annan’s feedback.
Annan:
Provide a formula for estimating total accuracy based on misclassification assumptions.
Review and give feedback on the weekly reports.
Next Steps
Annan will review reports and provide feedback soon.
The team will continue refining the model and resolving the Teams file access issue.
The data processing phase is slower than expected due to intensive data validation (“garbage in, garbage out”).
There was discussion about setting appropriate probability thresholds (e.g., using a 0.5 cutoff instead of 0.1) for classifying bird species accurately.
Handling overlapping bird calls: The focus remains on retrieving samples where the Hume leaf warbler is present, even if other birds are detected.
Handling Real-World Data Limitations:
Acknowledgment of the noise and variability inherent in real-world data, including issues like microphone distance and environmental interference.
Discussion on filtering out unexpected species based on field expert input and discarding data points with ambiguous or low-confidence detections.
Emphasis on documenting all assumptions and limitations to account for these challenges in the final analysis.
Further Analysis & Hypothesis Testing:
Plan to analyze daily activity patterns (e.g., comparing morning vs. afternoon bird activity).
Exploration of a “social network” model for birds by mapping spatial relationships (using node types for cameras/mics and nests).
Consideration of graph-based techniques, including graph convolutional neural networks (GCNs), to model real-world interactions among birds.
Data Visualization & Graphs:
Review of map data showing camera and mic locations, with discussion on how to represent different data points (e.g., using different node types).
Consideration of using graph analytics to better understand spatial relationships and interactions.
Action Items
Marco Postiglione:
Request and provide access to necessary files/documents for the team.
Draft and circulate a document listing all assumptions and limitations for the analysis.
Search for a beginner-friendly tutorial on graph convolutional neural networks (GCNs) and share it with the team.
Vanessa Prema:
Develop and share a template for documenting analysis and graph ideas.
Coordinate the sharing of updated documents with the team once the template is ready.
Andrei Hushcha:
Prepare a preliminary summary of findings regarding node mapping and spatial analysis for review.
Share screen/document details during the next session to clarify data visualization approaches.
Team (Collectively)
Consolidate feedback on data processing, confidence level thresholds, and filtering criteria.
Schedule the next meeting in two weeks, and continue to coordinate via Teams for ongoing questions and updates.
A significant portion of bird activity occurs at middle elevations.
Peak bird singing is observed between 6:00 and 7:00 AM with slightly fewer observations at 5:00 AM.
Analysis yielded over 15,000 observations of songs longer than 3 seconds, with an average continuous song duration of approximately 9 seconds.
Notable Findings:
A specific bird (referred to as Western Crowned Warbler) dominated the observations
Some instances showed continuous singing episodes
Technical Challenges & Data Accuracy:
Concerns were raised that the current code may not be capturing the correct details from the audio clips, leading to potential misclassification or data inaccuracies.
There is an ongoing need to refine the audio processing algorithm to better capture the duration and continuity of bird songs.
Visualization & Data Presentation:
The team discussed what the best visual representations might be to effectively display the observed patterns and durations in the bird song data.
Considerations were made about how to represent multi-bird song overlaps in the analysis.
Broader Discussion on Computer Vision & Deep Learning:
Scott shared insights from his computer vision and deep learning classes, noting the limitations of traditional techniques (e.g., template matching) in detecting well-camouflaged birds in video data.
The potential for transformer-based models in improving object recognition in challenging conditions (dense foliage, varying weather) was highlighted.
Discussion included how factors like cooler, denser early morning air might affect audio transmission and analysis.
Action Items
Walters, Scott:
Transformer Models: Focus on fine-tuning the transformer-based models for object recognition to better address challenges posed by camouflage and environmental variables in video data.
Andrei Hushcha:
Code Refinement: Revisit and recode the audio processing algorithm to improve accuracy in capturing bird song durations and classifying the data correctly.
Data Visualization: Investigate and propose e ective visualization techniques that clearly illustrate the timing, duration, and frequency of bird songs.
Course Enrollment: Consider signing up for the computer vision course to enhance expertise in related analytical techniques.
A significant portion of bird activity occurs at middle elevations.
Peak bird singing is observed between 6:00 and 7:00 AM with slightly fewer observations at 5:00 AM.
Analysis yielded over 15,000 observations of songs longer than 3 seconds, with an average continuous song duration of approximately 9 seconds.
Data Access & Sharing:
Marco noted di iculties accessing the files using the current shared links.
There was discussion about possibly sharing an entire folder to enable easier data updates.
Data Timeline & Observations:
The dataset covers approximately five months, with a focus on three months of observations.
Discussion on using months (rather than days or hours) as the main time scale for analysis, with some reference to early morning time bins (e.g., 5:00–7:00 AM).
Preliminary Data Analysis Functions:
Andrei presented functions developed for:
Visualizing general data information
Identifying singing patterns
Evaluating continuous song segments
Grouping data based on confidence levels and calculating average “accuracy” (mean probability) across observations.
There was discussion on how to interpret the accuracy function, noting that an average of 0.75 (for a 0.5 confidence threshold) might suggest 75% of predictions are “accurate.”
Data Splitting & Threshold Selection:
Andrei explained that he split the dataset into five subsets based on varying confidence thresholds.
Marco questioned if the splitting criteria were overlapping (e.g., including higher confidence values when setting a lower threshold). Andrei confirmed the approach.
A plot indicated that as the confidence level increases, both the average accuracy and share of evidence increase—leading to a preliminary decision to use a 0.7 threshold as a trade-o .
Annan raised a point about the plotting strategy—suggesting that plotting count against accuracy (rather than confidence level) might yield a more robust insight due to potential rescaling issues.
Overall Modeling & Future Experiments:
Marco stressed that while choosing a threshold (e.g., 0.7) seems appropriate now, the overall model architecture should be tested with various thresholds.
There was also discussion on potential classification tasks (e.g., divs. calls) and predictive analysis differentiating bots
Analysis of Singing Patterns & Interactions:
Andrei reviewed additional analysis details:
Observations about Hume’s model at mid elevations, typical singing times, and geographic distribution.
Investigation into continuous song durations and identifying “champions” (e.g., instances with very long continuous singing).
Analysis of bird interactions within the same 3-second channel, including pair interactions (such asume’s bulbar with other species).
Marco suggested that these interaction analyses might benefit from being further broken down by time bins
Domain Expert Input:
It was noted that the current exploratory analysis has not yet been discussed with the domain experts, who were absent during the last scheduled discussion.
Marco emphasized the importance of consulting the experts to ensure that the analysis and any further classification tasks (such as predictive models based on nest activity) are aligned with the domain’s priorities.
Action Items
Data Sharing:
Andrei: Work on a solution for file sharing, potentially by setting up an entire shared
folder to streamline data updates.
Plotting Strategy Revision:
Andrei: Revise the plotting approach to compare counts versus accuracy instead of counts versus confidence level, per Annan’s suggestion.
Refinement of Analysis Functions:
Revisit the functions used for identifying song durations to address any issues.
Reassess the function grouping birds by interaction within the 3-second window
Threshold Experimentation:
Team: Plan to experiment with various confidence thresholds in the overall model architecture, ensuring that the selected threshold (e.g., 0.7) is optimal.
Consult Domain Experts:
Schedule a follow-up meeting or discussion with the domain experts to review the current exploratory analysis and gather feedback on:
The usefulness of the current interaction analysis.
Potential adjustments in analysis (e.g., time bin comparisons).
Future classification or predictive tasks (e.g., differentiating between buzz calls vs. other song types or forecasting nest activity).
Preprocessing steps completed: audio standardization (1 channel, consistent sample rate), chunking into 3-second segments based on BirdNET’s format, and inspection of audio properties.
Audio Preprocessing & Class Definition:
Files without Hume’s Warbler are used for the “Other” class.
Discussion on whether to match elevation distributions (Low: 50%, Medium/High: ~25% each) when building the “Other” class.
Concern about over-representation or under-representation of certain bird species within the “Other” class.
Custom Model Justification:
The need for a custom model arises from limitations in BirdNET’s coverage (e.g., it does not distinguish between “buzz” and “call” for Hume’s Warbler).
The team emphasized that this model is not meant to replace BirdNET but rather supplement it, especially where BirdNET lacks class granularity or confidence level tuning.
Model Performance Consideration:
Both BirdNET and the new model may su er from threshold ambiguity. There’s currently no clear metric to determine the best cutoff .
Concerns about building a model evaluated by the same outputs it’s trained on (BirdNET), which may limit potential improvement or generalization.
Buzz vs Call Classification:
The current priority is to develop a binary classification model to distinguish “buzz” and “call” types from Hume’s Warbler recordings.
Manual annotation is necessary due to the lack of labeled field data for this task.
Generalization on Field Data:
Acknowledged that models trained on Xeno-Canto data may perform very well (95%+ F1 score), but could degrade significantly on real field recordings.
Need to test generalizability and potentially include field data in the training set later.
Action Items
Andrei:
Continue developing the custom binary classifier (buzz vs. call) using annotated Xeno-Canto samples
Manually annotate ~30 buzz and ~30 call samples from field data for real-world testing
Marco:
Decide whether elevation proportions should be mirrored in the “Other” class sample