Passively collected acoustic data have become increasingly common in wildlife research and have prompted the development of machine‐learning approaches to extract and classify large sets of audio files. BirdNET is an open‐source automatic prediction model that is popular because of its lack of training requirements for end users. Several studies have sought to test the accuracy of BirdNET and illustrate its potential in occupancy modelling of single or multiple species. However, these techniques either require extensive statistical knowledge or computational power to be applied to large datasets. In addition, there is a lack of comparisons of occupancy and community composition calculated using BirdNET and typical field methods. Here we develop a framework for assessing the accuracy of BirdNET using generalized linear mixed models to determine species‐specific confidence score thresholds. We then compare community composition under our model and another post‐processing approach to field data collected from co‐located point count surveys in northeastern Vermont. Our framework outperformed the other post‐processing method and resulted in species composition similar to that of point count surveys. Our work highlights the potential mismatch between accuracy and confidence score and the importance of developing species‐specific thresholds. The framework can facilitate research on large acoustic datasets and can be applied to output from BirdNET or other automatic prediction models.
