Our paper “What is the ground truth? Reliability of multi-annotator data for audio tagging” was accepted for presentation at EUSIPCO 2021. The presented work studies annotator competence and estimation of a reliable reference annotation for audio data from crowdsourced annotation.
Many thanks to the students of the Introduction to Audio course (autumn 2020) who were the annotators, as part of their course project work. Most of them hated the task!
You can find the paper in the Publications tab. The data will also be published soon.