Learning from multiple annotators with varying expertise (Q2251480)

This paper deals with the development of probabilistic approaches for statistical learning related to learning from multiple annotators, or knowledge sources. It mainly focuses on the case of unreliable annotators, or in the case their expertise varies depending on the observed data. The authors experimentally show that annotator expertise can vary in real tasks. The proposed evaluation strategy was thoroughly developed and tested. Overall, the paper, based on a probabilistic point of view, brings together a quite complete set of modeling tools, experiments, and insights to address the multi-labeler machine learning problem.