Millionaire: a hint-guided approach for crowdsourcing

DOI10.1007/S10994-018-5766-5zbMATH Open1493.68301DBLPjournals/ml/HanYPTXYS19arXiv1802.09172OpenAlexW2964150070WikidataQ62039105 ScholiaQ62039105MaRDI QIDQ2425234FDOQ2425234

Qiang Yang, Ivor W. Tsang, Yuangang Pan, Quanming Yao, Bo Han, Masashi Sugiyama, Xiaokui Xiao

Publication date: 26 June 2019

Published in: Machine Learning (Search for Journal in Brave)

Abstract: Modern machine learning is migrating to the era of complex models, which requires a plethora of well-annotated data. While crowdsourcing is a promising tool to achieve this goal, existing crowdsourcing approaches barely acquire a sufficient amount of high-quality labels. In this paper, motivated by the "Guess-with-Hints" answer strategy from the Millionaire game show, we introduce the hint-guided approach into crowdsourcing to deal with this challenge. Our approach encourages workers to get help from hints when they are unsure of questions. Specifically, we propose a hybrid-stage setting, consisting of the main stage and the hint stage. When workers face any uncertain question on the main stage, they are allowed to enter the hint stage and look up hints before making any answer. A unique payment mechanism that meets two important design principles for crowdsourcing is developed. Besides, the proposed mechanism further encourages high-quality workers less using hints, which helps identify and assigns larger possible payment to them. Experiments are performed on Amazon Mechanical Turk, which show that our approach ensures a sufficient number of high-quality labels with low expenditure and detects high-quality workers.

Full work available at URL: https://arxiv.org/abs/1802.09172

zbMATH Keywords

game theoryquality controlcomputational modelinghuman factorscrowdsourcing

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Applications of game theory (91A80) Decision theory for games (91A35)

Cites Work

• Algorithmic Game Theory
• An axiomatic characterization of wagering mechanisms
• Robust Plackett-Luce model for \(k\)-ary crowdsourced preferences
• Crowdsourcing with unsure option
• Double or nothing: multiplicative incentive mechanisms for crowdsourcing
• Sequence labeling with multiple annotators
• Learning from multiple annotators with varying expertise
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)

Cited In (1)

• Beyond confusion matrix: learning from multiple annotators with awareness of instance features

This page was built for publication: Millionaire: a hint-guided approach for crowdsourcing

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2425234)