Millionaire: a hint-guided approach for crowdsourcing

DOI10.1007/S10994-018-5766-5MaRDI QIDQ2425234zbMATH OpenOpenAlexDBLPWikidataFDO

Authors Bo Han, Quanming Yao, Yuangang Pan, Ivor W. Tsang, Xiaokui Xiao, Qiang Yang, Masashi Sugiyama

Publication date 26 June 2019

Published in Machine Learning (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1802.09172

zbMATH Keywords

game theoryquality controlcomputational modelinghuman factorscrowdsourcing

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Applications of game theory (91A80) Decision theory for games (91A35)

Abstract: Modern machine learning is migrating to the era of complex models, which requires a plethora of well-annotated data. While crowdsourcing is a promising tool to achieve this goal, existing crowdsourcing approaches barely acquire a sufficient amount of high-quality labels. In this paper, motivated by the "Guess-with-Hints" answer strategy from the Millionaire game show, we introduce the hint-guided approach into crowdsourcing to deal with this challenge. Our approach encourages workers to get help from hints when they are unsure of questions. Specifically, we propose a hybrid-stage setting, consisting of the main stage and the hint stage. When workers face any uncertain question on the main stage, they are allowed to enter the hint stage and look up hints before making any answer. A unique payment mechanism that meets two important design principles for crowdsourcing is developed. Besides, the proposed mechanism further encourages high-quality workers less using hints, which helps identify and assigns larger possible payment to them. Experiments are performed on Amazon Mechanical Turk, which show that our approach ensures a sufficient number of high-quality labels with low expenditure and detects high-quality workers.

Recommendations

• Double or nothing: multiplicative incentive mechanisms for crowdsourcing
• Statistical Decision Making for Optimal Budget Allocation in Crowd Labeling
• More for less: adaptive labeling payments in online labor markets
• Crowdsourcing with unsure option
• Learning Complex Concepts Using Crowdsourcing: A Bayesian Approach

Cites work

• scientific article; zbMATH DE number 6982911 (Why is no real title available?)
• Algorithmic Game Theory
• An axiomatic characterization of wagering mechanisms
• Crowdsourcing with unsure option
• Double or nothing: multiplicative incentive mechanisms for crowdsourcing
• Eliminating spammers and ranking annotators for crowdsourced labeling tasks
• Learning from multiple annotators with varying expertise
• Robust Plackett-Luce model for \(k\)-ary crowdsourced preferences
• Sequence labeling with multiple annotators
• Spectral methods meet EM: a provably optimal algorithm for crowdsourcing

Cited in

(6)

• Credit and quality intelligent learning based multi-armed bandit scheme for unknown worker selection in multimedia MCS
• Learning from crowds with sparse and imbalanced annotations
• Beyond confusion matrix: learning from multiple annotators with awareness of instance features
• Crowdsourcing with unsure option
• More for less: adaptive labeling payments in online labor markets
• Double or nothing: multiplicative incentive mechanisms for crowdsourcing

This page was built for publication: Millionaire: a hint-guided approach for crowdsourcing

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2425234)