Millionaire: a hint-guided approach for crowdsourcing

DOI10.1007/S10994-018-5766-5zbMATH Open1493.68301DBLPjournals/ml/HanYPTXYS19arXiv1802.09172OpenAlexW2964150070WikidataQ62039105 ScholiaQ62039105MaRDI QIDQ2425234FDOQ2425234

Authors: Bo Han, Quanming Yao, Yuangang Pan, Ivor W. Tsang, Xiaokui Xiao, Qiang Yang, Masashi Sugiyama 

Publication date: 26 June 2019

Published in: Machine Learning (Search for Journal in Brave)

Abstract: Modern machine learning is migrating to the era of complex models, which requires a plethora of well-annotated data. While crowdsourcing is a promising tool to achieve this goal, existing crowdsourcing approaches barely acquire a sufficient amount of high-quality labels. In this paper, motivated by the "Guess-with-Hints" answer strategy from the Millionaire game show, we introduce the hint-guided approach into crowdsourcing to deal with this challenge. Our approach encourages workers to get help from hints when they are unsure of questions. Specifically, we propose a hybrid-stage setting, consisting of the main stage and the hint stage. When workers face any uncertain question on the main stage, they are allowed to enter the hint stage and look up hints before making any answer. A unique payment mechanism that meets two important design principles for crowdsourcing is developed. Besides, the proposed mechanism further encourages high-quality workers less using hints, which helps identify and assigns larger possible payment to them. Experiments are performed on Amazon Mechanical Turk, which show that our approach ensures a sufficient number of high-quality labels with low expenditure and detects high-quality workers.

Full work available at URL: https://arxiv.org/abs/1802.09172

Recommendations

• Double or nothing: multiplicative incentive mechanisms for crowdsourcing
• Statistical Decision Making for Optimal Budget Allocation in Crowd Labeling
• More for less: adaptive labeling payments in online labor markets
• Crowdsourcing with unsure option
• Learning Complex Concepts Using Crowdsourcing: A Bayesian Approach

zbMATH Keywords

game theoryquality controlcomputational modelinghuman factorscrowdsourcing

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Applications of game theory (91A80) Decision theory for games (91A35)

Cites Work

• Algorithmic Game Theory
• Title not available (Why is that?)
• An axiomatic characterization of wagering mechanisms
• Robust Plackett-Luce model for \(k\)-ary crowdsourced preferences
• Crowdsourcing with unsure option
• Double or nothing: multiplicative incentive mechanisms for crowdsourcing
• Spectral methods meet EM: a provably optimal algorithm for crowdsourcing
• Sequence labeling with multiple annotators
• Eliminating spammers and ranking annotators for crowdsourced labeling tasks
• Learning from multiple annotators with varying expertise

Cited In (6)

• More for less: adaptive labeling payments in online labor markets
• Crowdsourcing with unsure option
• Learning from crowds with sparse and imbalanced annotations
• Double or nothing: multiplicative incentive mechanisms for crowdsourcing
• Credit and quality intelligent learning based multi-armed bandit scheme for unknown worker selection in multimedia MCS
• Beyond confusion matrix: learning from multiple annotators with awareness of instance features

This page was built for publication: Millionaire: a hint-guided approach for crowdsourcing

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2425234)