Ranking and combining multiple predictors without labeled data

DOI10.1073/PNAS.1219097111zbMATH Open1359.62259arXiv1303.3257OpenAlexW1991137221WikidataQ35086460 ScholiaQ35086460MaRDI QIDQ2962204FDOQ2962204

Authors: Fabio Parisi, Francesco Strino, Boaz Nadler, Yuval Kluger 

Publication date: 16 February 2017

Published in: Proceedings of the National Academy of Sciences (Search for Journal in Brave)

Abstract: In a broad range of classification and decision making problems, one is given the advice or predictions of several classifiers, of unknown reliability, over multiple questions or queries. This scenario is different from the standard supervised setting, where each classifier accuracy can be assessed using available labeled data, and raises two questions: given only the predictions of several classifiers over a large set of unlabeled test data, is it possible to a) reliably rank them; and b) construct a meta-classifier more accurate than most classifiers in the ensemble? Here we present a novel spectral approach to address these questions. First, assuming conditional independence between classifiers, we show that the off-diagonal entries of their covariance matrix correspond to a rank-one matrix. Moreover, the classifiers can be ranked using the leading eigenvector of this covariance matrix, as its entries are proportional to their balanced accuracies. Second, via a linear approximation to the maximum likelihood estimator, we derive the Spectral Meta-Learner (SML), a novel ensemble classifier whose weights are equal to this eigenvector entries. On both simulated and real data, SML typically achieves a higher accuracy than most classifiers in the ensemble and can provide a better starting point than majority voting, for estimating the maximum likelihood solution. Furthermore, SML is robust to the presence of small malicious groups of classifiers designed to veer the ensemble prediction away from the (unknown) ground truth.

Full work available at URL: https://arxiv.org/abs/1303.3257

Recommendations

• Unsupervised evaluation and weighted aggregation of ranked classification predictions
• Context-based unsupervised ensemble learning and feature ranking
• Unsupervised weight parameter estimation method for ensemble learning
• Transductive Methods for the Distributed Ensemble Classification Problem
• The value of agreement a new boosting algorithm

Mathematics Subject Classification ID

Classification and discrimination; cluster analysis (statistical aspects) (62H30) Estimation in multivariate analysis (62H12)

Cites Work

• Handbook of economic forecasting. Volume 1
• The EM algorithm for graphical association models with missing data

Cited In (6)

• Spectral neighbor joining for reconstruction of latent tree models
• Unsupervised evaluation and weighted aggregation of ranked classification predictions
• Context-based unsupervised ensemble learning and feature ranking
• Machine truth serum: a surprisingly popular approach to improving ensemble methods
• Ensembles of change-point detectors: implications for real-time BMI applications
• Unsupervised weight parameter estimation method for ensemble learning

This page was built for publication: Ranking and combining multiple predictors without labeled data

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2962204)