Surrogate regret bounds for generalized classification performance metrics

DOI10.1007/S10994-016-5591-7MaRDI QIDQ2398092zbMATH OpenOpenAlexDBLPWikidataFDO

Authors Wojciech Kotłowski, Krzysztof Dembczyński

Publication date 15 August 2017

Published in Machine Learning (Search for Journal in Brave)

Copyright license Creative Commons Attribution 4.0 International

Full work available at URL https://arxiv.org/abs/1504.07272

zbMATH Keywords

binary classificationmultilabel classificationregret boundsurrogate loss functionF-measureJaccard similaritygeneralized performance metric

Mathematics Subject Classification ID

Classification and discrimination; cluster analysis (statistical aspects) (62H30) Learning and adaptive systems in artificial intelligence (68T05)

Abstract: We consider optimization of generalized performance metrics for binary classification by means of surrogate losses. We focus on a class of metrics, which are linear-fractional functions of the false positive and false negative rates (examples of which include $$-measure, Jaccard similarity coefficient, AM measure, and many others). Our analysis concerns the following two-step procedure. First, a real-valued function $f$ is learned by minimizing a surrogate loss for binary classification on the training sample. It is assumed that the surrogate loss is a strongly proper composite loss function (examples of which include logistic loss, squared-error loss, exponential loss, etc.). Then, given $f$, a threshold $widehatheta$ is tuned on a separate validation sample, by direct optimization of the target performance metric. We show that the regret of the resulting classifier (obtained from thresholding $f$ on $widehatheta$) measured with respect to the target metric is upperbounded by the regret of $f$ measured with respect to the surrogate loss. We also extend our results to cover multilabel classification and provide regret bounds for micro- and macro-averaging measures. Our findings are further analyzed in a computational study on both synthetic and real data sets.

Recommendations

• Composite binary losses
• On the Bayes-optimality of F-measure maximizers
• Surrogate regret bounds for bipartite ranking via strongly proper losses
• Calibrated asymmetric surrogate losses
• Calibration and regret bounds for order-preserving surrogate losses in learning to rank

Cites work

• scientific article; zbMATH DE number 893887 (Why is no real title available?)
• Beyond Fano's inequality: bounds on the optimal \(F\)-score, BER, and cost-sensitive risk and their implications
• Composite binary losses
• Convexity, Classification, and Risk Bounds
• Information, divergence and risk for binary experiments
• Introduction to Information Retrieval
• LIBLINEAR: a library for large linear classification
• Large margin methods for structured and interdependent output variables
• On label dependence and loss minimization in multi-label classification
• On the Bayes-optimality of F-measure maximizers
• On the consistency of multi-label learning
• Surrogate regret bounds for bipartite ranking via strongly proper losses

Cited in

(6)

• Learning with mitigating random consistency from the accuracy measure
• On the Bayes-optimality of F-measure maximizers
• Binarised regression tasks: methods and evaluation metrics
• Optimal rates for nonparametric F-score binary classification via post-processing
• On Loss Functions and Regret Bounds for Multi-Category Classification
• Goal scoring, coherent loss and applications to machine learning

Describes a project that uses

Uses Software

• ElemStatLearn
• LIBLINEAR

This page was built for publication: Surrogate regret bounds for generalized classification performance metrics

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2398092)