Marginal singularity and the benefits of labels in covariate-shift

DOI10.1214/21-AOS2084zbMATH Open1486.62186arXiv1803.01833OpenAlexW2964250041MaRDI QIDQ2073708FDOQ2073708

Guillaume Martinet, Samory Kpotufe

Publication date: 7 February 2022

Published in: The Annals of Statistics (Search for Journal in Brave)

Abstract: We present new minimax results that concisely capture the relative benefits of source and target labeled data, under covariate-shift. Namely, we show that the benefits of target labels are controlled by a transfer-exponent $gamma$ that encodes how singular Q is locally w.r.t. P, and interestingly allows situations where transfer did not seem possible under previous insights. In fact, our new minimax analysis - in terms of $gamma$ - reveals a continuum of regimes ranging from situations where target labels have little benefit, to regimes where target labels dramatically improve classification. We then show that a recently proposed semi-supervised procedure can be extended to adapt to unknown $gamma$, and therefore requests labels only when beneficial, while achieving minimax transfer rates.

Full work available at URL: https://arxiv.org/abs/1803.01833

zbMATH Keywords

transfer learningcovariate shiftnonparametric classificationdomain adaptation

Mathematics Subject Classification ID

Nonparametric regression and quantile regression (62G08) Classification and discrimination; cluster analysis (statistical aspects) (62H30) Learning and adaptive systems in artificial intelligence (68T05)

Cites Work

• Title not available (Why is that?)
• All of Nonparametric Statistics
• Local nearest neighbour classification with applications to semi-supervised learning
• Transfer learning for nonparametric classification: minimax rate and adaptive classifier
• Understanding Machine Learning
• Optimal weighted nearest neighbour classifiers
• A distribution-free theory of nonparametric regression
• Introduction to nonparametric estimation
• Optimal spatial adaptation to inhomogeneous smoothness: An approach based on kernel estimates with variable bandwidth selectors
• Fast learning rates for plug-in classifiers
• A theory of learning from different domains
• Density ratio estimation in machine learning. Foreword by Thomas G. Dietterich
• Rates of convergence of nearest neighbor estimation under arbitrary sampling
• New Analysis and Algorithm for Learning with Drifting Distributions
• On spatially adaptive estimation of nonparametric regression
• Bounding the Vapnik-Chervonenkis dimension of concept classes parameterized by real numbers
• A theory of transfer learning with applications to active learning
• On the Hardness of Domain Adaptation and the Utility of Unlabeled Target Samples

Cited In (5)

• Local convergence rates of the nonparametric least squares estimator with applications to transfer learning
• A no-free-lunch theorem for multitask learning
• Transfer learning for contextual multi-armed bandits
• Tnn: a transfer learning classifier based on weighted nearest neighbors
• Optimally tackling covariate shift in RKHS-based nonparametric regression

Recommendations

• Conditional variance penalties and domain shift robustness 👍 👎
• Input-dependent estimation of generalization error under covariate shift 👍 👎
• Discriminative learning under covariate shift 👍 👎
• Training classifiers under covariate shift by constructing the maximum consistent distribution subset 👍 👎
• A view of margin losses as regularizers of probability estimates 👍 👎
• Learning Coefficients of Layered Models When the True Distribution Mismatches the Singularities 👍 👎
• On semi-supervised linear regression in covariate shift problems 👍 👎
• High dimensional binary classification under label shift: phase transition and regularization 👍 👎

This page was built for publication: Marginal singularity and the benefits of labels in covariate-shift

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2073708)