Agnostic Learning of Monomials by Halfspaces Is Hard
From MaRDI portal
Publication:4910575
DOI10.1137/120865094zbMath1261.68063arXiv1012.0729OpenAlexW2038663599MaRDI QIDQ4910575
Vitaly Feldman, Yi Wu, Prasad Raghavendra, Venkatesan Guruswami
Publication date: 19 March 2013
Published in: SIAM Journal on Computing, 2009 50th Annual IEEE Symposium on Foundations of Computer Science (Search for Journal in Brave)
Abstract: We prove the following strong hardness result for learning: Given a distribution of labeled examples from the hypercube such that there exists a monomial consistent with $(1-eps)$ of the examples, it is NP-hard to find a halfspace that is correct on $(1/2+eps)$ of the examples, for arbitrary constants $eps > 0$. In learning theory terms, weak agnostic learning of monomials is hard, even if one is allowed to output a hypothesis from the much bigger concept class of halfspaces. This hardness result subsumes a long line of previous results, including two recent hardness results for the proper learning of monomials and halfspaces. As an immediate corollary of our result we show that weak agnostic learning of decision lists is NP-hard. Our techniques are quite different from previous hardness proofs for learning. We define distributions on positive and negative examples for monomials whose first few moments match. We use the invariance principle to argue that regular halfspaces (all of whose coefficients have small absolute value relative to the total $ell_2$ norm) cannot distinguish between distributions whose first few moments match. For highly non-regular subspaces, we use a structural lemma from recent work on fooling halfspaces to argue that they are ``junta-like and one can zero out all but the top few coefficients without affecting the performance of the halfspace. The top few coefficients form the natural list decoding of a halfspace in the context of dictatorship tests/Label Cover reductions. We note that unlike previous invariance principle based proofs which are only known to give Unique-Games hardness, we are able to reduce from a version of Label Cover problem that is known to be NP-hard. This has inspired follow-up work on bypassing the Unique Games conjecture in some optimal geometric inapproximability results.
Full work available at URL: https://arxiv.org/abs/1012.0729
Computational learning theory (68Q32) Computational difficulty of problems (lower bounds, completeness, difficulty of approximation, etc.) (68Q17)
Related Items (19)
Tikhonov, Ivanov and Morozov regularization for support vector machine learning ⋮ Nearly Optimal Solutions for the Chow Parameters Problem and Low-Weight Approximation of Halfspaces ⋮ Regularizing conjunctive features for classification ⋮ Complexity and approximability of parameterized MAX-CSPs ⋮ A Fisher consistent multiclass loss function with variable margin on positive examples ⋮ On the hardness of learning intersections of two halfspaces ⋮ A nonlinear kernel SVM Classifier via \(L_{0/1}\) soft-margin loss with classification performance ⋮ \(\alpha\)QBoost: an iteratively weighted adiabatic trained classifier ⋮ Robust classification via MOM minimization ⋮ Kernel variable selection for multicategory support vector machines ⋮ Three candidate plurality is stablest for small correlations ⋮ A moment-matching approach to testable learning and a new characterization of Rademacher complexity ⋮ Statistical computational learning ⋮ Gaussian bounds for noise correlation of resilient functions ⋮ The Quest for Strong Inapproximability Results with Perfect Completeness ⋮ Unnamed Item ⋮ Provably training overparameterized neural network classifiers with non-convex constraints ⋮ Quadratic Convergence of Smoothing Newton's Method for 0/1 Loss Optimization ⋮ Tight approximation bounds for maximum multi-coverage
This page was built for publication: Agnostic Learning of Monomials by Halfspaces Is Hard
