From average case complexity to improper learning complexity

DOI10.1145/2591796.2591820zbMATH Open1315.68164arXiv1311.2272OpenAlexW2019120673MaRDI QIDQ5259579FDOQ5259579

Shai Shalev-Shwartz, Nathan Linial, Amit Daniely

Publication date: 26 June 2015

Published in: Proceedings of the forty-sixth annual ACM symposium on Theory of computing (Search for Journal in Brave)

Abstract: The basic problem in the PAC model of computational learning theory is to determine which hypothesis classes are efficiently learnable. There is presently a dearth of results showing hardness of learning problems. Moreover, the existing lower bounds fall short of the best known algorithms. The biggest challenge in proving complexity results is to establish hardness of {em improper learning} (a.k.a. representation independent learning).The difficulty in proving lower bounds for improper learning is that the standard reductions from $mathbfNP$-hard problems do not seem to apply in this context. There is essentially only one known approach to proving lower bounds on improper learning. It was initiated in (Kearns and Valiant 89) and relies on cryptographic assumptions. We introduce a new technique for proving hardness of improper learning, based on reductions from problems that are hard on average. We put forward a (fairly strong) generalization of Feige's assumption (Feige 02) about the complexity of refuting random constraint satisfaction problems. Combining this assumption with our new technique yields far reaching implications. In particular, 1. Learning $mathrmDNF$'s is hard. 2. Agnostically learning halfspaces with a constant approximation ratio is hard. 3. Learning an intersection of $omega(1)$ halfspaces is hard.

Full work available at URL: https://arxiv.org/abs/1311.2272

zbMATH Keywords

average case complexityhalfspacesDNFsCSP problemshardness of improper learningresolution lower bounds

Mathematics Subject Classification ID

Analysis of algorithms and problem complexity (68Q25) Computational learning theory (68Q32) Computational difficulty of problems (lower bounds, completeness, difficulty of approximation, etc.) (68Q17)

Cites Work

• Theory of Cryptography
• The geometry of differential privacy: the small database and approximate cases
• On the geometry of differential privacy
• Interactive privacy via the median mechanism
• The price of privately releasing contingency tables and the spectra of random matrices with correlated rows
• Lower Bounds in Differential Privacy
• Iterative Constructions and Private Data Release
• Differential Privacy and the Fat-Shattering Dimension of Linear Queries
• Our Data, Ourselves: Privacy Via Distributed Noise Generation
• On the complexity of differentially private data release
• Title not available (Why is that?)
• Collusion-secure fingerprinting for digital data
• Advances in Cryptology – CRYPTO 2004
• Title not available (Why is that?)
• New Efficient Attacks on Statistical Disclosure Control Mechanisms
• Bounds on the Sample Complexity for Private Learning and Private Data Release
• Answering n _{2+o(1)} counting queries with differential privacy is hard
• Characterizing the sample complexity of private learners
• Faster Algorithms for Privately Releasing Marginals
• Faster private release of marginals on small databases
• Private Learning and Sanitization: Pure vs. Approximate Differential Privacy
• Privately releasing conjunctions and the statistical query barrier
• Efficient algorithms for privately releasing marginals via convex relaxations

Cited In (9)

• Title not available (Why is that?)
• Title not available (Why is that?)
• Limits of preprocessing
• Learning infinite-word automata with loop-index queries
• Constructing concrete hard instances of the maximum independent set problem
• Fast Pseudorandom Functions Based on Expander Graphs
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)

Uses Software

• SuLQ

   Recommendations

• The complexity of properly learning simple concept classes 👍 👎
• Complexity theoretic limitations on learning halfspaces 👍 👎
• Complexity theoretic hardness results for query learning 👍 👎
• Title not available (Why is that?) 👍 👎
• On agnostic learning of parities, monomials, and halfspaces 👍 👎

This page was built for publication: From average case complexity to improper learning complexity

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5259579)