The no-free-lunch theorems of supervised learning

DOI10.1007/S11229-021-03233-1MaRDI QIDQ6187752zbMATH OpenOpenAlexWikidataFDO

Authors Tom F. Sterkenburg, Peter D. Grünwald

Publication date 1 February 2024

Published in Synthese (Search for Journal in Brave)

Copyright license Creative Commons Attribution 4.0 International

Full work available at URL https://arxiv.org/abs/2202.04513

zbMATH Keywords

machine learningproblem of inductionno-free-lunch theorem

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Foundations and philosophical topics in statistics (62A01)

Abstract: The no-free-lunch theorems promote a skeptical conclusion that all possible machine learning algorithms equally lack justification. But how could this leave room for a learning theory, that shows that some algorithms are better than others? Drawing parallels to the philosophy of induction, we point out that the no-free-lunch results presuppose a conception of learning algorithms as purely data-driven. On this conception, every algorithm must have an inherent inductive bias, that wants justification. We argue that many standard learning algorithms should rather be understood as model-dependent: in each application they also require for input a model, representing a bias. Generic algorithms themselves, they can be given a model-relative justification.

Recommendations

• What Is Important About the No Free Lunch Theorems?
• No free lunch versus Occam's razor in supervised learning
• Determination and the no-free-lunch paradox
• A ``no free lunch tutorial: sharpened and focused no free lunch

Cites work

• scientific article; zbMATH DE number 6011007 (Why is no real title available?)
• scientific article; zbMATH DE number 96338 (Why is no real title available?)
• scientific article; zbMATH DE number 1332320 (Why is no real title available?)
• scientific article; zbMATH DE number 1098878 (Why is no real title available?)
• scientific article; zbMATH DE number 1179314 (Why is no real title available?)
• scientific article; zbMATH DE number 914840 (Why is no real title available?)
• scientific article; zbMATH DE number 3327941 (Why is no real title available?)
• scientific article; zbMATH DE number 3060434 (Why is no real title available?)
• Absolutely no free lunches!
• Advanced Lectures on Machine Learning
• Black box optimization, machine learning, and no-free lunch theorems
• Can the strengths of AIC and BIC be shared? A conflict between model indentification and regression estimation
• Catching up Faster by Switching Sooner: A Predictive Approach to Adaptive Estimation with an Application to the AIC–BIC Dilemma
• Convergence rates of posterior distributions.
• Determination and the no-free-lunch paradox
• Fast rates for general unbounded loss functions: from ERM to generalized Bayes
• Inconsistency of Bayesian inference for misspecified linear models, and a proposal for repairing it
• Inductive logic
• Local induction
• Mechanizing induction
• Minimum description length revisited
• No free lunch versus Occam's razor in supervised learning
• Nonparametric Bayesian model selection and averaging
• On the marginal likelihood and cross-validation
• Pattern classification.
• Pattern recognition and machine learning.
• Present Position and Potential Developments: Some Personal Views: Statistical Theory: The Prequential Approach
• Putnam's diagonal argument and the impossibility of a universal learning machine
• Simple explanation of the no-free-lunch theorem and its implications
• Statistical learning theory: models, concepts, and results
• Suboptimal behavior of Bayes and MDL in classification under misspecification
• The Oxford handbook of probability and philosophy
• Understanding machine learning. From theory to algorithms
• Would “Direct Realism” Resolve the Classical Problem of Induction?

This page was built for publication: The no-free-lunch theorems of supervised learning

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6187752)