Statistical Inference, Occam's Razor, and Statistical Mechanics on the Space of Probability Distributions

DOI10.1162/NECO.1997.9.2.349zbMATH Open0870.62006arXivcond-mat/9601030OpenAlexW2124641450MaRDI QIDQ3125239FDOQ3125239

Authors: Vijay Balasubramanian 

Publication date: 1 September 1997

Published in: Neural Computation (Search for Journal in Brave)

Abstract: The task of parametric model selection is cast in terms of a statistical mechanics on the space of probability distributions. Using the techniques of low-temperature expansions, we arrive at a systematic series for the Bayesian posterior probability of a model family that significantly extends known results in the literature. In particular, we arrive at a precise understanding of how Occam's Razor, the principle that simpler models should be preferred until the data justifies more complex models, is automatically embodied by probability theory. These results require a measure on the space of model parameters and we derive and discuss an interpretation of Jeffreys' prior distribution as a uniform prior over the distributions indexed by a family. Finally, we derive a theoretical index of the complexity of a parametric family relative to some true distribution that we call the {it razor} of the model. The form of the razor immediately suggests several interesting questions in the theory of learning that can be studied using the techniques of statistical mechanics.

Full work available at URL: https://arxiv.org/abs/cond-mat/9601030

Recommendations

• scientific article; zbMATH DE number 4005258
• On the foundations of statistical mechanics: ergodicity, many degrees of freedom and inference
• Probability and logical structure of statistical theories
• scientific article; zbMATH DE number 1114407
• Observational nonidentifiability, generalized likelihood and free energy
• scientific article; zbMATH DE number 890740
• Statistical Inference Based on the Possibility and Belief Measures
• scientific article; zbMATH DE number 1124538
• On Some Principles of Statistical Inference

zbMATH Keywords

learningstatistical mechanicsOccam's razoruniform priorspace of probability distributionslow-temperature expansionsBayesian posterior probabilityindex of complexityindistinguishability of distributionsJeffreys' prior distributionparametric model selectionrazor

Mathematics Subject Classification ID

Foundations and philosophical topics in statistics (62A01) Applications of statistical mechanics to specific types of physical systems (82D99)

Cites Work

• Minimum complexity density estimation
• Stochastic complexity and modeling
• Information-theoretic asymptotics of Bayes methods
• Universal coding, information, prediction, and estimation

Cited In (39)

• On the computation of entropy prior complexity and marginal prior distribution for the Bernoulli model
• How Many Clusters? An Information-Theoretic Perspective
• Fluctuation-Dissipation Theorem and Models of Learning
• Objective Bayesian estimation for the differential entropy measure under generalized half-normal distribution
• Confidence intervals, significance values, maximum likelihood estimates, etc. sharpened into Occam's razors
• Counting probability distributions: Differential geometry and model selection
• On the complexity of logistic regression models
• A Fisher-Rao metric for curves using the information in edges
• Title not available (Why is that?)
• Coincidences and estimation of entropies of random variables with large cardinalities
• Application of the Fisher-Rao metric to structure detection
• Bayes factors: Prior sensitivity and model generalizability
• Discrepancy risk model selection test theory for comparing possibly misspecified or nonnested models
• Bayesian feature selection with strongly regularizing priors maps to the Ising model
• Selecting amongst multinomial models: an apologia for normalized maximum likelihood
• Complexity through nonextensivity
• Theoretical investigations of an information geometric approach to complexity
• Harold Jeffreys's \textit{Theory of probability} revisited
• Comparative noninformativities of quantum priors based on monotone metrics
• Bayesian maximum entropy based algorithm for digital X-ray mammogram processing
• The flexibility of models of recognition memory: the case of confidence ratings
• Cooperation, competition and the emergence of criticality in communities of adaptive systems
• Relative entropy and proximity of quantum field theories
• An automatic Ockham's razor for Bayesians?
• An explanatory rationale for priors sharpened into Occam's razors
• A Note on the Applied Use of MDL Approximations
• Estimating Entropy Rates with Bayesian Confidence Intervals
• Schwarz, Wallace, and Rissanen: Intertwining Themes in Theories of Model Selection
• Minimum message length inference of the Poisson and geometric models using heavy-tailed prior distributions
• The flexibility of models of recognition memory: an analysis by the minimum-description length principle
• Title not available (Why is that?)
• Predictability, complexity, and learning
• Functional uniform priors for nonlinear modeling
• A quantitative Occam's razor
• Marginal Likelihood Computation for Model Selection and Hypothesis Testing: An Extensive Review
• An empirical study of minimum description length model selection with infinite parametric complexity
• Latent Features in Similarity Judgments: A Nonparametric Bayesian Approach
• Model selection by normalized maximum likelihood
• COSMOLOGICAL MODEL SELECTION: STATISTICS AND PHYSICS

This page was built for publication: Statistical Inference, Occam's Razor, and Statistical Mechanics on the Space of Probability Distributions

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q3125239)