Statistical Inference, Occam's Razor, and Statistical Mechanics on the Space of Probability Distributions

DOI10.1162/NECO.1997.9.2.349MaRDI QIDQ3125239zbMATH OpenOpenAlexFDO

Authors Vijay Balasubramanian

Publication date 1 September 1997

Published in Neural Computation (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/cond-mat/9601030

zbMATH Keywords

learningstatistical mechanicsOccam's razoruniform priorspace of probability distributionslow-temperature expansionsBayesian posterior probabilityindex of complexityindistinguishability of distributionsJeffreys' prior distributionparametric model selectionrazor

Mathematics Subject Classification ID

Foundations and philosophical topics in statistics (62A01) Applications of statistical mechanics to specific types of physical systems (82D99)

Abstract: The task of parametric model selection is cast in terms of a statistical mechanics on the space of probability distributions. Using the techniques of low-temperature expansions, we arrive at a systematic series for the Bayesian posterior probability of a model family that significantly extends known results in the literature. In particular, we arrive at a precise understanding of how Occam's Razor, the principle that simpler models should be preferred until the data justifies more complex models, is automatically embodied by probability theory. These results require a measure on the space of model parameters and we derive and discuss an interpretation of Jeffreys' prior distribution as a uniform prior over the distributions indexed by a family. Finally, we derive a theoretical index of the complexity of a parametric family relative to some true distribution that we call the {it razor} of the model. The form of the razor immediately suggests several interesting questions in the theory of learning that can be studied using the techniques of statistical mechanics.

Recommendations

• scientific article; zbMATH DE number 4005258
• On the foundations of statistical mechanics: ergodicity, many degrees of freedom and inference
• Probability and logical structure of statistical theories
• scientific article; zbMATH DE number 1114407
• Observational nonidentifiability, generalized likelihood and free energy
• scientific article; zbMATH DE number 890740
• Statistical Inference Based on the Possibility and Belief Measures
• scientific article; zbMATH DE number 1124538
• On Some Principles of Statistical Inference

Cites work

• Information-theoretic asymptotics of Bayes methods
• Minimum complexity density estimation
• Stochastic complexity and modeling
• Universal coding, information, prediction, and estimation

Cited in

(39)

• On the computation of entropy prior complexity and marginal prior distribution for the Bernoulli model
• How Many Clusters? An Information-Theoretic Perspective
• Objective Bayesian estimation for the differential entropy measure under generalized half-normal distribution
• Fluctuation-Dissipation Theorem and Models of Learning
• Confidence intervals, significance values, maximum likelihood estimates, etc. sharpened into Occam's razors
• A Fisher-Rao metric for curves using the information in edges
• Counting probability distributions: Differential geometry and model selection
• On the complexity of logistic regression models
• Coincidences and estimation of entropies of random variables with large cardinalities
• scientific article; zbMATH DE number 1883494 (Why is no real title available?)
• Application of the Fisher-Rao metric to structure detection
• Discrepancy risk model selection test theory for comparing possibly misspecified or nonnested models
• Bayes factors: Prior sensitivity and model generalizability
• Bayesian feature selection with strongly regularizing priors maps to the Ising model
• Selecting amongst multinomial models: an apologia for normalized maximum likelihood
• Complexity through nonextensivity
• Harold Jeffreys's \textit{Theory of probability} revisited
• Theoretical investigations of an information geometric approach to complexity
• Comparative noninformativities of quantum priors based on monotone metrics
• Bayesian maximum entropy based algorithm for digital X-ray mammogram processing
• The flexibility of models of recognition memory: the case of confidence ratings
• Cooperation, competition and the emergence of criticality in communities of adaptive systems
• Relative entropy and proximity of quantum field theories
• An automatic Ockham's razor for Bayesians?
• An explanatory rationale for priors sharpened into Occam's razors
• Estimating Entropy Rates with Bayesian Confidence Intervals
• A Note on the Applied Use of MDL Approximations
• Minimum message length inference of the Poisson and geometric models using heavy-tailed prior distributions
• Schwarz, Wallace, and Rissanen: Intertwining Themes in Theories of Model Selection
• The flexibility of models of recognition memory: an analysis by the minimum-description length principle
• scientific article; zbMATH DE number 922658 (Why is no real title available?)
• Predictability, complexity, and learning
• Functional uniform priors for nonlinear modeling
• A quantitative Occam's razor
• Marginal Likelihood Computation for Model Selection and Hypothesis Testing: An Extensive Review
• An empirical study of minimum description length model selection with infinite parametric complexity
• Latent Features in Similarity Judgments: A Nonparametric Bayesian Approach
• Model selection by normalized maximum likelihood
• COSMOLOGICAL MODEL SELECTION: STATISTICS AND PHYSICS

This page was built for publication: Statistical Inference, Occam's Razor, and Statistical Mechanics on the Space of Probability Distributions

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q3125239)