Minimum description length revisited

DOI10.1142/S2661335219300018zbMATH Open1476.62020arXiv1908.08484OpenAlexW3098044966WikidataQ109278499 ScholiaQ109278499MaRDI QIDQ4997077FDOQ4997077

Authors: Peter D. Grünwald, Teemu Roos 

Publication date: 28 June 2021

Published in: International Journal of Mathematics for Industry (Search for Journal in Brave)

Abstract: This is an up-to-date introduction to and overview of the Minimum Description Length (MDL) Principle, a theory of inductive inference that can be applied to general problems in statistics, machine learning and pattern recognition. While MDL was originally based on data compression ideas, this introduction can be read without any knowledge thereof. It takes into account all major developments since 2007, the last time an extensive overview was written. These include new methods for model selection and averaging and hypothesis testing, as well as the first completely general definition of {em MDL estimators}. Incorporating these developments, MDL can be seen as a powerful extension of both penalized likelihood and Bayesian approaches, in which penalization functions and prior distributions are replaced by more general luckiness functions, average-case methodology is replaced by a more robust worst-case approach, and in which methods classically viewed as highly distinct, such as AIC vs BIC and cross-validation vs Bayes can, to a large extent, be viewed from a unified perspective.

Full work available at URL: https://arxiv.org/abs/1908.08484

Recommendations

• Model Selection and the Principle of Minimum Description Length
• Model Selection and Testing by the MDL Principle
• Model selection based on minimum description length
• Model Selection Using the Minimum Description Length Principle
• Information and complexity in statistical modeling.

zbMATH Keywords

model selectionpenalized estimationuniversal predictionminimum description length (MDL) principle

Mathematics Subject Classification ID

Statistical aspects of information-theoretic topics (62B10) Foundations and philosophical topics in statistics (62A01) Statistical ranking and selection procedures (62F07) Inference from stochastic processes and prediction (62M20)

Cites Work

• The elements of statistical learning. Data mining, inference, and prediction
• Krimp: mining itemsets that compress
• Asymptotic equivalence of Bayes cross validation and widely applicable information criterion in singular learning theory
• Title not available (Why is that?)
• Prediction, Learning, and Games
• Learning Bayesian networks: The combination of knowledge and statistical data
• Present Position and Potential Developments: Some Personal Views: Statistical Theory: The Prequential Approach
• Can the strengths of AIC and BIC be shared? A conflict between model indentification and regression estimation
• Almost the best of three worlds: risk, consistency and optional stopping for the switch criterion in nested model selection
• Bayes Factors
• Probabilistic graphical models.
• Harold Jeffreys's default Bayes factor hypothesis tests: explanation, extension, and application in psychology
• Understanding machine learning. From theory to algorithms
• Inconsistency of Bayesian inference for misspecified linear models, and a proposal for repairing it
• Minimum complexity density estimation
• Information and complexity in statistical modeling.
• Modeling by shortest data description
• A widely applicable Bayesian information criterion
• An Information Measure for Classification
• Universal coding, information, prediction, and estimation
• Title not available (Why is that?)
• The minimum description length principle in coding and modeling
• Fisher information and stochastic complexity
• Bayes factors and marginal distributions in invariant situations
• Unified Conditional Frequentist and Bayesian Testing of Composite Hypotheses
• Minimum description length induction, Bayesianism, and Kolmogorov complexity
• Counting probability distributions: Differential geometry and model selection
• Catching up Faster by Switching Sooner: A Predictive Approach to Adaptive Estimation with an Application to the AIC–BIC Dilemma
• Title not available (Why is that?)
• The geometry of proper scoring rules
• PAC-Bayesian stochastic model selection
• On model selection, Bayesian networks, and the Fisher information integral
• From \(\varepsilon\)-entropy to KL-entropy: analysis of minimum information complexity density estima\-tion
• Title not available (Why is that?)
• Jeffreys versus Shtarkov distributions associated with some natural exponential families
• A linear-time algorithm for computing the multinomial stochastic complexity
• Finite-Sample Risk Bounds for Maximum Likelihood Estimation With Arbitrary Penalties
• Summarizing and understanding large graphs
• Information-theoretic upper and lower bounds for statistical estimation
• Learning Theory
• Flat Minima
• High-dimensional penalty selection via minimum description length principle
• Safe probability
• The safe Bayesian. Learning the learning rate via the mixability gap
• PAC-MDL bounds.
• Minimax optimal Bayes mixtures for memoryless sources over large alphabets
• Efficient Computation of Normalized Maximum Likelihood Codes for Gaussian Mixture Models With Its Applications to Clustering
• Model selection by sequentially normalized least squares
• Subset selection in linear regression using sequentially normalized least squares: asymptotic theory
• Relations Between the Conditional Normalized Maximum Likelihood Distributions and the Latent Information Priors
• Minimax Pointwise Redundancy for Memoryless Models Over Large Alphabets
• Achievability of asymptotic minimax regret by horizon-dependent and horizon-independent strategies

Cited In (16)

• On the parametric description of log-growth rates of Romanian city sizes
• Robust subgroup discovery. Discovering subgroup lists using MDL
• The whole and the parts: the minimum description length principle and the a-contrario framework
• The minimum description length principle for pattern mining: a survey
• Title not available (Why is that?)
• Title not available (Why is that?)
• An extended minimum description length (MDL) penalty function for binomial proportions
• Minimum Description Length Principle for Fat-Tailed Distributions
• Information and complexity in statistical modeling.
• The no-free-lunch theorems of supervised learning
• Title not available (Why is that?)
• Title not available (Why is that?)
• Explanatory and creative alternatives to the MDL principle
• Title not available (Why is that?)
• Marginal Likelihood Computation for Model Selection and Hypothesis Testing: An Extensive Review
• Unsupervised discretization by two-dimensional MDL-based histogram

Uses Software

• ElemStatLearn
• Krimp

This page was built for publication: Minimum description length revisited

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q4997077)