Entropy concentration and the empirical coding game

DOI10.1111/J.1467-9574.2008.00401.XMaRDI QIDQ3525732zbMATH OpenOpenAlexFDO

Authors Peter D. Grünwald

Publication date 18 September 2008

Published in Statistica Neerlandica (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/0809.1017

Mathematics Subject Classification ID

Applications of game theory (91A80) Measures of information, entropy (94A17)

Abstract: We give a characterization of Maximum Entropy/Minimum Relative Entropy inference by providing two `strong entropy concentration' theorems. These theorems unify and generalize Jaynes' `concentration phenomenon' and Van Campenhout and Cover's `conditional limit theorem'. The theorems characterize exactly in what sense a prior distribution Q conditioned on a given constraint, and the distribution P, minimizing the relative entropy D(P ||Q) over all distributions satisfying the constraint, are `close' to each other. We then apply our theorems to establish the relationship between entropy concentration and a game-theoretic characterization of Maximum Entropy Inference due to Topsoe and others.

Recommendations

• scientific article; zbMATH DE number 1804111
• Concentration theorems for entropy and free energy
• Game theory, maximum entropy, minimum discrepancy and robust Bayesian decision theory
• Maximum entropy fundamentals
• The constraint rule of the maximum entropy principle

Cites work

• scientific article; zbMATH DE number 45100 (Why is no real title available?)
• scientific article; zbMATH DE number 107482 (Why is no real title available?)
• scientific article; zbMATH DE number 3621626 (Why is no real title available?)
• scientific article; zbMATH DE number 3249395 (Why is no real title available?)
• Game theory, maximum entropy, minimum discrepancy and robust Bayesian decision theory
• Maximum entropy and conditional probability
• Probability Theory
• Sanov property, generalized I-projection and a conditional limit theorem
• Why least squares and maximum entropy? An axiomatic approach to inference for linear inverse problems

Cited in

(8)

• scientific article; zbMATH DE number 3621626 (Why is no real title available?)
• A correspondence principle for relative entropy minimization
• From Winner-Takes-All to Winners-Share-All: Exploiting the Information Capacity in Temporal Codes
• Concentration theorems for entropy and free energy
• The soccer game, bit by bit: an information-theoretic analysis
• Bounds on entropy in a guessing game
• Generalized maximum entropy estimation
• scientific article; zbMATH DE number 1804111 (Why is no real title available?)

This page was built for publication: Entropy concentration and the empirical coding game

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q3525732)