Entropy and the Law of Small Numbers

DOI10.1109/TIT.2004.840861MaRDI QIDQ3547166zbMATH OpenOpenAlexWikidataFDO

Authors Ioannis Kontoyiannis, Peter Harremoës, Oliver Johnson

Publication date 21 December 2008

Published in IEEE Transactions on Information Theory (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/math/0211020

zbMATH Keywords

total variationPoisson approximationFisher informationlogarithmic Sobolev inequalitysubadditivitylaw of small numbersconvergence in relative entropy

Mathematics Subject Classification ID

Statistical aspects of information-theoretic topics (62B10) Inequalities; stochastic orderings (60E15) Information theory (general) (94A15) Measures of information, entropy (94A17)

Abstract: Two new information-theoretic methods are introduced for establishing Poisson approximation inequalities. First, using only elementary information-theoretic techniques it is shown that, when $S_n=su{m}_{i=1}^n{X}_i$ is the sum of the (possibly dependent) binary random variables $X_1,X_2,\mathrm{...},X_n$, with $E(X_i)=p_i$ and $E(S_n)=la$, then �en D(P_{S_n}|Pol)leq sum_{i=1}^n p_i^2 + Big[sum_{i=1}^nH(X_i) - H(X_1,X_2,..., X_n)Big], een where $D(P_{S_n}|Po(la))$ is the relative entropy between the distribution of $S_n$ and the Poisson($la$) distribution. The first term in this bound measures the individual smallness of the $X_i$ and the second term measures their dependence. A general method is outlined for obtaining corresponding bounds when approximating the distribution of a sum of general discrete random variables by an infinitely divisible distribution. Second, in the particular case when the $X_i$ are independent, the following sharper bound is established, �en D(P_{S_n}|Pol)leq frac{1}{lambda} sum_{i=1}^n frac{p_i^3}{1-p_i}, % label{eq:abs2} een and it is also generalized to the case when the $X_i$ are general integer-valued random variables. Its proof is based on the derivation of a subadditivity property for a new discrete version of the Fisher information, and uses a recent logarithmic Sobolev inequality for the Poisson distribution.

Recommendations

• Compact Proofs of Retrievability
• Compound Poisson approximation via information functionals
• Nonuniform bounds in the Poisson approximation with applications to informational distances. II
• scientific article; zbMATH DE number 4007340
• Entropy and thinning of discrete random variables

Cited in

(25)

• Compound Poisson approximation via information functionals
• Discrete versions of the transport equation and the Shepp-Olkin conjecture
• Log-concavity and the maximum entropy property of the Poisson distribution
• Poisson approximation
• scientific article; zbMATH DE number 4007340 (Why is no real title available?)
• Entropy inequalities for sums in prime cyclic groups
• Compact Proofs of Retrievability
• Improved lower bounds on the total variation distance for the Poisson approximation
• Information in Probability: Another Information-Theoretic Proof of a Finite de Finetti Theorem
• A Charlier-Parseval approach to Poisson approximation and its applications
• Majorization and Rényi entropy inequalities via Sperner theory
• Information-theoretic convergence of extreme values to the Gumbel distribution
• Log-concavity and strong log-concavity: a review
• Entropy and thinning of discrete random variables
• Non-uniform bounds in the Poisson approximation with applications to informational distances. I
• The entropic Erdős-Kac limit theorem
• A discrete log-Sobolev inequality under a Bakry-Émery type condition
• Nonuniform bounds in the Poisson approximation with applications to informational distances. II
• Convergence of Markov chains in information divergence
• An information-theoretic proof of a finite de Finetti theorem
• Stam inequality on Z_n
• Log-concavity, ultra-log-concavity, and a maximum entropy property of discrete compound Poisson measures
• Strong log-concavity is preserved by convolution
• Entropy and the discrete central limit theorem
• Estimation on reliability models of bearing failure data

This page was built for publication: Entropy and the Law of Small Numbers

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q3547166)