Entropy and the Law of Small Numbers

DOI10.1109/TIT.2004.840861zbMATH Open1297.94016arXivmath/0211020OpenAlexW3106311672WikidataQ60522099 ScholiaQ60522099MaRDI QIDQ3547166FDOQ3547166

Ioannis Kontoyiannis, Peter Harremoës, Oliver Johnson

Publication date: 21 December 2008

Published in: IEEE Transactions on Information Theory (Search for Journal in Brave)

Abstract: Two new information-theoretic methods are introduced for establishing Poisson approximation inequalities. First, using only elementary information-theoretic techniques it is shown that, when $S_n=su{m}_{i=1}^n{X}_i$ is the sum of the (possibly dependent) binary random variables $X_1,X_2,\mathrm{...},X_n$, with $E(X_i)=p_i$ and $E(S_n)=la$, then �en D(P_{S_n}|Pol)leq sum_{i=1}^n p_i^2 + Big[sum_{i=1}^nH(X_i) - H(X_1,X_2,..., X_n)Big], een where $D(P_{S_n}|Po(la))$ is the relative entropy between the distribution of $S_n$ and the Poisson($la$) distribution. The first term in this bound measures the individual smallness of the $X_i$ and the second term measures their dependence. A general method is outlined for obtaining corresponding bounds when approximating the distribution of a sum of general discrete random variables by an infinitely divisible distribution. Second, in the particular case when the $X_i$ are independent, the following sharper bound is established, �en D(P_{S_n}|Pol)leq frac{1}{lambda} sum_{i=1}^n frac{p_i^3}{1-p_i}, % label{eq:abs2} een and it is also generalized to the case when the $X_i$ are general integer-valued random variables. Its proof is based on the derivation of a subadditivity property for a new discrete version of the Fisher information, and uses a recent logarithmic Sobolev inequality for the Poisson distribution.

Full work available at URL: https://arxiv.org/abs/math/0211020

zbMATH Keywords

total variationPoisson approximationFisher informationlogarithmic Sobolev inequalitysubadditivitylaw of small numbersconvergence in relative entropy

Mathematics Subject Classification ID

Statistical aspects of information-theoretic topics (62B10) Inequalities; stochastic orderings (60E15) Information theory (general) (94A15) Measures of information, entropy (94A17)

Cited In (20)

• Discrete versions of the transport equation and the Shepp-Olkin conjecture
• Strong Log-concavity is Preserved by Convolution
• Log-concavity and the maximum entropy property of the Poisson distribution
• Poisson approximation
• Compact Proofs of Retrievability
• Information in Probability: Another Information-Theoretic Proof of a Finite de Finetti Theorem
• Improved lower bounds on the total variation distance for the Poisson approximation
• A Charlier-Parseval approach to Poisson approximation and its applications
• Information-theoretic convergence of extreme values to the Gumbel distribution
• Majorization and Rényi entropy inequalities via Sperner theory
• Log-concavity and strong log-concavity: a review
• A discrete log-Sobolev inequality under a Bakry-Émery type condition
• The entropic Erdős-Kac limit theorem
• An information-theoretic proof of a finite de Finetti theorem
• Convergence of Markov chains in information divergence
• Entropy Inequalities for Sums in Prime Cyclic Groups
• Entropy and the discrete central limit theorem
• Stam inequality on \(\mathbb Z_n\)
• Log-concavity, ultra-log-concavity, and a maximum entropy property of discrete compound Poisson measures
• Estimation on reliability models of bearing failure data

   Recommendations

• Compact Proofs of Retrievability 👍 👎
• Compound Poisson approximation via information functionals 👍 👎
• Nonuniform bounds in the Poisson approximation with applications to informational distances. II 👍 👎
• Title not available (Why is that?) 👍 👎
• Entropy and thinning of discrete random variables 👍 👎

This page was built for publication: Entropy and the Law of Small Numbers

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q3547166)