Mutual Information Bounds via Adjacency Events

DOI10.1109/TIT.2016.2609390MaRDI QIDQ2976484zbMATH OpenOpenAlexFDO

Authors Yanjun Han, Or Ordentlich, Ofer Shayevitz

Publication date 28 April 2017

Published in IEEE Transactions on Information Theory (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1507.06296

Mathematics Subject Classification ID

Measures of information, entropy (94A17)

Abstract: The mutual information between two jointly distributed random variables $X$ and $Y$ is a functional of the joint distribution $P_{XY},$ which is sometimes difficult to handle or estimate. A coarser description of the statistical behavior of $(X,Y)$ is given by the marginal distributions $P_X,P_Y$ and the adjacency relation induced by the joint distribution, where $x$ and $y$ are adjacent if $P(x,y)>0$. We derive a lower bound on the mutual information in terms of these entities. The bound is obtained by viewing the channel from $X$ to $Y$ as a probability distribution on a set of possible actions, where an action determines the output for any possible input, and is independently drawn. We also provide an alternative proof based on convex optimization, that yields a generally tighter bound. Finally, we derive an upper bound on the mutual information in terms of adjacency events between the action and the pair $(X,Y)$, where in this case an action $a$ and a pair $(x,y)$ are adjacent if $y=a(x)$. As an example, we apply our bounds to the binary deletion channel and show that for the special case of an i.i.d. input distribution and a range of deletion probabilities, our lower and upper bounds both outperform the best known bounds for the mutual information.

Recommendations

• A Finite-Sample, Distribution-Free, Probabilistic Lower Bound on Mutual Information
• Mutual information and the F-theorem
• Correlation distance and bounds for mutual information
• scientific article; zbMATH DE number 5644803
• Mutual information, variation, and Fano's inequality
• On inequalities between mutual information and variation
• Distribution of mutual information
• On mutual information estimation for mixed-pair random variables
• Information Inequalities for Joint Distributions, With Interpretations and Applications
• On some extremal problems for mutual information and entropy

Cited in

(2)

• On Suspicious Coincidences and Pointwise Mutual Information
• A Finite-Sample, Distribution-Free, Probabilistic Lower Bound on Mutual Information

This page was built for publication: Mutual Information Bounds via Adjacency Events

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2976484)