Comparison of contraction coefficients for f-divergences

DOI10.1134/S0032946020020015zbMATH Open1457.94063arXiv1510.01844OpenAlexW3042492913MaRDI QIDQ784380FDOQ784380

Authors: D. Kharzeev 

Publication date: 3 August 2020

Published in: Problems of Information Transmission (Search for Journal in Brave)

Abstract: Data processing inequalities for $f$-divergences can be sharpened using constants called "contraction coefficients" to produce strong data processing inequalities. For any discrete source-channel pair, the contraction coefficients for $f$-divergences are lower bounded by the contraction coefficient for $ch{i}^2$-divergence. In this paper, we elucidate that this lower bound can be achieved by driving the input $f$-divergences of the contraction coefficients to zero. Then, we establish a linear upper bound on the contraction coefficients for a certain class of $f$-divergences using the contraction coefficient for $ch{i}^2$-divergence, and refine this upper bound for the salient special case of Kullback-Leibler (KL) divergence. Furthermore, we present an alternative proof of the fact that the contraction coefficients for KL and $ch{i}^2$-divergences are equal for a Gaussian source with an additive Gaussian noise channel (where the former coefficient can be power constrained). Finally, we generalize the well-known result that contraction coefficients of channels (after extremizing over all possible sources) for all $f$-divergences with non-linear operator convex $f$ are equal. In particular, we prove that the so called "less noisy" preorder over channels can be equivalently characterized by any non-linear operator convex $f$-divergence.

Full work available at URL: https://arxiv.org/abs/1510.01844

Recommendations

• scientific article; zbMATH DE number 222635
• scientific article; zbMATH DE number 4197134
• Bounds for \(f\)-divergences under likelihood ratio constraints.
• On an Extension of the Notion off-Divergence
• $f$ -Divergence Inequalities
• Bregman divergences from comparative convexity
• Convergence of contrastive divergence algorithm in exponential family

zbMATH Keywords

contraction coefficientmaximal correlation\(f\)-divergence/relative \(f\)-entropyless noisy preorderstrong data processing inequality

Mathematics Subject Classification ID

Statistical aspects of information-theoretic topics (62B10) Measures of information, entropy (94A17)

Cites Work

• Markov chains and mixing times. With a chapter on ``Coupling from the past by James G. Propp and David B. Wilson.
• Title not available (Why is that?)
• Title not available (Why is that?)
• Elements of Information Theory
• Diffusion maps
• Title not available (Why is that?)
• Title not available (Why is that?)
• On Information and Sufficiency
• On measures of dependence
• Estimating Optimal Transformations for Multiple Regression and Correlation
• Title not available (Why is that?)
• On Choosing and Bounding Probability Metrics
• RELATIONS BETWEEN TWO SETS OF VARIATES
• Asymptotic methods in statistical decision theory
• Clustering with Bregman divergences.
• Title not available (Why is that?)
• Title not available (Why is that?)
• Context tree estimation for not necessarily finite memory processes, via BIC and MDL
• Relative entropy under mappings by stochastic matrices
• Ergodicity coefficients defined by vector norms
• On Divergences and Informations in Statistics and Information Theory
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Information Theory and Statistics: A Tutorial
• Non-negative matrices and Markov chains. 2nd ed
• A class of measures of informativity of observation channels
• Title not available (Why is that?)
• On functionals satisfying a data-processing theorem
• Trace inequalities and quantum entropy: an introductory course
• The Structure of Bivariate Distributions
• Information theoretic inequalities
• Refinements of Pinsker's inequality
• Title not available (Why is that?)
• Title not available (Why is that?)
• Equivalence of certain entropy contraction coefficients
• Markov Processes and the H-Theorem
• Information theory. Coding theorems for discrete memoryless systems
• Signal propagation and noisy circuits
• On Pinsker's and Vajda's Type Inequalities for Csiszár's $f$-Divergences
• Obtaining measure concentration from Markov contraction
• Maximal coupling
• Estimation Efficiency Under Privacy Constraints
• Title not available (Why is that?)
• Title not available (Why is that?)
• A variational characterization of canonical angles between subspaces
• Spreading of sets in product spaces and hypercontraction of the Markov operator
• On Sequences of Pairs of Dependent Random Variables
• Title not available (Why is that?)
• Coefficients of ergodicity: structure and applications
• Title not available (Why is that?)
• A Distribution Dependent Refinement of Pinsker's Inequality
• Network information theory
• Strong data-processing inequalities for channels and Bayesian networks
• A class of modified Pearson and Neyman statistics
• $f$ -Divergence Inequalities
• On the Entropy of a Noisy Function
• Strong Data Processing Inequalities and <inline-formula> <tex-math notation="LaTeX">$\Phi $ </tex-math> </inline-formula>-Sobolev Inequalities for Discrete Channels
• Dissipation of Information in Channels With Input Constraints
• The Geometric Interpretation of Correspondence Analysis
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• On a special class of broadcast channels with confidential messages
• The efficiency of investment information
• Title not available (Why is that?)
• Polynomial Singular Value Decompositions of a Family of Source-Channel Models
• Strong Data Processing Inequalities for Input Constrained Additive Noise Channels
• Comparison of Channels: Criteria for Domination by a Symmetric Channel
• Evaluation of Marton's Inner Bound for the General Broadcast Channel
• A Coordinate System for Gaussian Networks
• On Pairs of $f$-Divergences and Their Joint Range
• A New Data Processing Inequality and Its Applications in Distributed Source and Channel Coding
• Principal Inertia Components and Applications

Cited In (3)

• On the maximum \(f\)-divergence of probability distributions given the value of their coupling
• The \(f\)-divergence and coupling of probability distributions
• Universal Features for High-Dimensional Learning and Inference

This page was built for publication: Comparison of contraction coefficients for \(f\)-divergences

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q784380)