Entropy and Relative Entropy From Information-Theoretic Principles

DOI10.1109/TIT.2021.3078337MaRDI QIDQ5032495zbMATH OpenOpenAlexFDO

Authors Gilad Gour, Marco Tomamichel

Publication date 17 February 2022

Published in IEEE Transactions on Information Theory (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2006.11164

Mathematics Subject Classification ID

Information theory (general) (94A15) Measures of information, entropy (94A17)

Abstract: We introduce an axiomatic approach to entropies and relative entropies that relies only on minimal information-theoretic axioms, namely monotonicity under mixing and data-processing as well as additivity for product distributions. We find that these axioms induce sufficient structure to establish continuity in the interior of the probability simplex and meaningful upper and lower bounds, e.g., we find that every relative entropy must lie between the R'enyi divergences of order $0$ and $infty$. We further show simple conditions for positive definiteness of such relative entropies and a characterisation in term of a variant of relative trumping. Our main result is a one-to-one correspondence between entropies and relative entropies.

Cited in

(11)

• Information vs. entropy vs. probability
• The permutation entropy rate equals the metric entropy rate for ergodic information sources and ergodic dynamical systems
• An information-theoretic equality implying the Jarzynski relation
• Complex entropy and resultant information measures
• Information Theory, Relative Entropy and Statistics
• A Justification for Applying the Principle of Minimum Relative Entropy to Information Integration Problems
• Information entropy in problems of classical and quantum statistical mechanics
• On the information obtained using Shannon's entropy through spin density
• Majorization and randomness measures
• Quantum conditional entropies from convex trace functionals
• Some properties of the relative entropy density of arbitrary information source

This page was built for publication: Entropy and Relative Entropy From Information-Theoretic Principles

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5032495)