A Bayesian characterization of relative entropy

zbMATH Open1321.94023arXiv1402.3067MaRDI QIDQ2877683FDOQ2877683

Authors: John Baez, Tobias Fritz 

Publication date: 25 August 2014

Published in: Theory and Applications of Categories (Search for Journal in Brave)

Abstract: We give a new characterization of relative entropy, also known as the Kullback-Leibler divergence. We use a number of interesting categories related to probability theory. In particular, we consider a category FinStat where an object is a finite set equipped with a probability distribution, while a morphism is a measure-preserving function $f:XoY$ together with a stochastic right inverse $s:YoX$. The function $f$ can be thought of as a measurement process, while s provides a hypothesis about the state of the measured system given the result of a measurement. Given this data we can define the entropy of the probability distribution on $X$ relative to the "prior" given by pushing the probability distribution on $Y$ forwards along $s$. We say that $s$ is "optimal" if these distributions agree. We show that any convex linear, lower semicontinuous functor from FinStat to the additive monoid $[0,infty]$ which vanishes when $s$ is optimal must be a scalar multiple of this relative entropy. Our proof is independent of all earlier characterizations, but inspired by the work of Petz.

Full work available at URL: https://arxiv.org/abs/1402.3067

File on IPFS (Hint: this is only the Hash - if you get a timeout, this file is not available on our server.)

Recommendations

• Towards a functorial description of quantum relative entropy
• A functorial characterization of von Neumann entropy
• A short characterization of relative entropy
• A characterization of entropy in terms of information loss
• Further results on the relative entropy

zbMATH Keywords

Kullback-Leibler divergencerelative entropymeasures of information

Mathematics Subject Classification ID

Bayesian inference (62F15) Measures of information, entropy (94A17) Special categories (18B99)

Cited In (14)

• Objective Bayesian estimation for the differential entropy measure under generalized half-normal distribution
• Bayesian revision of a prior given prior-data conflict, expert opinion, or a similar insight: a large-deviation approach
• A characterization of entropy in terms of information loss
• On information links
• A categorical characterization of relative entropy on standard Borel spaces
• A non-commutative Bayes' theorem
• A probability monad as the colimit of spaces of finite samples
• Beyond Bell's theorem. II: Scenarios with arbitrary causal structure
• A mathematical theory of resources
• A short characterization of relative entropy
• A categorical characterization of relative entropy on standard Borel spaces
• Title not available (Why is that?)
• Entropy and diversity. The axiomatic approach
• Towards a functorial description of quantum relative entropy

This page was built for publication: A Bayesian characterization of relative entropy

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2877683)