Relative entropy derivative bounds

DOI10.3390/E15072861MaRDI QIDQ280464zbMATH OpenOpenAlexFDO

Authors Pablo Zegers, Alexis Fuentes, Carlos Alarcón

Publication date 10 May 2016

Published in Entropy (Search for Journal in Brave)

Full work available at URL https://doi.org/10.3390/e15072861

zbMATH Keywords

asymptotic equipartition principleFisher informationKullback-Leibler divergencemaximum log likelihoodrelative entropyShannon differential entropytypical set

Mathematics Subject Classification ID

Measures of information, entropy (94A17)

Review text

Summary: We show that the derivative of the relative entropy with respect to its parameters is lower and upper bounded. We characterize the conditions under which this derivative can reach zero. We use these results to explain when the minimum relative entropy and the maximum log likelihood approaches can be valid. We show that these approaches naturally activate in the presence of large data sets and that they are inherent properties of any density estimation process involving large numbers of random variables.

Recommendations

• A correspondence principle for relative entropy minimization
• Some upper bounds for relative entropy and applications
• Learning Theory
• On density estimation under relative entropy loss criterion
• Consistency and generalization bounds for maximum entropy density estimation

Cites work

• A Mathematical Theory of Communication
• Analysis of signals in the Fisher-Shannon information plane
• scientific article; zbMATH DE number 3304505 (Why is no real title available?)

Describes a project that uses

Uses Software

• JPEG2000

This page was built for publication: Relative entropy derivative bounds

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q280464)