Sharp Concentration Inequalities for the Centered Relative Entropy

DOI10.1093/IMAIAI/IAAC009MaRDI QIDQ6377984arXivFDO

Authors Alankrita Bhatt, Ankit Pensia

Publication date 18 September 2021

Published in arXiv

Full work available at URL https://doi.org/10.1093/IMAIAI/IAAC009

Mathematics Subject Classification ID

Inequalities; stochastic orderings (60E15) Statistical aspects of information-theoretic topics (62B10) Measures of information, entropy (94A17)

arXiv classification

cs.ITmath.ITmath.STstat.TH

Abstract: We study the relative entropy between the empirical estimate of a discrete distribution and the true underlying distribution. If the minimum value of the probability mass function exceeds an $alpha>0$ (i.e. when the true underlying distribution is bounded sufficiently away from the boundary of the simplex), we prove an upper bound on the moment generating function of the centered relative entropy that matches (up to logarithmic factors in the alphabet size and $alpha$) the optimal asymptotic rates, subsequently leading to a sharp concentration inequality for the centered relative entropy. As a corollary of this result we also obtain confidence intervals and moment bounds for the centered relative entropy that are sharp up to logarithmic factors in the alphabet size and $alpha$.

Recommendations

• An optimal uniform concentration inequality for discrete entropies on finite alphabets in the high-dimensional setting
• Concentration inequalities for the empirical distribution of discrete distributions: beyond the method of types
• Optimal non-asymptotic concentration of centered empirical relative entropy in the high-dimensional regime
• Concentration functions and entropy bounds for discrete log-concave distributions
• Entropy and concentration

Cited in

(1)

• Centered Sobolev inequality and exponential convergence in \(\Phi\)-entropy

This page was built for publication: Sharp Concentration Inequalities for the Centered Relative Entropy

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6377984)