Optimal rates of entropy estimation over Lipschitz balls

DOI10.1214/19-AOS1927MaRDI QIDQ1996767zbMATH OpenFDO

Authors Yanyan Li

Publication date 26 February 2021

Published in The Annals of Statistics (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1711.02141, https://projecteuclid.org/euclid.aos/1607677234

zbMATH Keywords

minimax ratenonparametricsnonsmooth functional estimationpolynomial approximation

Mathematics Subject Classification ID

Statistical aspects of information-theoretic topics (62B10) Minimax procedures in statistical decision theory (62C20) Density estimation (62G07) Measures of information, entropy (94A17)

Abstract: We consider the problem of minimax estimation of the entropy of a density over Lipschitz balls. Dropping the usual assumption that the density is bounded away from zero, we obtain the minimax rates $(nlnn{)}^{-s/(s+d)}+n^{-1/2}$ for $0<sleq2$ for densities supported on $[0,1{]}^d$, where $s$ is the smoothness parameter and $n$ is the number of independent samples. We generalize the results to densities with unbounded support: given an Orlicz functions $Psi$ of rapid growth (such as the sub-exponential and sub-Gaussian classes), the minimax rates for densities with bounded $Psi$-Orlicz norm increase to $(nlnn{)}^{-s/(s+d)}(Ps{i}^{-1}(n){)}^{d(1-d/p(s+d))}+n^{-1/2}$, where $p$ is the norm parameter in the Lipschitz ball. We also show that the integral-form plug-in estimators with kernel density estimates fail to achieve the minimax rates, and characterize their worst case performances over the Lipschitz ball. One of the key steps in analyzing the bias relies on a novel application of the Hardy-Littlewood maximal inequality, which also leads to a new inequality on the Fisher information that may be of independent interest.

Recommendations

• Information-theoretic determination of minimax rates of convergence
• Best asymptotic normality of the kernel density entropy estimator for smooth densities
• Undersmoothed Kernel Entropy Estimators
• From -entropy to KL-entropy: analysis of minimum information complexity density estima\-tion
• ε-Entropy and moduli of smoothness in $L^{p}$-spaces

Cites work

• A note on nonparametric estimation of linear functionals.
• Adaptation to lowest density regions with application to support recovery
• An Efron-Stein inequality for nonsymmetric statistics
• Asymptotic equivalence of density estimation and Gaussian white noise
• Asymptotic nonequivalence of density estimation and Gaussian white noise for small densities
• Asymptotic Statistics
• Breaking the Bandwidth Barrier: Geometrical Adaptive Entropy Estimation
• Chebyshev polynomials, moment matching, and optimal estimation of the unseen
• Demystifying Fixed <inline-formula> <tex-math notation="LaTeX">$k$ </tex-math> </inline-formula>-Nearest Neighbor Information Estimators
• Efficient estimation of integral functionals of a density
• Efficient multivariate entropy estimation via \(k\)-nearest neighbour distances
• Estimating Entropy on&lt;tex&gt;$m$&lt;/tex&gt;Bins Given Fewer Than&lt;tex&gt;$m$&lt;/tex&gt;Samples
• Estimating nonquadratic functionals of a density using Haar wavelets
• Estimation of entropy and other functionals of a multivariate density
• Estimation of integral functionals of a density
• Estimation of integrated squared density derivatives
• Estimation of KL Divergence: Optimal Minimax Rate
• Estimation of Nonlinear Functionals of Densities With Confidence
• Geodesic Entropic Graphs for Dimension and Entropy Estimation in Manifold Learning
• Higher order influence functions and minimax estimation of nonlinear functionals
• scientific article; zbMATH DE number 4107941 (Why is no real title available?)
• scientific article; zbMATH DE number 3656968 (Why is no real title available?)
• scientific article; zbMATH DE number 17209 (Why is no real title available?)
• scientific article; zbMATH DE number 65177 (Why is no real title available?)
• scientific article; zbMATH DE number 490141 (Why is no real title available?)
• scientific article; zbMATH DE number 1064649 (Why is no real title available?)
• scientific article; zbMATH DE number 1092005 (Why is no real title available?)
• scientific article; zbMATH DE number 1522808 (Why is no real title available?)
• scientific article; zbMATH DE number 922427 (Why is no real title available?)
• Introduction to nonparametric estimation
• Limit theorems for sums of general functions of m-spacings
• Minimax estimation of a functional on a structured high-dimensional model
• Minimax Estimation of Functionals of Discrete Distributions
• Minimax Estimation of the <inline-formula> <tex-math notation="LaTeX">$L_{1}$ </tex-math> </inline-formula> Distance
• Minimax estimation of the integral of a power of a density
• Minimax quadratic estimation of a quadratic functional
• Minimax Rates of Entropy Estimation on Large Alphabets via Best Polynomial Approximation
• Minimax theory of image reconstruction
• Nonquadratic estimators of a quadratic functional
• On entropy estimation by \(m\)-spacing method
• On estimation of L_r-norms in Gaussian white noise models
• On estimation of the \(L_r\) norm of a regression function
• On the dimension and entropy of probability distributions
• On the estimation of entropy
• On the estimation of quadratic functionals
• On the Kozachenko-Leonenko entropy estimator
• Optimal plug-in estimators for nonparametric functional estimation
• Some observations on Besov and Lizorkin-Triebel spaces.
• Some Problems on Nonparametric Estimation in Gaussian White Noise
• Testing composite hypotheses, Hermite polynomials and optimal estimation of a nonsmooth functional
• The Power of Linear Estimators
• Undersmoothed Kernel Entropy Estimators
• Universal estimation of information measures for analog sources
• Wavelets, approximation, and statistical applications

Cited in

(16)

• Minimax estimation of norms of a probability density. I: Lower bounds
• On estimation of nonsmooth functionals of sparse normal means
• On the estimation of entropy for non-negative data
• Optimal bandwidth selection for re-substitution entropy estimation
• Estimation of a local Lipschitz constant of the B_q-entropy
• Best asymptotic normality of the kernel density entropy estimator for smooth densities
• Adaptive estimation of nonparametric functionals
• Entropy estimation via uniformization
• Efficient functional estimation and the super-oracle phenomenon
• Plugin estimation of smooth optimal transport maps
• Estimation of density functionals via cross-validation
• An integrated test on the linear structure of high-dimensional covariance matrices
• Transport dependency: optimal transport based dependency measures
• Distribution-free tests for lossless feature selection in classification and regression
• Correcting convexity bias in function and functional estimate
• Sharp convergence rates of empirical unbalanced optimal transport for spatio-temporal point processes

This page was built for publication: Optimal rates of entropy estimation over Lipschitz balls

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q1996767)