Optimal rates of entropy estimation over Lipschitz balls

DOI10.1214/19-AOS1927MaRDI QIDQ1996767zbMATH OpenFDO

Publication date 26 February 2021

Published in The Annals of Statistics (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1711.02141, https://projecteuclid.org/euclid.aos/1607677234

minimax rate nonparametrics nonsmooth functional estimation polynomial approximation

Statistical aspects of information-theoretic topics (62B10) Minimax procedures in statistical decision theory (62C20) Density estimation (62G07) Measures of information, entropy (94A17)

Abstract: We consider the problem of minimax estimation of the entropy of a density over Lipschitz balls. Dropping the usual assumption that the density is bounded away from zero, we obtain the minimax rates

(n l n n)^{- s / (s + d)} + n^{- 1 / 2}

for

0 < s l e q 2

for densities supported on

[0, 1]^{d}

, where

s

is the smoothness parameter and

n

is the number of independent samples. We generalize the results to densities with unbounded support: given an Orlicz functions

P s i

of rapid growth (such as the sub-exponential and sub-Gaussian classes), the minimax rates for densities with bounded

P s i

-Orlicz norm increase to

(n l n n)^{- s / (s + d)} (P s i^{- 1} (n))^{d (1 - d / p (s + d))} + n^{- 1 / 2}

, where

p

is the norm parameter in the Lipschitz ball. We also show that the integral-form plug-in estimators with kernel density estimates fail to achieve the minimax rates, and characterize their worst case performances over the Lipschitz ball. One of the key steps in analyzing the bias relies on a novel application of the Hardy-Littlewood maximal inequality, which also leads to a new inequality on the Fisher information that may be of independent interest.

Review text

The authors considers the i.i.d. sampling model, \(X_1,\dots,X_n\sim f\), where \(f\) is a probability density function on \(\mathbb{R}^d\) and the goal of this paper is to estimate the entropy of the density function \(f\), \[ H(f)\triangle q\int_{\mathbb{R}^d}-f(x)\ln f(x)\,dx. \] The second section of the article is dedicated to the construction of the estimator, and in the third section, one provides the proof of a result on upper bounds. To construct an optimal estimator for the entropy \(H(f)\) one uses the following strategy: one defines a smooth density \(f_h(x)\) by means of a kernel function \(K_h(.)\), approximates \(f\) by \(f_h\), constructs an estimator for \(H(f_h)\) based on the kernel density estimator and looks for bounds to the approximation error. Generalizations and open problems are discussed in the fourth section. One reports that proofs and auxiliary used results can be found in a supplement to this paper.

Recommendations

Cites work

Cited in

(16)

This page was built for publication: Optimal rates of entropy estimation over Lipschitz balls

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q1996767)