Maximum Likelihood Estimation of Functionals of Discrete Distributions

DOI10.1109/TIT.2017.2733537MaRDI QIDQ4566527zbMATH OpenOpenAlexFDO

Authors Jiantao Jiao, Kartik Venkat, Yanjun Han, Tsachy Weissman

Publication date 27 June 2018

Published in IEEE Transactions on Information Theory (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1406.6959

Mathematics Subject Classification ID

Statistical aspects of information-theoretic topics (62B10) Bayesian inference (62F15) Nonparametric estimation (62G05) Measures of information, entropy (94A17)

Abstract: We consider the problem of estimating functionals of discrete distributions, and focus on tight nonasymptotic analysis of the worst case squared error risk of widely used estimators. We apply concentration inequalities to analyze the random fluctuation of these estimators around their expectations, and the theory of approximation using positive linear operators to analyze the deviation of their expectations from the true functional, namely their emph{bias}. We characterize the worst case squared error risk incurred by the Maximum Likelihood Estimator (MLE) in estimating the Shannon entropy $H(P)=su{m}_{i=1}^S-p_{i}ln{p}_i$, and $F_{a}lpha(P)=su{m}_{i=1}^S{p}_i^{a}lpha,alpha>0$, up to multiplicative constants, for any alphabet size $Sleqinfty$ and sample size $n$ for which the risk may vanish. As a corollary, for Shannon entropy estimation, we show that it is necessary and sufficient to have $nggS$ observations for the MLE to be consistent. In addition, we establish that it is necessary and sufficient to consider $ngg{S}^{1/alpha}$ samples for the MLE to consistently estimate $F_{a}lpha(P),0<alpha<1$. The minimax rate-optimal estimators for both problems require $S/lnS$ and $S^{1/alpha}/lnS$ samples, which implies that the MLE has a strictly sub-optimal sample complexity. When $1<alpha<3/2$, we show that the worst-case squared error rate of convergence for the MLE is $n^{-2(alpha-1)}$ for infinite alphabet size, while the minimax squared error rate is $(nlnn{)}^{-2(alpha-1)}$. When $alphageq3/2$, the MLE achieves the minimax optimal rate $n^{-1}$ regardless of the alphabet size. As an application of the general theory, we analyze the Dirichlet prior smoothing techniques for Shannon entropy estimation. We show that no matter how we tune the parameters in the Dirichlet prior, this technique cannot achieve the minimax rates in entropy estimation.

Recommendations

• Minimax Estimation of Functionals of Discrete Distributions
• scientific article; zbMATH DE number 238311
• Maximum likelihood estimation of dirichlet distributions
• Moments of maximum likelihood estimators in the discrete case
• scientific article; zbMATH DE number 774832
• Maximum likelihood characterization of distributions
• Maximum likelihood characterization of distributions

Cited in

(10)

• Mean-square estimation of nonlinear functionals via Kalman filtering
• Computation of distribution functions from likelihood information near observed data
• Exact P-values for discrete models obtained by estimation and maximization
• Discrete power distributions and inference using likelihood
• Maximum likelihood gradient identification for multivariate equation‐error moving average systems using the multi‐innovation theory
• Optimal non-asymptotic concentration of centered empirical relative entropy in the high-dimensional regime
• Optimal estimation of the mean function based on discretely sampled functional data: phase transition
• Distribution estimation under the infinity norm
• On estimation of L_r-norms in Gaussian white noise models
• Two-stage algorithm for estimation of nonlinear functions of state vector in linear Gaussian continuous dynamical systems

This page was built for publication: Maximum Likelihood Estimation of Functionals of Discrete Distributions

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q4566527)