Approximation of density functions by sequences of exponential families (Q1178945)

The problem of estimating a probability density function \(p(x)\) defined on a bounded interval is considered. The authors approximate \(\log p(x)\) by a basis functions expansion consisting of polynomials, splines or trigonometric series. The expansion yields a regular exponential family within which \(p(x)\) is estimated by the method of maximum likelihood. This method of density estimation arises by application of the principle of maximum entropy or minimum relative entropy subject to empirical constraints. It is shown that if \(\log p(x)\) has \(r\) square-integrable derivatives, then the sequence of the density function estimators \(\hat p_ n\) converges to \(p\) in the sense of relative entropy \(\int p \log(p/\hat p_ n)\) at rate \(O_{pr}(1/m^{2r}+m/n)\) as \(m\to \infty\) and \(m^ 2/n\to 0\) in the spline and trigonometric cases, and \(m^ 3/n\to 0\) in the polynomial case, where \(m\) is the dimension of the family and \(n\) is the sample size. Boundary conditions are assumed for \(p(x)\) in the trigonometric case. This convergence rate specializes to \(O_{pr}(n^{- 2r/(2r+1)})\) by setting \(m=n^{1/(2r+1)}\) when \(\log p(x)\) is known to have degree of smoothness at least \(r\). Analogous convergence results for the relative entropy are shown to hold in general, for any class of log-density functions and sequences of finite-dimensional linear spaces having \(L_ 2\) and \(L_ \infty\) approximation properties.