Bayesian entropy estimation for countable discrete distributions

MaRDI QIDQ2934104zbMATH OpenFDO

Authors Evan Archer, Il Memming Park, Jonathan W. Pillow

Publication date 8 December 2014

Full work available at URL https://arxiv.org/abs/1302.0328, http://jmlr.csail.mit.edu/papers/v15/archer14a.html

zbMATH Keywords

Bayesian estimationBayesian nonparametricsDirichlet processinformation theoryneural codingPitman-Yor process

Mathematics Subject Classification ID

Random measures (60G57) Statistical aspects of information-theoretic topics (62B10) Bayesian inference (62F15) Density estimation (62G07)

Abstract: We consider the problem of estimating Shannon's entropy $H$ from discrete data, in cases where the number of possible symbols is unknown or even countably infinite. The Pitman-Yor process, a generalization of Dirichlet process, provides a tractable prior distribution over the space of countably infinite discrete distributions, and has found major applications in Bayesian non-parametric statistics and machine learning. Here we show that it also provides a natural family of priors for Bayesian entropy estimation, due to the fact that moments of the induced posterior distribution over $H$ can be computed analytically. We derive formulas for the posterior mean (Bayes' least squares estimate) and variance under Dirichlet and Pitman-Yor process priors. Moreover, we show that a fixed Dirichlet or Pitman-Yor process prior implies a narrow prior distribution over $H$, meaning the prior strongly determines the entropy estimate in the under-sampled regime. We derive a family of continuous mixing measures such that the resulting mixture of Pitman-Yor processes produces an approximately flat prior over $H$. We show that the resulting Pitman-Yor Mixture (PYM) entropy estimator is consistent for a large class of distributions. We explore the theoretical properties of the resulting estimator, and show that it performs well both in simulation and in application to real data.

Recommendations

• A Bayesian nonparametric estimation to entropy
• Bayesian estimation of shannon entropy
• Bayes Estimate and Inference for Entropy and Information Index of Fit
• On entropy estimation for distributions with countable support.
• Bayes' estimators of generalized entropies

Cited in

(9)

• On the permutation entropy Bayesian estimation
• A Bayesian nonparametric estimation to entropy
• One step entropy variation in sequential sampling of species for the Poisson-Dirichlet process
• Estimating functions of distributions defined over spaces of unknown size
• Bayes Estimate and Inference for Entropy and Information Index of Fit
• Bayesian estimation of shannon entropy
• Statistical estimation of conditional Shannon entropy
• A note on entropy estimation
• Properties of Shannon and Rényi entropies of the Poisson distribution as the functions of intensity parameter

This page was built for publication: Bayesian entropy estimation for countable discrete distributions

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2934104)