Estimating functions of distributions defined over spaces of unknown size

DOI10.3390/E15114668MaRDI QIDQ280679zbMATH OpenOpenAlexWikidataFDO

Authors David Wolpert, Simon DeDeo

Publication date 10 May 2016

Published in Entropy (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1311.4548

zbMATH Keywords

entropyBayesian analysisDirichlet priormutual informationhidden variablesvariable number of bins

Mathematics Subject Classification ID

Bayesian inference (62F15)

Abstract: We consider Bayesian estimation of information-theoretic quantities from data, using a Dirichlet prior. Acknowledging the uncertainty of the event space size $m$ and the Dirichlet prior's concentration parameter $c$, we treat both as random variables set by a hyperprior. We show that the associated hyperprior, $P(c,m)$, obeys a simple "Irrelevance of Unseen Variables" (IUV) desideratum iff $P(c,m)=P(c)P(m)$. Thus, requiring IUV greatly reduces the number of degrees of freedom of the hyperprior. Some information-theoretic quantities can be expressed multiple ways, in terms of different event spaces, e.g., mutual information. With all hyperpriors (implicitly) used in earlier work, different choices of this event space lead to different posterior expected values of these information-theoretic quantities. We show that there is no such dependence on the choice of event space for a hyperprior that obeys IUV. We also derive a result that allows us to exploit IUV to greatly simplify calculations, like the posterior expected mutual information or posterior expected multi-information. We also use computer experiments to favorably compare an IUV-based estimator of entropy to three alternative methods in common use. We end by discussing how seemingly innocuous changes to the formalization of an estimation problem can substantially affect the resultant estimates of posterior expectations.

Recommendations

• Bayesian entropy estimation for countable discrete distributions
• Bayesian estimation of shannon entropy
• Bayes Estimate and Inference for Entropy and Information Index of Fit
• Bayesian and quasi-Bayesian estimators for mutual information from discrete data
• Estimation of Entropy and Mutual Information

Cites work

• scientific article; zbMATH DE number 1098878 (Why is no real title available?)
• scientific article; zbMATH DE number 2061729 (Why is no real title available?)
• scientific article; zbMATH DE number 940826 (Why is no real title available?)
• Bayesian and quasi-Bayesian estimators for mutual information from discrete data
• Coincidences and estimation of entropies of random variables with large cardinalities
• Estimation of Entropy and Mutual Information

Cited in

(1)

• Evaluating accuracy of community detection using the relative normalized mutual information

This page was built for publication: Estimating functions of distributions defined over spaces of unknown size

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q280679)