Rediscovery of Good-Turing estimators via Bayesian nonparametrics

DOI10.1111/BIOM.12366zbMATH Open1393.62062arXiv1401.0303OpenAlexW1836729566WikidataQ35725181 ScholiaQ35725181MaRDI QIDQ2805189FDOQ2805189

Authors: Bernardo Nipoti, Stefano Favaro, Yee Whye Teh 

Publication date: 10 May 2016

Published in: Biometrics (Search for Journal in Brave)

Abstract: The problem of estimating discovery probabilities originated in the context of statistical ecology, and in recent years it has become popular due to its frequent appearance in challenging applications arising in genetics, bioinformatics, linguistics, designs of experiments, machine learning, etc. A full range of statistical approaches, parametric and nonparametric as well as frequentist and Bayesian, has been proposed for estimating discovery probabilities. In this paper we investigate the relationships between the celebrated Good-Turing approach, which is a frequentist nonparametric approach developed in the 1940s, and a Bayesian nonparametric approach recently introduced in the literature. Specifically, under the assumption of a two parameter Poisson-Dirichlet prior, we show that Bayesian nonparametric estimators of discovery probabilities are asymptotically equivalent, for a large sample size, to suitably smoothed Good-Turing estimators. As a by-product of this result, we introduce and investigate a methodology for deriving exact and asymptotic credible intervals to be associated with the Bayesian nonparametric estimators of discovery probabilities. The proposed methodology is illustrated through a comprehensive simulation study and the analysis of Expressed Sequence Tags data generated by sequencing a benchmark complementary DNA library.

Full work available at URL: https://arxiv.org/abs/1401.0303

Recommendations

• Bayesian nonparametric inference for discovery probabilities: credible intervals and large sample asymptotics
• Bayesian Nonparametric Estimation of the Probability of Discovering New Species
• A new estimator of the discovery probability
• Predicting the Conditional Probability of Discovering a New Class
• Nonparametric bayes estimation of the probability of discovering a new species

zbMATH Keywords

Bayesian nonparametricsasymptotic equivalencesmoothing techniquecredible intervalsdiscovery probabilitytwo parameter Poisson-Dirichlet priorexpressed sequence tagsGood-Turing estimatorGood-Toulmin estimator

Mathematics Subject Classification ID

Nonparametric estimation (62G05) Applications of statistics to biology and medical sciences; meta analysis (62P10) Nonparametric tolerance and confidence regions (62G15) Protein sequences, DNA sequences (92D20)

Cites Work

• Bayesian Nonparametric Estimation of the Probability of Discovering New Species
• Efficiently sampling nested Archimedean copulas
• A triptych of discrete distributions related to the stable law
• THE POPULATION FREQUENCIES OF SPECIES AND THE ESTIMATION OF POPULATION PARAMETERS
• Exchangeable and partially exchangeable random partitions
• THE NUMBER OF NEW SPECIES, AND THE INCREASE IN POPULATION COVERAGE, WHEN A SAMPLE IS INCREASED
• Notes on the occupancy problem with infinitely many boxes: general asymptotics and power laws
• Conditional formulae for Gibbs-type exchangeable random partitions
• Predicting the Conditional Probability of Discovering a New Class
• A new estimator of the discovery probability

Cited In (8)

• Asymptotic properties of Turing's formula in relative error
• A Good-Turing estimator for feature allocation models
• Bayesian nonparametric inference for discovery probabilities: credible intervals and large sample asymptotics
• A new estimator of the discovery probability
• Bayesian Nonparametric Estimation of the Probability of Discovering New Species
• Generalized Good-Turing Improves Missing Mass Estimation
• Good-Turing frequency estimation in a finite population
• Perfect sampling of the posterior in the hierarchical Pitman-Yor process

This page was built for publication: Rediscovery of Good-Turing estimators via Bayesian nonparametrics

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2805189)