General type-token distribution

DOI10.1093/BIOMET/ASU035MaRDI QIDQ2934778zbMATH OpenOpenAlexFDO

Authors

Publication date 22 December 2014

Published in Biometrika (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1305.0328

zbMATH Keywords

Poisson-binomial distributionspecies samplingtype-token ratio

Mathematics Subject Classification ID

Applications of Markov chains and discrete-time Markov processes on general state spaces (social mobility, learning theory, industrial processes, etc.) (60J20) Nonparametric estimation (62G05) Asymptotic properties of nonparametric inference (62G20)

Abstract: We consider the problem of estimating the number of types in a corpus using the number of types observed in a sample of tokens from that corpus. We derive exact and asymptotic distributions for the number of observed types, conditioned upon the number of tokens and the latent type distribution. We use the asymptotic distributions to derive an estimator of the latent number of types and we validate this estimator numerically.

Recommendations

• Parameter estimation for a word frequency distribution based on occupancy theory
• scientific article; zbMATH DE number 4024592
• scientific article; zbMATH DE number 4213276
• scientific article; zbMATH DE number 775845
• Estimating word frequencies from lexical dispersion data1

Cited in

(5)

• scientific article; zbMATH DE number 4213276 (Why is no real title available?)
• Parameter estimation for a word frequency distribution based on occupancy theory
• scientific article; zbMATH DE number 775845 (Why is no real title available?)
• Modelling Population Size Using Horvitz-Thompson Approach Based on the Zero-Truncated Poisson Lindley Distribution
• Statistical inference for natural language processing algorithms with a demonstration using type 2 diabetes prediction from electronic health record notes

This page was built for publication: General type-token distribution

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2934778)