Word frequency distributions
This book is an introduction to the statistical analysis of word frequency distributions, i.e., the quantitative aspects of lexical structure. Word frequency distributions are characterized by very large numbers of rare words. This property leads to strange statistical phenomena such as mean frequencies that systematically keep changing as the number of observations is increased, relative frequencies that even in large samples are not fully reliable estimators of population probabilities, and model parameters that emerge as functions of the text size. The aim of this monograph is to make statistical techniques for the analysis of distributions with large numbers of rare events (LNRE distributions) more accessible for non-specialist. First, basic concepts of lexical statistics (e.g., sample size, randomness of occurrence of words, frequency spectrum, Zipf's rank-frequency model, lognormal distributions) and their notations are introduced. Then, non-parametric methods for the analysis of word frequency distributions are discussed, such as the binomial model and its Poisson approximation, LNRE zone (the range of sample sizes where the sample relative frequencies are not good estimates of the corresponding population probabilities), Good-Turing estimates (which adjust sample relative frequencies for the non-negligible frequency weight of the unseen words). The next chapter describes in detail three parametric models, viz. the lognormal model, the Yule-Simon Zipfian model, and the generalized inverse Gauss-Poisson model. Furthermore, the concept of mixture distributions is introduced and illustrated by examples of mixture analyses of morphological data. Also, the effect of non-randomness in word use on the accuracy of the non-parametric and parametric models is explored, all of which are based on the assumption that words occur independently and randomly in texts. The final chapter presents various examples of applications such as a study of distributional properties of the lexicon and morphological productivity. Throughout the book, concepts of probability theory and statistics necessary to understand the analysis of word frequency distributions are carefully introduced. Appendices are included, which give solutions to exercises at the end of various chapters, and contain a documentation of C programs for carrying out statistical analyses of word frequencies (LEXSTATS running under LINUX).
- On rank distributions of letter frequencies in human languages
- Dumont's statistic on words
- Modelling of count data using nonparametric mixtures
- A language as a self-organized critical system
- Calculation of precise constants in a probability model of Zipf's law generation and asymptotics of sums of multinomial coefficients
- Extended truncated Tweedie-Poisson model
- A computationally efficient approach to estimating species richness and rarefaction curve
- Completely monotone distributions: mixing, approximation and estimation of number of species
- On recovering a mixed Poisson distribution from its left-truncated version
- Relative abundances of mineral species: a statistical measure to characterize Earth-like planets based on Earth's mineralogy
- Identifying trends in word frequency dynamics
- Probability-theoretic and statistical analysis of dictionary texts
- Bayesian estimation of Earth's undiscovered mineralogical diversity using noninformative priors
- On the measure and the estimation of evenness and diversity
- Statistical simulation and the distribution of distances between identical elements in a random sequence
- On zero-truncating and mixing Poisson distributions
- Convergence properties in certain occupancy problems including the Karlin-Rouault law
- Scaling laws and fluctuations in the statistics of word frequencies
- A Model for Function Word Counts
- On the probability distribution of typological frequencies
- Probability for linguists
- Simple and efficient classification scheme based on specific vocabulary
- Parameter estimation for a word frequency distribution based on occupancy theory
- scientific article; zbMATH DE number 4024592 (Why is no real title available?)
- Estimating word frequencies from lexical dispersion data1
- scientific article; zbMATH DE number 845590 (Why is no real title available?)
- Empirical Bayes estimators of structural distribution of words in Lithuanian texts
- Extended truncated Inverse Gaussian–Poisson model
- Two halves of a meaningful text are statistically different
- Zero-inflated beta distribution applied to word frequency and lexical dispersion in corpus linguistics
- The Sichel model and the mixing and truncation order
- Zipf's law unzipped
- A scaling law beyond Zipf's law and its relation to heaps' law
- scientific article; zbMATH DE number 7300696 (Why is no real title available?)
- Zipf's law for randomly generated frequencies: explicit tests for the goodness-of-fit
- DISTRIBUTION OF MULTI-WORDS IN CHINESE AND ENGLISH DOCUMENTS
- Producing power-law distributions and damping word frequencies with two-stage language models
- Martingale limit theorems of divisible statistics in a multinomial scheme with mixed frequencies
- Confidence intervals for ratios of means applied to corpus-based word frequency classes
- Memoryless reigns of the ``Sons of Heaven
- A review on authorship attribution in text mining
- Near-optimal estimation of the unseen under regularly varying tail populations
- Extending the applicability of the Zipf's laws to the sequences of byte data
- Quantum algorithms for compositional text processing
- Contaminated Gibbs-type priors
- The exact rank-frequency function and size-frequency function of N-grams and N-word phrases with applications
- A firm foundation for statistical disclosure control
- Mineral species frequency distribution conforms to a large number of rare events model: prediction of Earth's missing minerals
- Diversity analysis in multiple-choice questionnaires
This page was built for publication: Word frequency distributions
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5943510)
