Partitions, hypergeometric systems, and Dirichlet processes in statistics (Q1662968)

This short monograph presents key ideas in the study of random partitions, hypergeometric systems and Dirichlet processes, and in particular the links between them, in the context of statistical inference, taking advantage of the underlying exchangeability present in these models. The first chapter gives a short (and very readable) introduction to the key ideas that will be needed through the rest of the book: partitions and exchangeability. Two simple motivating examples are then given, to help the reader understand the goal of the work before starting the more technical material presented in Chapters 2--5. The first example is a two-cluster Bayesian mixture model, and the second a problem of testing goodness-of-fit in a Poisson regression model. Chapter 2 discusses probability measures on combinatorial structures. The author begins by introducing measures on partitions, and in particular the multiplicative measures and combinatorial structures associated with binomial, negative binomial and Poisson coefficients: powersets, multisets and assemblies. Some examples are given (for example, integer partitions with or without the constraint that the elements in the partition be distinct) with clear references to lead the reader to other sources of examples. Exponential structures are then discussed, and examples are again provided (such as set partitions, where the numbers of possible instances are given by the Bell numbers). There is some brief discussion of statistical applications, including to the first of the motivating examples of Chapter 1. Tilting of probability measures and partial Bell polynomials are introduced (again, accompanied by examples), before the chapter concludes with asymptotic results on the size of extreme (both large and small) parts of Gibbs partitions, in particular for the Pitman partition and Ewens sampling formula. The main object of Chapter 3 is to introduce the \(A\)-hypergeometric system and distribution for a nonnegative, integer-valued matrix \(A\). The hypergeometric system is a system of linear partial differential equations; the author makes clear the relationship of this system to the integer partitions and partial Bell polynomials (as introduced in the previous chapter) in the case where \(A\) has two rows. The \(A\)-hypergeometric distribution is a discrete probability measure appearing, for example, in multinomial sampling from log-affine models, whose normalising constants are the \(A\)-hypergeometric polynomials. An explicit algorithm (using holonomic gradient methods) is given for the evaluation of these normalising constants, as they will be needed in algorithms for sampling from these distributions which are discussed in Chapter 5. Statistical properties (for example, nonexistence of the MLE in the case of a two-row matrix \(A\)) are also discussed, as well as links to the second motivating example from Chapter 1. The first part of Chapter 4 introduces concepts from probability theory (such as conditional probability and martingale limit theorems) with which the reader is not assumed to be familiar, and also gives a proof of de Finetti's representation theorem. The main part of the chapter is then spent introducing and exploring properties of the Dirichlet process. This includes two constructions of the Dirichlet process (one based on a the gamma process, and the other on stick-breaking), its use as a conjugate prior in Bayesian statistics, sampling from the Dirichlet process, and its relationship with exchangeability and the Ewens sampling formula. The chapter concludes with some discussion of other prior processes in Bayesian statistics related to the problems discussed in Chapter 2. Finally, Chapter 5 explores aspects of sampling and statistical inference. Both direct and MCMC samplers are given (and illustrated with examples) for \(A\)-hypergeometric distributions and for random partitions. In the case of random partitions, direct sampling is achieved either by making use of the direct sampling from the \(A\)-hypergeometric distribution, or by simulating random graphs generated by processes on partitions. The final section considers maximum likelihood estimation of curved exponential families (using tools from information geometry), which are closely linked with \(A\)-hypergeometric distributions. The book concludes with two appendices, giving background material on symmetric functions and on stochastic processes on partitions. Overall, this monograph should prove to be a useful asset for researchers in various aspects of statistical inference and combinatorial probability. The author assumes some familiarity with discrete probability and with some concepts from algebra (ideals and Gröbner bases, for example), but more advanced topics are introduced from scratch, and references are given throughout to further resources that the reader may consult, either for background or more advanced material.