The exact asymptotic form of Bayesian generalization error in latent Dirichlet allocation

DOI10.1016/J.NEUNET.2021.01.024MaRDI QIDQ6078699zbMATH OpenOpenAlexFDO

Authors Naoki Hayashi

Publication date 28 September 2023

Published in Neural Networks (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2008.01304

zbMATH Keywords

Bayesian inferencegeneralization errorlatent Dirichlet allocation (LDA)learning coefficientreal log canonical threshold (RLCT)singular learning theory

Mathematics Subject Classification ID

Bayesian inference (62F15) Learning and adaptive systems in artificial intelligence (68T05)

Abstract: Latent Dirichlet allocation (LDA) obtains essential information from data by using Bayesian inference. It is applied to knowledge discovery via dimension reducing and clustering in many fields. However, its generalization error had not been yet clarified since it is a singular statistical model where there is no one-to-one mapping from parameters to probability distributions. In this paper, we give the exact asymptotic form of its generalization error and marginal likelihood, by theoretical analysis of its learning coefficient using algebraic geometry. The theoretical result shows that the Bayesian generalization error in LDA is expressed in terms of that in matrix factorization and a penalty from the simplex restriction of LDA's parameter region. A numerical experiment is consistent to the theoretical result.

Recommendations

• Principled selection of hyperparameters in the latent Dirichlet allocation model
• 10.1162/jmlr.2003.3.4-5.993
• Asymptotic analysis of Bayesian generalization error with Newton diagram
• Asymptotic accuracy of Bayes estimation for latent variables with redundancy
• Convergence rates of latent topic models under relaxed identifiability conditions

Cites work

• 10.1162/jmlr.2003.3.4-5.993
• A Bayesian information criterion for singular models. With discussion and authors' reply
• A new look at the statistical model identification
• A widely applicable Bayesian information criterion
• Algebraic Geometry and Statistical Learning Theory
• An asymptotic behaviour of the marginal likelihood for general Markov models
• Asymptotic behavior of exchange ratio in exchange Monte Carlo method
• Asymptotic equivalence of Bayes cross validation and widely applicable information criterion in singular learning theory
• Estimating the dimension of a model
• scientific article; zbMATH DE number 5957252 (Why is no real title available?)
• Julia: a fresh approach to numerical computing
• Learning coefficient in Bayesian estimation of restricted Boltzmann machine
• Marginal likelihood and model selection for Gaussian latent tree and forest models
• Mathematical theory of Bayesian statistics
• On zeta functions associated with prehomogeneous vector spaces
• Resolution of Singularities and Division of Distributions
• Resolution of singularities of an algebraic variety over a field of characteristic zero. II
• Singularities in mixture models and upper bounds of stochastic complexity.
• Stochastic complexities of reduced rank regression in Bayesian estimation
• Stochastic complexity and generalization error of a restricted Boltzmann machine in Bayesian estimation
• The analytic continuation of generalized functions with respect to a parameter
• Variational approximation error in non-negative matrix factorization

Cited in

(2)

• Recent advances in algebraic geometry and Bayesian statistics
• Information criteria and cross validation for Bayesian inference in regular and singular cases

This page was built for publication: The exact asymptotic form of Bayesian generalization error in latent Dirichlet allocation

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6078699)