Improved model-based clustering performance using Bayesian initialization averaging

DOI10.1007/S00180-018-0855-2zbMATH Open1417.62177arXiv1504.06870OpenAlexW2962942556MaRDI QIDQ1729337FDOQ1729337

Authors: Adrian O'Hagan, Arthur T. White 

Publication date: 27 February 2019

Published in: Computational Statistics (Search for Journal in Brave)

Abstract: The Expectation-Maximization (EM) algorithm is a commonly used method for finding the maximum likelihood estimates of the parameters in a mixture model via coordinate ascent. A serious pitfall with the algorithm is that in the case of multimodal likelihood functions, it can get trapped at a local maximum. This problem often occurs when sub-optimal starting values are used to initialize the algorithm. Bayesian initialization averaging (BIA) is proposed as an ensemble method to generate high quality starting values for the EM algorithm. Competing sets of trial starting values are combined as a weighted average, which is then used as the starting position for a full EM run. The method can also be extended to variational Bayes (VB) methods, a class of algorithm similar to EM that is based on an approximation of the model posterior. The BIA method is demonstrated on real continuous, categorical and network data sets, and the convergent log-likelihoods and associated clustering solutions presented. These compare favorably with the output produced using competing initialization methods such as random starts, hierarchical clustering and deterministic annealing, with the highest available maximum likelihood estimates obtained in a higher percentage of cases, at reasonable computational cost. The implications of the different clustering solutions obtained by local maxima are also discussed.

Full work available at URL: https://arxiv.org/abs/1504.06870

Recommendations

• A Gaussian mixture model based \(k\)-means to initialize the EM algorithm
• Improved initialisation of model-based clustering using Gaussian hierarchical partitions
• EM for mixtures
• Choosing starting values for the EM algorithm for getting the highest likelihood in multivariate Gaussian mixture models
• An experimental comparison of model-based clustering methods

zbMATH Keywords

expectation-maximization algorithmmodel-based clusteringBayesian model averagingmultimodal likelihoodfinite mixture modelshierarchical clustering

Mathematics Subject Classification ID

Computational methods for problems pertaining to statistics (62-08) Point estimation (62F10) Bayesian inference (62F15) Classification and discrimination; cluster analysis (statistical aspects) (62H30) Applications of statistics to biology and medical sciences; meta analysis (62P10)

Cites Work

• MCLUST: Software for model-based cluster analysis
• Algorithm 778: L-BFGS-B
• Estimating the dimension of a model
• Bayesian model averaging: A tutorial. (with comments and a rejoinder).
• Title not available (Why is that?)
• Finite mixture models
• Computational aspects of fitting mixture models via the expectation-maximization algorithm
• A classification EM algorithm for clustering and two stochastic versions
• Consistent estimation of the order of mixture models.
• Maximum likelihood estimation via the ECM algorithm: A general framework
• Title not available (Why is that?)
• A Limited Memory Algorithm for Bound Constrained Optimization
• Finite mixtures of multivariate skew \(t\)-distributions: some recent and new results
• Choosing starting values for the EM algorithm for getting the highest likelihood in multivariate Gaussian mixture models
• Choosing initial values for the EM algorithm for finite mixtures
• Estimation and prediction for stochastic blockmodels for graphs with latent block structure
• Mixture Densities, Maximum Likelihood and the EM Algorithm
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Model-based clustering, classification, and discriminant analysis via mixtures of multivariate \(t\)-distributions
• Title not available (Why is that?)
• Title not available (Why is that?)
• Exploratory latent structure analysis using both identifiable and unidentifiable models
• EM for mixtures
• Enhancing the selection of a model-based clustering with external categorical variables
• Title not available (Why is that?)
• Variational Bayes approach for model aggregation in unsupervised classification with Markovian dependency
• Data Mining and Knowledge Discovery Handbook
• On the Bumpy Road to the Dominant Mode
• Bayesian Model Averaging in Proportional Hazard Models: Assessing the Risk of a Stroke
• Information-based clustering

Cited In (4)

• Eigenvalues and constraints in mixture modeling: geometric and computational issues
• Improved initialisation of model-based clustering using Gaussian hierarchical partitions
• Bayesian k-Means as a “Maximization-Expectation” Algorithm
• Variational approximation for importance sampling

Uses Software

• mclust
• L-BFGS-B
• L-BFGS
• LBFGS-B
• igraph
• e1071
• poLCA
• BayesLCA
• Algorithm 548

This page was built for publication: Improved model-based clustering performance using Bayesian initialization averaging

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q1729337)