A similarity-based Bayesian mixture-of-experts model

DOI10.1007/S11222-023-10238-YzbMATH Open1517.62051arXiv2012.02130MaRDI QIDQ6173564FDOQ6173564

Authors: Tianfang Zhang, Rasmus Bokrantz, Jimmy Olsson 

Publication date: 21 July 2023

Published in: Statistics and Computing (Search for Journal in Brave)

Abstract: We present a new nonparametric mixture-of-experts model for multivariate regression problems, inspired by the probabilistic k-nearest neighbors algorithm. Using a conditionally specified model, predictions for out-of-sample inputs are based on similarities to each observed data point, yielding predictive distributions represented by Gaussian mixtures. Posterior inference is performed on the parameters of the mixture components as well as the distance metric using a mean-field variational Bayes algorithm accompanied with a stochastic gradient-based optimization procedure. The proposed method is especially advantageous in settings where inputs are of relatively high dimension in comparison to the data size, where input-output relationships are complex, and where predictive distributions may be skewed or multimodal. Computational studies on five datasets, of which two are synthetically generated, illustrate clear advantages of our mixture-of-experts method for high-dimensional inputs, outperforming competitor models both in terms of validation metrics and visual inspection.

Full work available at URL: https://arxiv.org/abs/2012.02130

Recommendations

• Adaptive-modal Bayesian nonparametric regression
• Bayesian regression and classification using mixtures of Gaussian processes
• Stochastic variational hierarchical mixture of sparse Gaussian processes for regression
• Mixture of experts architectures for neural networks as a special case of conditional expectation formula.
• Mixtures of regressions with predictor-dependent mixing proportions

zbMATH Keywords

variational inferencepseudolikelihood\(k\)-nearest neighborsnonparametric Bayesian regressionmixture-of-expertsreparameterization trick

Mathematics Subject Classification ID

Computational methods for problems pertaining to statistics (62-08) Bayesian inference (62F15) Nonparametric regression and quantile regression (62G08) Classification and discrimination; cluster analysis (statistical aspects) (62H30)

Cites Work

• Scikit-learn: machine learning in Python
• Automatic differentiation variational inference
• Gaussian parsimonious clustering models with covariates and a noise component
• Bayesian nonparametric data analysis
• Maximum likelihood estimation via the ECM algorithm: A general framework
• Gaussian processes for machine learning.
• Kernel stick-breaking processes
• Title not available (Why is that?)
• Bayesian curve fitting using multivariate normal mixtures
• Title not available (Why is that?)
• An ANOVA Model for Dependent Random Measures
• Pattern recognition and machine learning.
• Mixtures of Dirichlet processes with applications to Bayesian nonparametric problems
• A Bayesian analysis of some nonparametric problems
• Local statistical modeling via a cluster-weighted approach with elliptical distributions
• Machine learning. A probabilistic perspective
• A unifying view of sparse approximate Gaussian process regression
• Title not available (Why is that?)
• Variational inference for Dirichlet process mixtures
• Title not available (Why is that?)
• Bayesian Nonparametric Nonproportional Hazards Survival Modeling
• Title not available (Why is that?)
• Title not available (Why is that?)
• Divergence measures and a general framework for local variational approximation
• Sparse spatial autoregressions
• A Probabilistic Nearest Neighbour Method for Statistical Pattern Recognition
• A Bayesian Reassessment of Nearest-Neighbor Classification
• A class of mixtures of dependent tail-free processes
• Effect on prediction when modeling covariates in Bayesian nonparametric models
• On approximations via convolution-defined mixture models

Cited In (4)

• Guiding Local Regression Using Visualisation
• Smooth aggregation of Bayesian experts
• Title not available (Why is that?)
• Latent Features in Similarity Judgments: A Nonparametric Bayesian Approach

This page was built for publication: A similarity-based Bayesian mixture-of-experts model

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6173564)