SIMD parallel MCMC sampling with applications for big-data Bayesian analytics
From MaRDI portal
Abstract: Computational intensity and sequential nature of estimation techniques for Bayesian methods in statistics and machine learning, combined with their increasing applications for big data analytics, necessitate both the identification of potential opportunities to parallelize techniques such as MCMC sampling, and the development of general strategies for mapping such parallel algorithms to modern CPUs in order to elicit the performance up the compute-based and/or memory-based hardware limits. Two opportunities for Single-Instruction Multiple-Data (SIMD) parallelization of MCMC sampling for probabilistic graphical models are presented. In exchangeable models with many observations such as Bayesian Generalized Linear Models, child-node contributions to the conditional posterior of each node can be calculated concurrently. In undirected graphs with discrete nodes, concurrent sampling of conditionally-independent nodes can be transformed into a SIMD form. High-performance libraries with multi-threading and vectorization capabilities can be readily applied to such SIMD opportunities to gain decent speedup, while a series of high-level source-code and runtime modifications provide further performance boost by reducing parallelization overhead and increasing data locality for NUMA architectures. For big-data Bayesian GLM graphs, the end-result is a routine for evaluating the conditional posterior and its gradient vector that is 5 times faster than a naive implementation using (built-in) multi-threaded Intel MKL BLAS, and reaches within the striking distance of the memory-bandwidth-induced hardware limit. The proposed optimization strategies improve the scaling of performance with number of cores and width of vector units (applicable to many-core SIMD processors such as Intel Xeon Phi and GPUs), resulting in cost-effectiveness, energy efficiency, and higher speed on multi-core x86 processors.
Recommendations
- Double-parallel Monte Carlo for Bayesian analysis of big data
- Parallel Markov chain Monte Carlo for Bayesian hierarchical models with big data, in two stages
- Parallelizing MCMC sampling via space partitioning
- Parallel MCMC with generalized elliptical slice sampling
- Accelerated parallel non-conjugate sampling for Bayesian non-parametric models
- Parallel MCMC methods for global optimization
- Parallel local approximation MCMC for expensive models
- Population-Based MCMC on Multi-Core CPUs, GPUs and FPGAs
Cites work
- scientific article; zbMATH DE number 1247560 (Why is no real title available?)
- scientific article; zbMATH DE number 1374541 (Why is no real title available?)
- scientific article; zbMATH DE number 2117879 (Why is no real title available?)
- scientific article; zbMATH DE number 814821 (Why is no real title available?)
- scientific article; zbMATH DE number 5060482 (Why is no real title available?)
- A Fast Learning Algorithm for Deep Belief Nets
- A simple method for generating gamma variables
- Actuarial statistics with generalized linear mixed models
- Adaptive Rejection Sampling for Gibbs Sampling
- Bayesian Statistics and Marketing
- Distributed algorithms for topic models
- Efficient parallelisation of Metropolis-Hastings algorithms using a prefetching approach
- Equation of state calculations by fast computing machines
- Hierarchical Bayesian Analysis of Changepoint Problems
- ImageNet
- Monte Carlo sampling methods using Markov chains and their applications
- Neural networks and physical systems with emergent collective computational abilities
- Numerical recipes. The art of scientific computing.
- Ordinary linear model estimation on a massively parallel SIMD computer
- PRMLT
- Parallel algorithms for linear models. Numerical methods and estimation problems
- Parallel multivariate slice sampling
- Pattern recognition and machine learning.
- Recursive least-squares using a hybrid Householder algorithm on massively parallel SIMD systems.
- Riemann manifold Langevin and Hamiltonian Monte Carlo methods. With discussion and authors' reply
- SOLVING THE UPDATED AND DOWNDATED ORDINARY LINEAR MODEL ON MASSIVELY PARALLEL SIMD SYSTEMS
- Slice sampling. (With discussions and rejoinder)
- Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images
- The Monte Carlo Method
- The no-U-turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo
Cited in
(5)- Powered embarrassing parallel MCMC sampling in Bayesian inference, a weighted average intuition
- Multifidelity multilevel Monte Carlo to accelerate approximate Bayesian parameter inference for partially observed stochastic processes
- A faster algorithm to estimate multiresolution densities
- Ensemble decision forest of RBF networks via hybrid feature clustering approach for high-dimensional data classification
- Vector operations for accelerating expensive Bayesian computations - a tutorial guide
Describes a project that uses
Uses Software
This page was built for publication: SIMD parallel MCMC sampling with applications for big-data Bayesian analytics
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q1663281)
