SIMD parallel MCMC sampling with applications for big-data Bayesian analytics
From MaRDI portal
Publication:1663281
DOI10.1016/J.CSDA.2015.02.010zbMATH Open1468.62131arXiv1310.1537OpenAlexW2013399084MaRDI QIDQ1663281FDOQ1663281
Mansour T. A. Sharabiani, A. Mahani
Publication date: 21 August 2018
Published in: Computational Statistics and Data Analysis (Search for Journal in Brave)
Abstract: Computational intensity and sequential nature of estimation techniques for Bayesian methods in statistics and machine learning, combined with their increasing applications for big data analytics, necessitate both the identification of potential opportunities to parallelize techniques such as MCMC sampling, and the development of general strategies for mapping such parallel algorithms to modern CPUs in order to elicit the performance up the compute-based and/or memory-based hardware limits. Two opportunities for Single-Instruction Multiple-Data (SIMD) parallelization of MCMC sampling for probabilistic graphical models are presented. In exchangeable models with many observations such as Bayesian Generalized Linear Models, child-node contributions to the conditional posterior of each node can be calculated concurrently. In undirected graphs with discrete nodes, concurrent sampling of conditionally-independent nodes can be transformed into a SIMD form. High-performance libraries with multi-threading and vectorization capabilities can be readily applied to such SIMD opportunities to gain decent speedup, while a series of high-level source-code and runtime modifications provide further performance boost by reducing parallelization overhead and increasing data locality for NUMA architectures. For big-data Bayesian GLM graphs, the end-result is a routine for evaluating the conditional posterior and its gradient vector that is 5 times faster than a naive implementation using (built-in) multi-threaded Intel MKL BLAS, and reaches within the striking distance of the memory-bandwidth-induced hardware limit. The proposed optimization strategies improve the scaling of performance with number of cores and width of vector units (applicable to many-core SIMD processors such as Intel Xeon Phi and GPUs), resulting in cost-effectiveness, energy efficiency, and higher speed on multi-core x86 processors.
Full work available at URL: https://arxiv.org/abs/1310.1537
Recommendations
- Double-parallel Monte Carlo for Bayesian analysis of big data
- Parallel Markov chain Monte Carlo for Bayesian hierarchical models with big data, in two stages
- Parallelizing MCMC sampling via space partitioning
- Parallel MCMC with generalized elliptical slice sampling
- Accelerated parallel non-conjugate sampling for Bayesian non-parametric models
- Parallel MCMC methods for global optimization
- Parallel local approximation MCMC for expensive models
- Population-Based MCMC on Multi-Core CPUs, GPUs and FPGAs
Computational methods for problems pertaining to statistics (62-08) Bayesian inference (62F15) Statistical aspects of big data and data science (62R07)
Cites Work
- ImageNet
- The No-U-Turn Sampler: Adaptively Setting Path Lengths in Hamiltonian Monte Carlo
- Slice sampling. (With discussions and rejoinder)
- PRMLT
- Title not available (Why is that?)
- Hierarchical Bayesian Analysis of Changepoint Problems
- Pattern recognition and machine learning.
- Title not available (Why is that?)
- Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images
- Adaptive Rejection Sampling for Gibbs Sampling
- Monte Carlo sampling methods using Markov chains and their applications
- Parallel algorithms for linear models. Numerical methods and estimation problems
- Bayesian Statistics and Marketing
- Equation of State Calculations by Fast Computing Machines
- Numerical recipes. The art of scientific computing.
- Riemann Manifold Langevin and Hamiltonian Monte Carlo Methods
- Neural networks and physical systems with emergent collective computational abilities.
- The Monte Carlo Method
- A simple method for generating gamma variables
- Title not available (Why is that?)
- A Fast Learning Algorithm for Deep Belief Nets
- Actuarial statistics with generalized linear mixed models
- Parallel multivariate slice sampling
- Recursive least-squares using a hybrid Householder algorithm on massively parallel SIMD systems.
- SOLVING THE UPDATED AND DOWNDATED ORDINARY LINEAR MODEL ON MASSIVELY PARALLEL SIMD SYSTEMS
- Efficient parallelisation of Metropolis-Hastings algorithms using a prefetching approach
- Title not available (Why is that?)
- Title not available (Why is that?)
- Distributed algorithms for topic models
- Ordinary linear model estimation on a massively parallel SIMD computer
Cited In (5)
- Multifidelity multilevel Monte Carlo to accelerate approximate Bayesian parameter inference for partially observed stochastic processes
- Ensemble decision forest of RBF networks via hybrid feature clustering approach for high-dimensional data classification
- Vector operations for accelerating expensive Bayesian computations - a tutorial guide
- Powered embarrassing parallel MCMC sampling in Bayesian inference, a weighted average intuition
- A faster algorithm to estimate multiresolution densities
Uses Software
This page was built for publication: SIMD parallel MCMC sampling with applications for big-data Bayesian analytics
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q1663281)
