Stochastic Tree Ensembles for Regularized Nonlinear Regression

DOI10.1080/01621459.2021.1942012MaRDI QIDQ6107230zbMATH OpenOpenAlexFDO

Authors Jingyu He, P. Richard Hahn

Publication date 3 July 2023

Published in Journal of the American Statistical Association (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2002.03375, https://figshare.com/articles/journal_contribution/Stochastic_tree_ensembles_for_regularized_nonlinear_regression/14779373

zbMATH Keywords

Bayesianmachine learningMarkov chain Monte Carloregression treessupervised learning

Mathematics Subject Classification ID

Computational methods for problems pertaining to statistics (62-08) Bayesian inference (62F15) General nonlinear regression (62J02) Monte Carlo methods (65C05) Learning and adaptive systems in artificial intelligence (68T05)

Abstract: This paper develops a novel stochastic tree ensemble method for nonlinear regression, which we refer to as XBART, short for Accelerated Bayesian Additive Regression Trees. By combining regularization and stochastic search strategies from Bayesian modeling with computationally efficient techniques from recursive partitioning approaches, the new method attains state-of-the-art performance: in many settings it is both faster and more accurate than the widely-used XGBoost algorithm. Via careful simulation studies, we demonstrate that our new approach provides accurate point-wise estimates of the mean function and does so faster than popular alternatives, such as BART, XGBoost and neural networks (using Keras). We also prove a number of basic theoretical results about the new algorithm, including consistency of the single tree version of the model and stationarity of the Markov chain produced by the ensemble version. Furthermore, we demonstrate that initializing standard Bayesian additive regression trees Markov chain Monte Carlo (MCMC) at XBART-fitted trees considerably improves credible interval coverage and reduces total run-time.

Recommendations

• BART: Bayesian additive regression trees
• Bayesian regression tree ensembles that adapt to smoothness and sparsity
• Bayesian additive regression trees with model trees
• Posterior concentration for Bayesian regression trees and forests
• Bayesian regression trees for high-dimensional prediction and variable selection

Cites work

• BART: Bayesian additive regression trees
• Bayesian regression trees for high-dimensional prediction and variable selection
• Consistency of random forests
• Distribution-free predictive inference for regression
• Essentials of Stochastic Processes
• Greedy function approximation: A gradient boosting machine.
• scientific article; zbMATH DE number 3860199 (Why is no real title available?)
• Perturbations, optimization, and statistics
• Random forests
• Stochastic gradient boosting.

Cited in

(10)

• Tree ensembles with rule structured horseshoe regularization
• Regularization-based model tree for multi-output regression
• BART with targeted smoothing: an analysis of patient-specific stillbirth risk
• Near-optimal nonlinear regression trees
• Bayesian regression tree ensembles that adapt to smoothness and sparsity
• GP-BART: a novel Bayesian additive regression trees approach using Gaussian processes
• Tree-based ensembles for multi-output regression: comparing multivariate approaches with separate univariate ones
• The how and why of Bayesian nonparametric causal inference
• On mixing rates for Bayesian CART
• stochtree

This page was built for publication: Stochastic Tree Ensembles for Regularized Nonlinear Regression

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6107230)