Accelerated Componentwise Gradient Boosting Using Efficient Data Representation and Momentum-Based Optimization

DOI10.1080/10618600.2022.2116446MaRDI QIDQ6094092zbMATH OpenOpenAlexFDO

Authors Bernd Bischl, David Rügamer

Publication date 9 October 2023

Published in Journal of Computational and Graphical Statistics (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2110.03513, https://figshare.com/articles/journal_contribution/Accelerated_Componentwise_Gradient_Boosting_using_Efficient_Data_Representation_and_Momentum-based_Optimization/20604833

zbMATH Keywords

binningdata structuresfunctional gradient descentmachine learningNesterov momentum

Mathematics Subject Classification ID

Statistics (62-XX)

Abstract: Componentwise boosting (CWB), also known as model-based boosting, is a variant of gradient boosting that builds on additive models as base learners to ensure interpretability. CWB is thus often used in research areas where models are employed as tools to explain relationships in data. One downside of CWB is its computational complexity in terms of memory and runtime. In this paper, we propose two techniques to overcome these issues without losing the properties of CWB: feature discretization of numerical features and incorporating Nesterov momentum into functional gradient descent. As the latter can be prone to early overfitting, we also propose a hybrid approach that prevents a possibly diverging gradient descent routine while ensuring faster convergence. We perform extensive benchmarks on multiple simulated and real-world data sets to demonstrate the improvements in runtime and memory consumption while maintaining state-of-the-art estimation and prediction performance.

Cites work

• Boosting algorithms: regularization, prediction and model fitting
• Boosting With theL₂Loss
• Faster model matrix crossproducts for large generalized linear models with discretized covariates
• Flexible smoothing with B-splines and penalties. With comments and a rejoinder by the authors
• Greedy function approximation: A gradient boosting machine.
• scientific article; zbMATH DE number 3850830 (Why is no real title available?)
• Hyperband: a novel bandit-based approach to hyperparameter optimization
• Inference for L₂-boosting
• Multilevel structured additive regression
• Probing for sparse and fast variable selection with model-based boosting
• Process consistency for AdaBoost.
• Sparse matrix test problems

Cited in

(1)

• Privacy-preserving and lossless distributed estimation of high-dimensional generalized additive mixed models

This page was built for publication: Accelerated Componentwise Gradient Boosting Using Efficient Data Representation and Momentum-Based Optimization

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6094092)