Estimating multi-index models with response-conditional least squares
From MaRDI portal
Publication:2044313
Abstract: The multi-index model is a simple yet powerful high-dimensional regression model which circumvents the curse of dimensionality assuming for some unknown index space and link function . In this paper we introduce a method for the estimation of the index space, and study the propagation error of an index space estimate in the regression of the link function. The proposed method approximates the index space by the span of linear regression slope coefficients computed over level sets of the data. Being based on ordinary least squares, our approach is easy to implement and computationally efficient. We prove a tight concentration bound that shows -convergence, but also faithfully describes the dependence on the chosen partition of level sets, hence giving indications on the hyperparameter tuning. The estimator's competitiveness is confirmed by extensive comparisons with state-of-the-art methods, both on synthetic and real data sets. As a second contribution, we establish minimax optimal generalization bounds for k-nearest neighbors and piecewise polynomial regression when trained on samples projected onto any -consistent estimate of the index space, thus providing complete and provable estimation of the multi-index model.
In the general area of sufficient dimension reduction, a novel methodology is derived and analyzed for estimation of the index space \(A\) of a high dimensional multi-index regression model \(E(Y|X) = g(A^T X)\) with \(g\) being a link function. The estimation method is based on the solution of localized least squares problems and uses the span of linear regression slope coefficients computed over level sets of the data. The propagation error of the index space estimate in the regression of the link function is studied in detail. The proposed method is shown to be attractive to practitioners by being computationally efficient and easy to implement as only one hyperparameter (the number of level sets) needs to be specified. Furthermore, finite sample generalization bounds for the regression model are provided which account for the projection error; in turn the bounds are used in quantifying the generalization bound for derived regression estimate and obtain that when response-conditional least squares are paired with piecewise polynomial regression, this leads to optimal estimation of the multi-index model.
Recommendations
- scientific article; zbMATH DE number 88830
- Specification and estimation of semiparametric multiple-index models
- Simultaneous estimation for semi-parametric multi-index models
- Estimation and variable selection for a class of quantile regression models with multiple index
- Variable selection and estimation for semi-parametric multiple-index models
- Multi-index regression models with missing covariates at random
- A novel regularization method for estimation and variable selection in multi-index models
- Approximate inferences in multiresponse regression analysis
- Estimation of a groupwise additive multiple-index model and its applications
Cites work
- A biased Kaczmarz algorithm for clustered equations
- A distribution-free theory of nonparametric regression
- A Multiple-Index Model and Dimension Reduction
- A new algorithm for estimating the effective dimension-reduction subspace
- A review on dimension reduction
- A robust and efficient estimation method for single index models
- A semiparametric approach to dimension reduction
- An Adaptive Estimation of Dimension Reduction Space
- Contour regression: a general approach to dimension reduction
- Determining the dimension of iterative Hessian transformation
- Dimension reduction for conditional mean in regression
- Direct estimation of the index coefficient in a single-index model
- Efficient estimation in single index models through smoothing splines
- Efficient estimation in sufficient dimension reduction
- Estimating covariance and precision matrices along subspaces
- High dimensional single index models
- High-dimensional probability. An introduction with applications in data science
- scientific article; zbMATH DE number 6676486 (Why is no real title available?)
- scientific article; zbMATH DE number 1220060 (Why is no real title available?)
- Investigating Smooth Multiple Regression by the Method of Average Derivatives
- Learning functions of few arbitrary linear parameters in high dimensions
- Linear Regression With a Sparse Parameter Vector
- Minimax Rates of Estimation for High-Dimensional Linear Regression Over \ell_q-Balls
- Multi-index regression models with missing covariates at random
- Multidimensional binary search trees used for associative searching
- Nonasymptotic upper bounds for the reconstruction error of PCA
- On Directional Regression for Dimension Reduction
- On hybrid methods of inverse regression-based algorithms
- On Principal Hessian Directions for Data Visualization and Dimension Reduction: Another Application of Stein's Lemma
- On strict sub-Gaussianity, optimal proxy variance and symmetry for bounded random variables
- On the Interpretation of Regression Plots
- Optimal global rates of convergence for nonparametric regression
- Probability and computing. Randomization and probabilistic techniques in algorithms and data analysis
- Rates of convergence for partitioning and nearest neighbor regression estimates with unbounded data
- Robust estimates in generalized partially linear single-index models
- Save: a method for dimension reduction and graphics in regression
- Single and multiple index functional regression models with nonparametric link
- Sliced Inverse Regression for Dimension Reduction
- Sliced Regression for Dimension Reduction
- Structure adaptive approach for dimension reduction.
- Sufficient dimension reduction and prediction in regression
- Sufficient dimension reduction: methods and applications with R
- Universal algorithms for learning theory. II: Piecewise polynomial functions
Cited in
(10)- Structure adaptive approach for dimension reduction.
- Multiscale regression on unknown manifolds
- Conditional regression for single-index models
- Nonlinear generalization of the monotone single index model
- A note on least squares sensitivity in single-index model estimation and the benefits of response transformations
- An alternating determination-optimization approach for an additive multi-index model
- Survey on algorithms for multi-index models
- Efficient estimation of the central mean subspace via smoothed gradient outer products
- Robust direction estimation in single-index models via cumulative divergence
- PAC-Bayes estimation for high-dimensional multi-index models with unknown active dimension
This page was built for publication: Estimating multi-index models with response-conditional least squares
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2044313)
