Component selection and smoothing in multivariate nonparametric regression

DOI10.1214/009053606000000722MaRDI QIDQ869970zbMATH OpenOpenAlexFDO

Authors Yi Lin, Hao Helen Zhang

Publication date 12 March 2007

Published in The Annals of Statistics (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/math/0702659

zbMATH Keywords

model selectionnonparametric classificationsmoothing splinemethod of regularization

Mathematics Subject Classification ID

Nonparametric regression and quantile regression (62G08) Asymptotic properties of nonparametric inference (62G20) Analysis of variance and covariance (ANOVA) (62J10)

Abstract: We propose a new method for model selection and model fitting in multivariate nonparametric regression models, in the framework of smoothing spline ANOVA. The ``COSSO is a method of regularization with the penalty functional being the sum of component norms, instead of the squared norm employed in the traditional smoothing spline method. The COSSO provides a unified framework for several recent proposals for model selection in linear models and smoothing spline ANOVA models. Theoretical properties, such as the existence and the rate of convergence of the COSSO estimator, are studied. In the special case of a tensor product design with periodic functions, a detailed analysis reveals that the COSSO does model selection by applying a novel soft thresholding type operation to the function components. We give an equivalent formulation of the COSSO estimator which leads naturally to an iterative algorithm. We compare the COSSO with MARS, a popular method that builds functional ANOVA models, in simulations and real examples. The COSSO method can be extended to classification problems and we compare its performance with those of a number of machine learning algorithms on real datasets. The COSSO gives very competitive performance in these studies.

Recommendations

• Component selection and smoothing for nonparametric regression in exponential families
• Selection of components and degrees of smoothing via Lasso in high dimensional nonparametric additive models
• Smoothing parameter selection for nonparametric regression using smoothing spline
• scientific article; zbMATH DE number 1036034
• Variable selection in nonparametric regression with continuous covariates
• Gradient-based smoothing parameter selection for nonparametric regression estimation
• Simultaneous selection of variables and smoothing parameters in structured additive regression models
• Variable selection in nonparametric functional concurrent regression

Cites work

• scientific article; zbMATH DE number 5654889 (Why is no real title available?)
• scientific article; zbMATH DE number 3703820 (Why is no real title available?)
• scientific article; zbMATH DE number 45848 (Why is no real title available?)
• scientific article; zbMATH DE number 472952 (Why is no real title available?)
• scientific article; zbMATH DE number 845714 (Why is no real title available?)
• A Statistical View of Some Chemometrics Regression Tools
• Adaptive Model Selection
• Atomic Decomposition by Basis Pursuit
• Benchmarking least squares support vector machine classifiers
• Better Subset Regression Using the Nonnegative Garrote
• Inference After Model Selection
• Least angle regression. (With discussion)
• Multivariate adaptive regression splines
• Natural spline functions, their associated eigenvalue problem
• On Measuring and Correcting the Effects of Data Mining and Model Selection
• Smoothing noisy data with spline functions: Estimating the correct degree of smoothing by the method of generalized cross-validation
• Smoothing spline ANOVA for exponential families, with application to the Wisconsin epidemiological study of diabetic retinopathy. (The 1994 Neyman Memorial Lecture)
• Smoothing spline ANOVA models
• Structural modelling with sparse kernels
• Variable Selection and Model Building via Likelihood Basis Pursuit
• Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties

Cited in

(only showing first 100 items - show all)

• Kernel Ordinary Differential Equations
• Statistical inference in sparse high-dimensional additive models
• Variable selection based on squared derivative averages
• Estimating functions and derivatives via adaptive penalized splines
• Kernel continuum regression
• Choosing shape parameters for regression in reproducing kernel Hilbert space and variable selection
• Nonparametric Interaction Selection
• Nonparametric augmented probability weighting with sparsity
• Spike-and-slab priors for function selection in structured additive regression models
• Variable selection in robust semiparametric modeling for longitudinal data
• Spectrally Sparse Nonparametric Regression via Elastic Net Regularized Smoothers
• A sparse additive model for high-dimensional interactions with an exposure variable
• Bayesian pathway selection
• Transposable regularized covariance models with an application to missing data imputation
• Nonparametric additive beta regression for fractional response with application to body fat data
• A reluctant additive model framework for interpretable nonlinear individualized treatment rules
• Input selection and shrinkage in multiresponse linear regression
• Kernel Cox partially linear regression: building predictive models for cancer patients' survival
• Joint semiparametric kernel network regression
• Block-wise primal-dual algorithms for large-scale doubly penalized ANOVA modeling
• Selecting the amount of smoothing in nonparametric regression estimation for complex surveys
• Improving the prediction performance of the Lasso by subtracting the additive structural noises
• Model selection in nonparametric hazard regression
• Feature selection algorithms in generalized additive models under concurvity
• Empirical Bayes oracle uncertainty quantification for regression
• Lag selection in stochastic additive models
• D-learning to estimate optimal individual treatment rules
• P-splines with an \(\ell_1\) penalty for repeated measures
• Learning non-parametric basis independent models from point queries via low-rank methods
• Additive monotone regression in high and lower dimensions
• Parametric component detection and variable selection in varying-coefficient partially linear models
• Metamodel construction for sensitivity analysis
• Layer-wise learning strategy for nonparametric tensor product smoothing spline regression and graphical models
• Variable selection in functional additive regression models
• An analysis of penalized interaction models
• Local independence feature screening for nonparametric and semiparametric models by marginal empirical likelihood
• Oracle inequalities for sparse additive quantile regression in reproducing kernel Hilbert space
• Multiresolution functional ANOVA for large-scale, many-input computer experiments
• Domain selection for the varying coefficient model via local polynomial regression
• Partially linear structure identification in generalized additive models with NP-dimensionality
• Regularization and model selection for quantile varying coefficient model with categorical effect modifiers
• Estimation of sparse functional additive models with adaptive group Lasso
• Using recursive algorithms for the efficient identification of smoothing spline ANOVA models
• Bayesian neural networks for selection of drug sensitive genes
• Kernel variable selection for multicategory support vector machines
• On the strong oracle property of concave penalized estimators with infinite penalty derivative at the origin
• Symmetrical and non-symmetrical variants of three-way correspondence analysis for ordered variables
• Semiparametric regression models with additive nonparametric components and high dimensional parametric components
• Saturating splines and feature selection
• Nonparametric variable selection and its application to additive models
• A hierarchical multiple kernel support vector machine for customer churn prediction using longitudinal behavioral data
• Additive models with trend filtering
• A distance metric-based space-filling subsampling method for nonparametric models
• Learning sparse gradients for variable selection and dimension reduction
• Grouped variable selection with discrete optimization: computational and statistical perspectives
• HARFE: hard-ridge random feature expansion
• Efficient kernel-based variable selection with sparsistency
• Testing for additivity in non-parametric regression
• Additive regression splines with total variation and non negative garrote penalties
• Robust penalized logistic regression with truncated loss functions
• Least angle regression for model selection
• Block-based refitting in \(\ell_{12}\) sparse regularization
• Robust spline-based variable selection in varying coefficient model
• Functional index coefficient models with variable selection
• Feature selection in the Laplacian support vector machine
• An Outer-Product-of-Gradient Approach to Dimension Reduction and its Application to Classification in High Dimensional Space
• A General Framework of Nonparametric Feature Selection in High-Dimensional Data
• A penalized least product relative error loss function based on wavelet decomposition for non-parametric multiplicative additive models
• Smoothing Splines Approximation Using Hilbert Curve Basis Selection
• Sensitivity analysis in classification using Bayesian smoothing spline ANOVA probit regression
• Elicitation of multiattribute value functions through high dimensional model representations: monotonicity and interactions
• Orthogonalized Kernel Debiased Machine Learning for Multimodal Data Analysis
• Component selection in the additive regression model
• Structured kernel quantile regression
• Automatic structure recovery for generalized additive models
• Measures of Uncertainty for Shrinkage Model Selection
• Another look at linear programming for feature selection via methods of regularization
• Bayesian group Lasso for nonparametric varying-coefficient models with application to functional genome-wide association studies
• A hierarchical integrative group least absolute shrinkage and selection operator for analyzing environmental mixtures
• A fused Lasso approach to nonstationary spatial covariance estimation
• Doubly penalized estimation in additive regression with high-dimensional data
• Adaptive penalized splines for data smoothing
• Integrative weighted group Lasso and generalized local quadratic approximation
• Time-varying quantile single-index model for multivariate responses
• Learning sparse conditional distribution: an efficient kernel-based approach
• Penalized variable selection procedure for Cox models with semiparametric relative risk
• Bayesian variable selection via a benchmark in normal linear models
• Efficient functional Lasso kernel smoothing for high-dimensional additive regression
• Convex Bidirectional Large Margin Classifiers
• Data-adaptive additive modeling
• Univariate measurement error selection likelihood for variable selection of additive model
• Refined generalization bounds of gradient learning over reproducing kernel Hilbert spaces
• Combining meta-modeling and categorical indicators for global sensitivity analysis of long-running flow simulators with spatially dependent inputs
• Sparse optimization for nonconvex group penalized estimation
• Partially linear monotone methods with automatic variable selection and monotonicity direction discovery
• Structured estimation for the nonparametric Cox model
• Bias-corrected inference for multivariate nonparametric regression: model selection and oracle property
• Modal additive models with data-driven structure identification
• Interpretable machine learning: fundamental principles and 10 grand challenges
• An RKHS-based approach to double-penalized regression in high-dimensional partially linear models

Describes a project that uses

Uses Software

• gss
• PDCO

This page was built for publication: Component selection and smoothing in multivariate nonparametric regression

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q869970)