Linear unlearning for cross-validation

DOI10.1007/BF02124747MaRDI QIDQ1923892zbMATH OpenOpenAlexFDO

Publication date 2 September 1997

Published in Advances in Computational Mathematics (Search for Journal in Brave)

Full work available at URL https://doi.org/10.1007/bf02124747

cross-validation input-output distribution linear unlearning neural network nonlinear regression quadratic approximation time series prediction

Mathematics Subject Classification ID

General nonlinear regression (62J02) Time series, auto-correlation, regression, etc. in statistics (GARCH) (62M10) Inference from stochastic processes and prediction (62M20) Probabilistic methods, stochastic differential equations (65C99)

Review text

Consider nonlinear regression in which the output \(y\) is regressed nonlinearly on the input vector \(x\). The authors focus on a neural network implementation, in which the output is predicted by \(\widehat y=F(x,w)\), where \(F(\cdot)\) denotes the nonlinear mapping of the neural net and \(w\) is the vector of network parameters. The conditional input-output distribution, i.e., the probability distribution of the output conditioned on a test input, is a basic objective for neural net modeling. A main source of uncertainty, when estimating the parameters of the conditional distribution, is the random selection of training data. The idea of cross-validation in neural net learning is based on training and testing on disjunct subsets resampled from the database, forming the cross-validation ensemble of models. The leave-one-out (LOO) ensemble of networks trained on all subsets leaving out one training example is an attractive -- though computationally expensive -- vehicle for generalization assessment of a neural network model. For the conventional neural net approaches unlearning of examples is not possible, and one basically has to train the full ensemble of networks, making the approach computationally unfeasible. This paper suggests the use of linear unlearning of examples to approximate the computationally expensive LOO cross-validation technique. It is assumed that unlearning of a single example only affects the network weights slightly. Under this hypothesis the change in the network parameters within the quadratic approximation of the network cost function is estimated. Using the ensemble an estimator for the test error of a regularized network is derived. The possibility of employing the ensemble of networks produced by the cross-validation scheme for constructing an ensemble predictor is analyzed. Considering a linear combination of networks, it is shown that the generalization performance is identical to that of using a single network trained on the full set of data. Numerical studies on the sunspot time series prediction benchmark demonstrates the viability of this approach.

Recommendations

Cites work

Cited in

(3)

This page was built for publication: Linear unlearning for cross-validation

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q1923892)