On the Optimality of the Discrete Karhunen--Loève Expansion (Q4210194)

The aim of the paper is to present a short proof of the following assertion. Theorem: Let \(X = (X_{1}, \ldots, X_{n})^{T}\) be a real-valued random vector with finite second moments, and let \(m\) be a fixed natural number not greater than \(n.\) Let \(a\) and \(A\) denote the expectation vector or the covariance matrix of \(X,\) respectively, and let \(\lambda_{1} \geq \lambda_{2} \geq \ldots \lambda_{n} \) be the eigenvalues of \(A\) with corresponding orthonormal eigenvectors \(w_{1}, \ldots, w_{n}.\) Then, for any vectors \(v_{0}, \ldots, v_{m}\) and any linear functionals \(\Phi_{1}, \ldots, \Phi_{m}\): \(R^{n} \rightarrow R\), the expectation \[ {\mathcal E}( \| X - (v_{0} + \sum_{i=1}^{m} \Phi_{i}(X - a)v_{i}) \| ^{2}) \] is minimal if \(v_{0} = a, v_{i} = w_{i},\) \(\Phi_i(x)=\langle x, w_{i} \rangle \) for \(i = 1, \ldots, m.\) Hence \[ Y := a + \sum\limits_{i=1}^{m}\langle X -a, w_{i} \rangle w_{i} \] is the best \(m\)-dimensional linear approximation to \(X\) in the quadratic mean and is called the Karhuen-Loève expansion of \(X\) of order \(m.\) The approximation error is \[ {\mathcal E} (\| X - Y \| ^{2}) = \lambda_{m+1} + \ldots + \lambda_{n} \] (\( \| \cdot \| \) denote the Euclidean norm in \(R^{n}; \) \(\langle \cdot, \cdot \rangle\) the corresponding scalar product).