Extrema of sums of heterogeneous quadratic forms (Q1375540): Difference between revisions

This big paper focuses on the following problem arising in various situations in multivariate statistical analysis. Find the maximum of the quadratic functional \[ f_{ \text{quad}} (X)=\sum^K_{i=1} x^T_iA_ix_i \] under the constraint that the vectors \(x_1, \dots, x_k\in \mathbb{R}^n\) form an orthonormal system where \(A_1, \dots, A_k\) \((k\leq n)\) are given \(k\) symmetric positive definite \(n\times n\) matrices and the \(n\times k\) matrix \(X=[x_1, \dots, x_k]\) contains the enumerated vectors as their columns. The set of orthonormal \(k\)-tuples in \(\mathbb{R}^n\) is called a Stiefel manifold and denoted by \(V_{n,k}\); \(X\in V_{n,k}\) when the set of column vectors of the \(n\times k\) matrix \(X\) is an element of \(V_{n,k}\). As \(V_{n,k}\) is a compact manifold and \(f_{\text{quad}}\) is continuous on \(V_{n,k}\) a finite global maximum exists and it is attained at some point. To characterize the critical points of the functional let us denote by \(A(X)= [A_1x_1, \dots, A_kx_k]\) the \(n\times k\) matrix. The main result is the following theorem. Theorem 3.1. \(X\in V_{n,k}\) is a critical point of \(f_{\text{quad}}\) if and only if \(A(X)=XS\), where \(S\) is a symmetric \(k\times k\) matrix. At an arbitrary critical point the matrix \(S=X^TA(X)\) is not necessarily positive semidefinite. The next lemma shows that this is necessary at the global maximum points of \(f_{\text{quad}}\). Lemma 3.1. If \(X\in V_{n,k}\) is a global maximum of the functional \(f_{\text{quad}}\), then the corresponding matrix \(S=X^T A(X)\) is positive semidefinite. Theorem 3.1 and Lemma 3.1 together yield that at global maximum points \(A(X)=XS\), where \(S\) is a positive semidefinite matrix. Beyond it a computational iteration algorithm is proposed. Choosing an arbitrary initial orthonormal system \(X^{(0)}= [x_1^{(0)}, \dots, x_k^{(0)}]\in V_{n,k}\), the sequence \(X^{(1)}, X^{(2)}, \dots\) in \(V_{n,k}\) is constructed in the following way: from the \(m\)th element of this sequence \(X^{(m)} =[x_1^{(m)}, \dots, x_k^{(m)}]\in V_{n,k}\) the next one is obtained by a polar decomposition of the matrix \(A(X^{(m)})\) as \(A(X^{(m)})= X^{(m+1)} S^{(m+1)}\) \((m=0,1, \dots,)\) where \((X^{(m+1)})^T X^{(m+1)} =I_k\) and \(S^{(m+1)}\geq 0\). This polar decomposition is unique if \(A_1x_1^{(m)}, \dots, A_kx_k^{(m)}\) are linearly independent. It is proved that \(f_{\text{quad}} (X^{(m)})\) is a nondecreasing sequence and \(\text{dist} (X^{(m)}, {\mathcal C})\to 0\) as \(m\to\infty\) where \({\mathcal C}= \{X\in V_{n,k}:X\) is a critical point of \(f_{\text{quad}}\}\). In the case when \(f_{\text{quad}}\) has isolated critical points the algorithm converges to one of the critical points. The critical points of the quadratic functional assuming some relations between the matrices \(A_1, \dots, A_k\) are considered. Structural properties of the functionals \(f_{\text{quad}}\) and \[ f_{\text{bilin}} (X,Y)= \sum^k_{i=1} y^T_iA_ix_i \] where \(x_1, \dots, x_k\) and \(y_1, \dots, y_k\) are orthonormal vectors are discussed.