Extrema of sums of heterogeneous quadratic forms (Q1375540)

This big paper focuses on the following problem arising in various situations in multivariate statistical analysis. Find the maximum of the quadratic functional \[ f_{ \text{quad}} (X)=\sum^K_{i=1} x^T_iA_ix_i \] under the constraint that the vectors \(x_1, \dots, x_k\in \mathbb{R}^n\) form an orthonormal system where \(A_1, \dots, A_k\) \((k\leq n)\) are given \(k\) symmetric positive definite \(n\times n\) matrices and the \(n\times k\) matrix \(X=[x_1, \dots, x_k]\) contains the enumerated vectors as their columns. The set of orthonormal \(k\)-tuples in \(\mathbb{R}^n\) is called a Stiefel manifold and denoted by \(V_{n,k}\); \(X\in V_{n,k}\) when the set of column vectors of the \(n\times k\) matrix \(X\) is an element of \(V_{n,k}\). As \(V_{n,k}\) is a compact manifold and \(f_{\text{quad}}\) is continuous on \(V_{n,k}\) a finite global maximum exists and it is attained at some point. To characterize the critical points of the functional let us denote by \(A(X)= [A_1x_1, \dots, A_kx_k]\) the \(n\times k\) matrix. The main result is the following theorem. Theorem 3.1. \(X\in V_{n,k}\) is a critical point of \(f_{\text{quad}}\) if and only if \(A(X)=XS\), where \(S\) is a symmetric \(k\times k\) matrix. At an arbitrary critical point the matrix \(S=X^TA(X)\) is not necessarily positive semidefinite. The next lemma shows that this is necessary at the global maximum points of \(f_{\text{quad}}\). Lemma 3.1. If \(X\in V_{n,k}\) is a global maximum of the functional \(f_{\text{quad}}\), then the corresponding matrix \(S=X^T A(X)\) is positive semidefinite. Theorem 3.1 and Lemma 3.1 together yield that at global maximum points \(A(X)=XS\), where \(S\) is a positive semidefinite matrix. Beyond it a computational iteration algorithm is proposed. Choosing an arbitrary initial orthonormal system \(X^{(0)}= [x_1^{(0)}, \dots, x_k^{(0)}]\in V_{n,k}\), the sequence \(X^{(1)}, X^{(2)}, \dots\) in \(V_{n,k}\) is constructed in the following way: from the \(m\)th element of this sequence \(X^{(m)} =[x_1^{(m)}, \dots, x_k^{(m)}]\in V_{n,k}\) the next one is obtained by a polar decomposition of the matrix \(A(X^{(m)})\) as \(A(X^{(m)})= X^{(m+1)} S^{(m+1)}\) \((m=0,1, \dots,)\) where \((X^{(m+1)})^T X^{(m+1)} =I_k\) and \(S^{(m+1)}\geq 0\). This polar decomposition is unique if \(A_1x_1^{(m)}, \dots, A_kx_k^{(m)}\) are linearly independent. It is proved that \(f_{\text{quad}} (X^{(m)})\) is a nondecreasing sequence and \(\text{dist} (X^{(m)}, {\mathcal C})\to 0\) as \(m\to\infty\) where \({\mathcal C}= \{X\in V_{n,k}:X\) is a critical point of \(f_{\text{quad}}\}\). In the case when \(f_{\text{quad}}\) has isolated critical points the algorithm converges to one of the critical points. The critical points of the quadratic functional assuming some relations between the matrices \(A_1, \dots, A_k\) are considered. Structural properties of the functionals \(f_{\text{quad}}\) and \[ f_{\text{bilin}} (X,Y)= \sum^k_{i=1} y^T_iA_ix_i \] where \(x_1, \dots, x_k\) and \(y_1, \dots, y_k\) are orthonormal vectors are discussed.