Common invariant subspaces of two matrices (Q1301299)

A finite rational procedure of Shemesh type is proposed to check whether given complex \(n\times n\) matrices \(A\) and \(B\) have a common invariant subspace of dimension \(k\), \(2\leq k<n\). The only restriction is that at least one of \(A\) and \(B\) must have distinct eigenvalues. This fact can also be verified by a finite rational computation. The main theorem is in Section 3; Section 4 contains the algorithm with remarks.