Infinite dimensional representations of orthogonal groups of quadratic forms with finite index (Q6090094)

The author studies some representations that preserve a quadratic or Hermitian form with finite index. The real, complex and quaternionic cases are considered. On the contrary to the rank one case there is no exotic such representations. These representations are classified. The first result shows that there is no exotic continuous representation of finite dimensional Lie groups of higher rank. All the representations come from finite dimensional representations and unitary representations: if \(G\) is a connected simple non-compact Lie group with trivial center and real rank at least \(2\) and \(G\to PO_{\mathbb{K}}(p,\infty)\) is a continuous representation without totally isotropic invariant subspace, then the underlying Hilbert space splits orthogonally as \(E_1\oplus\dots\oplus E_n\oplus\mathcal{K}\). Here each \(E_i\) is a finite dimensional, non-degenerate, irreducible \(G\)-invariant subspace and the induced representation on \(\mathcal{K}\) is unitary. The proof uses previous results of the author for lattices and extends them to the ambient group. The second main result shows that if \(\rho: PO_{\mathbb{K}}(p,\infty)\to PO_{\mathbb{L}}(q,\infty)\) (\(\mathbb{K},\mathbb{L}\in\{\mathbb{R},\mathbb{C},\mathbb{H}\}\), \(p\geq 2\), \(q\geq 1\)) is a continuous geometrically dense representations, then \(\mathbb{K}=\mathbb{L}\), \(p=q\) and \(\rho\) is induced by an isomorphism of quadratic spaces.