Restriction to symmetric subgroups of unitary representations of rank one semisimple Lie groups (Q284824)

The authors investigate representations \(\pi_\mu\) of rank one Lie groups \(H = \mathrm{SO}_0(n,1;\mathbb F)\) for \(\mathbb F = \mathbb R, \mathbb C, \mathbb H\) and their restrictions to the subgroups \(H_1 = \mathrm{SO}_0(n - 1,1;\mathbb F)\). In general \(\mu \in \mathbb C\), but in this paper only complementary series for which \(\mu \in \mathbb R\) are considered. Such representations are unitarizable. It is proved that for certain parameters \(\mu\) the restricted representation of \(H\) has a direct summand which is isomorphic to a complementary series representation of the smaller group \(H_1\); similar results are also proved for the exceptional Lie groups \(H = F_{4(-20)}\) and \(H_1 = \mathrm{Spin}(8,1)\). The proofs use the imbedding of the complementary series into the analytic continuation of the holomorphic discrete series of the Lie groups \(G = \mathrm{SU}(n,1), \mathrm{SU}(n,1) \times \mathrm{SU}(n,1)\) and \( \mathrm{SU}(2n,2)\) and consider the branching of holomorphic representations.