Symmetry breaking operators for the restriction of representations of indefinite orthogonal groups \(O(p,q)\) (Q1695830)

If \(G\supset G'\) are reductive groups and \(\pi\) is an irreducible representation of \(G\), then the restriction of \(\pi\) to the subgroup \(G'\) is no more irreducible in general as a representation of \(G'\). If \(G\) is compact, then any irreducible \(\pi\) is finite-dimensional and splits into a finite direct sum \(\pi_{|G'}=\bigoplus\limits_{\pi'\in\widehat G'}m(\pi,\pi')\pi'\) of irreducibles \(\pi'\) of \(G'\) with multiplicities \(m(\pi,\pi')\). These multiplicities have been studied by various techniques including combinatorial algorithms. For non-compact \(G'\) and for infinite-dimensional \(\pi\), the restriction \(\pi_{|G'}\) is not always a direct sum of irreducible representations. So, to define the multiplicity in general, a continuous representation \(\pi^\infty\) of a continuous representation \(\pi\) of a Lie group on a Banach space \(\mathcal H\) is defined on the Fréchet space \(\mathcal H^\infty\) of \(C^\infty\)-vectors of \(\mathcal H\). For another representation \(\pi'\) of a subgroup \(G'\), the space of continuous \(G'\)-intertwining operators, or symmetry breaking operators, is defined as \(\text{Hom}_{G'}(\pi^\infty_{|G'},(\pi')^\infty)\), and \(m(\pi,\pi')=\dim\text{Hom}_{G'}(\pi^\infty_{|G'},(\pi')^\infty)\). In this paper, the authors construct and give a complete classification of symmetry breaking operators between most degenerate spherical principal series representations \(\pi=I(\lambda)\) of \(G\) and \(\pi'=J(\nu)\) of the subgroup \(G'\) for the pair of higher real rank groups \((G,G')=(O(p+1,q+1),O(p,q+1))\), where \(O(p+1,q+1)\) is the automorphism group of the quadratic form on \(\mathbb R^{p+q+2}\) of signature \((p+1,q+1)\) defined by \(Q_{p+1,q+1}(x)=x_0^2+x_1^2+\dots+x_p^2-x_{p+1}^2-\dots-x_{p+q+1}^2\), \(I(\lambda)=\text{Ind}^G_P(\mathbb C_\lambda)\), and the parameter \(\lambda\in\mathbb C\) of \(G\) is induced from a character \(\mathbb C_\lambda\) of a maximal parabolic subgroup \(P=MAN_+\) with Levi part \(MA\simeq O(p,q)\times\{\pm1\}\times\mathbb R\). The authors prove that \(m(I(\lambda),J(\nu))\in\{1,2\}\) for all \(\lambda,\nu\in\mathbb R\). Also, functional identities and residue formulas of the regular symmetry breaking operators are provided explicitly.