Analysis on the minimal representation of \(\mathrm O(p,q)\). II: Branching laws. (Q1421385)

[For Part I, see ibid. 180, No. 2, 486--512 (2003; Zbl 1046.22004).] Let \(G\) be a Lie group and \(G'\) its subgroup, \(\widehat G\) the unitary dual of \(G\). If \(\pi\in\widehat G\), the restriction \(\pi_{\widehat G'}\) is generally not irreducible and we have an irreducible decomposition \[ \pi_{G'}\cong\int_{\widehat G'}m_{\pi}(\tau)\tau \,d\mu(\tau). \] This is the so-called branching law. The present paper is the second in a series of papers devoted to the analysis of the minimal representation \(\widetilde\omega^{p,q}\) of the indefinite orthogonal group \(G = O(p,q)\), i.e., from the point of view of conformal geometry, the authors find the restriction of \(\widetilde\omega^{p,q}\) with respect to the symmetric pair \((G,G') = (O(p,q),O(p'q')\times O(p'',q''))\): A canonical conclusion obtained by them is the branching law for \(O(pq)\downarrow O(p,q')\times O(q'')\): If \(q''\geq 1\) and \(q'+ q'' = q\), the twisted pull-back \(\widetilde\Phi{*_1}\) of the local conformal map between spheres and hyperboloids gives an explicit irreducible decomposition of the unitary representation \(\tilde\omega^{p,q}\) when restricted to \(O(p,q'')\times O(q'')\) \[ \widetilde\Phi*_1: \widetilde\omega^{p,q}|_{O(p,q')\times O(q'')} \to \sum_{l=0}^{\infty} \pi^{p,q'}_{+,l+q''/2-1}\otimes \mathcal H^l (R^{q''}), \] where the \(\mathcal H^l(R^q)\) are the usual spherical harmonics for the compact orthogonal group \(O(q)\) and the \(\pi^{p,q}_{+,\lambda}\) are the representations of the non-compact orthogonal group \(O(p,q)\), which can be regarded as discrete series representations on hyperboloids, or as cohomological induced representations from characters of certain \(\theta\)-stable parabolic subalgebras. The authors give also the Parseval-Plancherel formula for the above restriction: let \(\widetilde\bigtriangleup_M\) be the Yamabe operator on \(M = S^{p-1}\times S^{q-1}\) and \(F\in \operatorname{Ker} \widetilde\bigtriangleup_M\) and develop \(F\) as \(f =\sum_{l=0}^{\infty} F_l^{(1)}F_l^{(2)}\) according to the above decomposition, then \[ \| F\|^2_{\widetilde\omega^{p,q}}= \sum_{l=0}^{\infty}\| F^{(1)}\|^2_{\pi^{p,q'}_{+,l+q''/2-1}} \| F^{(2)}\|^2_{L^2 (S^{q''}-1)}. \] Finally the authors use certain Sobolev estimates to construct infinitely many discrete spectra when both factors in \(G'\) are non-compact, they also conjecture the form of the fall discrete spectrum.