Symmetry breaking for representations of rank one orthogonal groups. II (Q1991077)

These Lecture Notes in Mathematics are the continuation of a thorough study of symmetry breaking operators for representations of rank one orthogonal groups initiated by two leading experts in representation theory of Lie groups. The text under review consists of sixteen chapters for a total of 342 pages, including an introduction, three appendices and a complete list of references. The first volume [the authors, Symmetry breaking for representations of rank one orthogonal groups. Providence, RI: American Mathematical Society (AMS) (2015; Zbl 1334.22015)] is the 216th publication of the first author since 1986. Let \(G\) be a real reductive Lie group with complexified Lie algebra \(\mathfrak{g}\) and \(K\) a maximal compact subgroup of \(G\). Let \(\Pi\) be a continuous representation of \(G\) on a complete locally convex topological vector space \(\mathcal{H}\). Assume that \(\Pi\) is admissible of finite length and \(\mathcal{H}\) is a Banach space. The group \(G\) acts on the Fréchet space \(\mathcal{H}^\infty\) of smooth vectors. This admissible smooth representation, still denoted by \(\Pi\), is encoded by the underlying \((\mathfrak{g},K)\)-module \(\mathcal{H}_K\) of \(K\)-finite vectors. Write \(\text{Irr}(G)\) for the set of equivalence classes of irreducible admissible smooth representations of \(G\). Let \(G'\subset G\) be a closed subgroup of \(G\). Given an admissible smooth representation \(\pi\) of \(G'\), consider the complex vector space \(\Hom_{G'}(\Pi_{\mid_{G'}},\pi)\) of continuous \(G'\)-intertwinig operators, referred to as symmetry breaking operators. The multiplicity of \(\pi\) occuring in the restriction \(\Pi_{\mid_{G'}}\) of \(\Pi\) to \(G'\) is defined as the (possibly infinite) dimension of \(\Hom_{G^\prime}(\Pi_{\mid_{G'}},\pi)\): \[ m(\Pi,\pi):=\dim_{\mathbb{C}} \Hom_{G'}(\Pi_{\mid_{G^\prime}},\pi) \] Form now on, assume that \(G=\mathrm{O}(n+1,1)\) is the rank one indefinite orthogonal group and \(G'=\mathrm{O}(n,1)\). Let \(P=MAN_+\) be the Langlands decomposition of a minimal parabolic subgroup of \(G\) where \(M\simeq \mathrm{O}(n)\times \mathbb{Z}/2\mathbb{Z}\) and \(A\simeq \mathbb{R}_+\). Fix a representation \(V\) of \(\mathrm{O}(n)\), \(\delta\in\{\pm 1\}\) and \(\lambda\in\mathbb{C}\). The corresponding (unnormalized) principal series representation of \(G\) is the representation \(I_\delta(V,\lambda)\) induced from the \(P\)-representation \(V\otimes\delta\otimes \mathbb{C}_\lambda\). It is convenient to identify the representation space of \(I_\delta(V,\lambda)\) with the space of \(C^\infty\)-sections of the \(G\)-equivariant bundle \(G\times_P\mathcal{V}_{\delta,\lambda}\) associated with the \(P\)-representation \(V\otimes \delta\otimes\mathbb{C}_\lambda\). A complete description of the representations \(I_\delta(V,\lambda)\) is given in the first appendix. In particular, \(I_\delta(V,\lambda)\) is either irreducible or of length 2 composition series. In the text under review, the authors determine the multiplicities \(m(\Pi,\pi)\) for all irreducible representations \(\Pi\) of \(G=\mathrm{O}(n+1,1)\) and \(\pi\) of \(G'=\mathrm{O}(n,1)\) with trivial infinitesimal character, and for irreducible principal series representations. Moreover the authors provide a thorough study of symmetry breaking operators for general principal series, including a classification of these operators and an investigation of their distribution kernel. The particular case of spherical principal series representations, i.e when the \(M\)-representation \(V\otimes\delta\) is trivial, was considered in the first volume [loc. cit.]. The lecture notes are organized as follows. The introduction describes the problem of computing symmetry breaking operators and collects the main theorems, Chapter 2 is a review of principal series representations, Chapters 3--4 deal with the general theory of symmetry breaking operators including the special case of representations with trivial infinitesimal character, Chapter 5 discusses distribution kernel of symmetry breaking operators, Chapter 6 connects symmery breaking operators with differential operators and branching rules, Chapter 7 collects useful combinatorial formulas, Chapter 8 revisits Knapp-Stein intertwining operators, Chapter 9 provides a construction of a holomorphic family of regular symmetry breaking operators, Chapter 10 contains proofs of the main theorems, Chapters 11--12 apply the main results to conjectures by Gross and Prasad, and to periods an \((\mathfrak{g},K)\)-cohomology, Chapter 13 provides a conjecture that the main theorems hold in greater generality (i.e not only for \(G=\mathrm{O}(n+1,1)\) and \(G'=\mathrm{O}(n,1)\)), Chapter 14 is an appendix giving a classification of irreducible representations of \(\mathrm{O}(n+1,1)\) and cohomological induction, Chapter 15 is an appendix describing the restriction of representations of \(\mathrm{O}(n+1,1)\) to the connected component \(\mathrm{SO}(n+1,1)\), while Chapter 16 is an appendix dealing with translation functors for \(\mathrm{O}(n+1,1)\).