On the irreducibility of some induced representations of real reductive Lie groups (Q1728187)

In their earlier work on the Shimura-Waldspurger correpondence for \(\mathrm{Mp}_{\mathrm{2n}}\) [Ann. Math. (2) 188, No. 3, 965--1016 (2018; Zbl 1486.11076)], the authors use the irreducibility of certain induced representations of the metaplectic group \(\mathrm{Mp}_{2n}(F)\) and of the split orthogonal group \(\mathrm{SO}_{2n+1}(F)\), when \(F\) is a local field of characteristic zero. The paper under review establishes the necessary irreducibility theorems when \(F = \mathbb{R}\). When \(F \neq \mathbb{R}\), analogous irreducibility results follow from the work of \textit{C. Mœglin} [Clay Math. Proc. 13, 333--374 (2011; Zbl 1225.22015); Adv. Math. 228, No. 2, 1068--1134 (2011; Zbl 1225.22016)] for nonarchimedean \(F\), and [\textit{C. Moeglin} and \textit{D. Renard}, Contemp. Math. 691, 203--256 (2017; Zbl 1390.22013)] for \(F = \mathbb{C}\). The result for the metaplectic group \(\mathrm{Mp}_{2n}(\mathbb{R})\) (Proposition 2.4) is the irreducibility of \[ \mathrm{Ind}_{\widetilde{P}}^{\mathrm{Mp}_{2n}(\mathbb{R})}({\tau_\phi}\otimes \chi \otimes \pi_0) \] where \begin{itemize} \item \(\widetilde{P}\) is a parabolic subgroup of \(\mathrm{Mp}_{2n}(\mathbb{R})\) given by the lift to \(\mathrm{Mp}_{2n}(\mathbb{R})\) of a parabolic subgroup of \(\mathrm{Sp}_{2n}(\mathbb{R})\) whose Levi factor is isomorphic with \(\mathrm{GL}_{k}(\mathbb{R}) \times \mathrm{Sp}_{2n_0}(\mathbb{R})\), with \(k+n_0=n\). \item \(\phi: W_\mathbb{R} \to \mathrm{GL}_k(\mathbb{C})\) is a \(k\)-dimensional representation of the Weil group \(W_\mathbb{R}\) whose irreducible summands are all nonsymplectic and almost tempered, \item \(\tau_\phi\) is the irreducible representation of \(\mathrm{GL}_k(\mathbb{R})\) associated to \(\phi\) by the Langlands correspondence, \item \(\chi\) is a genuine character of the two-fold cover of \(\mathrm{GL}_k(\mathbb{R})\), \item \(\pi_0\) is a representation of \(\mathrm{Mp}_{2n_0}(\mathbb{R})\) in the \(L\)-packet attached to a Langlands parameter \(\psi_0: W_\mathbb{R} \to \mathrm{Sp}_{2n_0}(\mathbb{C})\) whose irreducible summands are all symplectic. \end{itemize} This is first proved under a certain dominance condition on \(\phi\) and \(\psi_0\), using cohomological induction (see [\textit{A. W. Knapp} and \textit{D. A. Vogan jun.}, Cohomological induction and unitary representations. Princeton, NJ: Univ. Press (1995; Zbl 0863.22011)]) and irreducibility theorems for the principal series of the metaplectic group proved by the authors in the main paper [Ann. Math. (2) 188, No. 3, 965--1016 (2018; Zbl 1486.11076)]. The general case is deduced from the previous one using translation functors. The result for the split form of the orthogonal group \(\mathrm{SO}_{2n+1}(\mathbb{R})\) (Proposition 3.4) is the irreducibility of \[ \mathrm{Ind}_{Q}^{\mathrm{SO}_{2n+1}(\mathbb{R})}({\tau_\phi}\otimes \sigma_0) \] where \begin{itemize} \item \(Q\) is a parabolic subgroup of \(\mathrm{SO}_{2n+1}(\mathbb{R})\) with Levi component \(\mathrm{GL}_k(\mathbb{R}) \times \mathrm{SO}_{2n_0+1}(\mathbb{R})\), with \(k+n_0=n\), \item \(\phi: W_\mathbb{R} \to \mathrm{GL}_k(\mathbb{C})\) and \(\tau_\phi\) are given as in the metaplectic case above, \item \(\sigma_0\) is a representation of \(\mathrm{SO}_{2n_0+1}(\mathbb{R})\) in the \(A\)-packet attached to an Arthur parameter \(\psi_0: W_\mathbb{R} \times \mathrm{SL}_2(\mathbb{C}) \to \mathrm{Sp}_{2n_0}(\mathbb{C})\) whose irreducible summands are all symplectic. \end{itemize} An important difference with the result for the metaplectic group is that under the above hypotheses, the representation \({\tau_\phi}\otimes \sigma_0\) is not necessarily an unramified twist of a (limit of) discrete series of the Levi factor. Therefore it is the irreducibility of a nonstandard module which must be established. One strategy of proof, explained in sections 3H and 3J, is to use the Langlands classification to write \(\sigma_0\) as the image of a certain intertwining operator on standard representations, and then to use the results of \textit{B. Speh} and \textit{D. A. Vogan jun.} [Acta Math. 145, 227--299 (1980; Zbl 0457.22011)] on the irreducibility of standard representations. Another strategy, pursued in sections 3F and 3G, is to use the Kazhdan-Lusztig algorithm and coherent continuation to show that the desired irreducibility result, for a nonstandard representation of \(\mathrm{SO}_{2n+1}(\mathbb{R})\), is equivalent to that of a standard module for an endoscopic group \(\mathrm{SO}_{2k+1}(\mathbb{R}) \times \mathrm{SO}_{2n+1}(\mathbb{R})\). Once this is done, one can again appeal to the results of Speh and Vogan on standard modules.