\(L\)-functions and Fourier-Jacobi coefficients for the unitary group \(U(3)\) (Q1177415)

Theta series are a classical tool for the construction of modular forms. \textit{A. Weil} discovered [Acta Math. 111, 143--211 (1964; Zbl 0203.03305)] that theta series may be viewed as matrix coefficients of a certain representation, nowadays called the oscillator representation. For a dual pair \((G_1,G_2)\) of reductive groups this gives a representation of \(G_1\times G_2\) whose irreducible quotients are of the form \(\sigma\otimes\tau\) and thus it gives a correspondence \(\sigma\leftrightarrow\tau\). If \(\tau\) occurs this way, \(\tau\) is called a theta lift, it is lifted from \(G_1\) to \(G_2\). To handle automorphic \(L\)-functions Langlands conjectured that there is a partition of the set of automorphic representations into \(L\)-packets \(\Pi(H,\rho)\) parametrized by pairs \((H,\rho)\) where \(H\) is an endoscopic group of \(G\) and \(\rho\) an automorphic representation of \(H\). The question arises, how these two things fit together. In the paper under consideration the case \(G=G_2=U(3)\) the unitary group in 3 variables is treated. One looks at theta lifts from \(U(1)\) and \(U(2)\) and at \(L\)-packets from the endoscopic group \(H=U(1)\times U(2)\). The main result asserts that a discrete cuspidal representation \(\pi\) is a theta lift from \(U(1)\) iff \(\pi\in\Pi(H,\rho)\) with \(\dim\rho=1\) and \(\pi\) is a lift from \(U(2)\) iff there is any \(\rho\) with \(\pi\in\Pi(H,\rho)\). This can be summarized by saying that the set of endoscopic representations of \(U(3)\) coincides with the set of theta lifts from \(U(1)\) or \(U(2)\). For the proofs one considers Fourier-Jacobi coefficients, i.e. the decomposition of the space of an automorphic representation with respect to suitable subgroups. Use of the special situation is made in the choice of these subgroups.