On Fourier coefficients of certain residual representations of symplectic groups (Q5962785)

This article is a natural continuation of a series of papers by the authors [Int. Math. Res. Not. 2013, No. 17, 4029--4071 (2013; Zbl 1359.11039); `` Arthur parameters and Fourier coefficients for automorphic forms on symplectic groups'', Preprint, \url{arXiv:1309.6239}; J. Number Theory 146, 343--389 (2015; Zbl 1386.11072)], where they study the Fourier coefficients for residual representations on certain classical groups. In this particular case, the authors want to extend the theory of automorphic descent of \textit{D. Ginzburg} et al. [The descent map from automorphic representations of \(\text{GL}(n)\) to classical groups. Hackensack, NJ: World Scientific (2011; Zbl 1233.11056)] for certain residual representations of \(\mathrm{Sp}_{2n}\). In order to extend this descent theory, the authors use the full strength of Arthur's parameters and Arthur global packets. Let us remind briefly that \textit{J. Arthur} provided a very nice description of the discrete spectrum for classical groups in terms of parameters [The endoscopic classification of representations. Orthogonal and symplectic groups. Providence, RI: American Mathematical Society (AMS) (2013; Zbl 1310.22014)]. Consider the symplectic group defined by the symplectic form \[ \begin{pmatrix} 0 & \alpha_n \\ -\alpha_n & 0 \end{pmatrix}, \] where \(\alpha_n\) is a \(n\times n\) matrix with 1 on the second main diagonal and 0 elsewhere. Denote this group by \(\mathrm{Sp}_{2n}\). Consider the maximal torus \(T\) consisting of the elements \[ \begin{pmatrix} t_1 & & & & & \\ & \ddots & & & & \\ & & t_n & & &\\& & & t_n^{-1} & & \\ & & & & \ddots & \\ & & & & & t_1^{-1} \end{pmatrix} \] and let \(U\) be all upper unipotent matrices in \(\mathrm{Sp}_{2n}\). We fix the Borel group \(B\) given by \(B= TU\). Let \(F\) be a number field, \(\mathbb A_F\) be the ring of adeles of \(F\), and \(\widetilde{\Psi}_2(\mathrm{Sp}_{2n})\) be the set of square integrable global Arthur parameters. The authors define a certain Fourier coefficient as in [Zbl 1386.11072], namely, for any symplectic partition \(\mathcal P\) of \(2n\), one constructs a \(F\)-unipotent subgroup \(V_{\mathcal P,2}\). With this construction it is possible to define a character \(\psi_{\mathcal P}\) of \(V_{\mathcal P,2} (\mathbb A_F)\) which is trivial on \(V_{\mathcal P,2} (F)\). Then, the \(\psi_{\mathcal P}\)-Fourier coefficient of an automorphic form \(\varphi\) on the space \(\mathrm{Sp}_{2n}(\mathbb A_F)\) is defined by \[ \phi^{\psi_{\mathcal P}} (g):= \int\limits_{V_{\mathcal P,2} (F) \backslash V_{\mathcal P,2} (\mathbb A_F)} \varphi(ug)\psi_{\mathcal P}(u)^{-1} du. \] For any irreducible automorphic representation \(\pi\) of \(\mathrm{Sp}_{2n}(\mathbb A_F)\), the authors define a set \(\mathcal P(\pi)\) of all partitions \(\mathcal P\) such that \(\pi\) possesses a nonzero \(\psi_{\mathcal P}\)-Fourier coefficient, but for any \(\mathcal P' > \mathcal P\) it does not have any nonzero Fourier coefficient attached to \(\mathcal P'\), with the usual order of partitions. For any global Arthur parameter \(\psi \in \widetilde{\Psi}_2(\mathrm{Sp}_{2n})\), the first author in [Contemp. Math. 614, 179--242 (2014; Zbl 1315.11037)] made several conjectures: Conjecture. Let \(\psi \in \widetilde{\Psi}_2(\mathrm{Sp}_{2n})\) and \(\widetilde{\Pi}_\psi(\varepsilon_\psi)\) be the authomorphic \(L^2\) packet attached to \(\psi\). Then the following assertions hold{\parindent= 6mm \begin{itemize}\item[(1)] Any symplectic partition \(\mathcal P\) of \(2n\) such that \(\mathcal P > \eta_{\mathfrak{g}^\vee, \mathfrak{g}}(\mathcal P(\psi))\) does not belong to \(\mathcal P(\pi)\) for \(\pi \in \widetilde{\Pi}_\psi(\varepsilon_\psi)\). Here, \(\eta_{\mathfrak{g}^\vee, \mathfrak{g}}(\mathcal P(\psi))\) is the Barbasch-Vogan duality map. \item[(2)] For every \(\pi \in \widetilde{\Pi}_\psi(\varepsilon_\psi)\), every partition \(\mathcal P \in \mathcal P(\pi)\) satisfies \(\mathcal P \leq \eta_{\mathfrak{g}^\vee, \mathfrak{g}}(\mathcal P(\psi))\). \item[(3)] There is at least one element \(\pi \in \widetilde{\Pi}_\psi(\varepsilon_\psi)\) with the property that \(\eta_{\mathfrak{g}^\vee, \mathfrak{g}}(\mathcal P(\psi)) \in \mathcal P(\pi)\). \end{itemize}} In this paper, the authors study part (3) of the previous conjecture. In the generic case, the elements \(\pi \in \widetilde{\Pi}_\psi(\varepsilon_\psi)\) should be distinguished members following the Whittaker normalization given by Arthur [loc. cit.]. In the case of nongeneric global Arthur parameters, such distinguished elements could be certain cuspidal automorphic representations conjectured by \textit{C. Mœglin} [Manuscr. Math. 127, No. 4, 411--467 (2008; Zbl 1306.11041)] or certain cuspidal automorphic representations constructed via endoscopic correspondences, as it is indicated in [Jiang, Zbl 1315.11037]. These two different cases are treated separately in this paper, since the construction methods are quite different. It should be noted that such nongeneric global Arthur parameters were already determined in [Zbl 1386.11072]. There are three different cases for these nongeneric parameters. Therefore, the authors obtain three different type of results using residual Eisenstein series, some classical results in the literature like the famous classification of the residual representations for \(\mathrm{GL}_n\) obtained by \textit{C. Moeglin} and \textit{J. L. Waldspurger} [Ann. Sci. Éc. Norm. Supér. (4) 22, No. 4, 605--674 (1989; Zbl 0696.10023)], and the Fourier coefficients that they defined.