Applications of the Kuznetsov formula on \(\mathrm{GL}(3)\). II: The level aspect (Q2405956)

The classical (Bruggeman)-Kuznetsov trace formula for congruence subgroups of \(\Gamma =\mathrm{SL}_2( {\mathbb Z})\) is a well-known and very powerful analytic tool in the theory of \(\mathrm{GL}(2)\)-automorphic forms. It relates spectral data to arithmetic data (Kloosterman sums). Among the applications of the Kuznetsov formula can be found subconvexity estimates for moments of \(L\)-functions and large sieve inequalities, as well as applications to counting problems and density theorems for exceptional eigenvalues. Generalizations of the Kuznetsov formula to higher Lie groups are desirable and known in some great generality, but it is difficult to derive them in an explicit and analytically useful form. During the last years, the study of explicit Kuznetsov type formulas for \(\mathrm{GL}(3)\)-automorphic forms is a central topic of study in the field. In the paper under review, the authors continue the study of Kuznetsov type formula for arithmetic subgroups of \(\mathrm{GL}_3(\mathbb{R})\) from [the first author, Invent. Math. 194, No. 3, 673--729 (2013; Zbl 1292.11064)]. Let \(N\) be a large prime level and \(\Gamma_0(N)\) the subgroup of \(\mathrm{SL}_3(\mathbb Z)\) with bottom row congruent to \((0,0,*)\) modulo \(N\). Let also \(\mathbb{H}^3\) denote the generalized upper half plane. For an \(\mathrm{SO}(3)\)-invariant subspace of a spherical cuspidal automorphic representation \(\pi \subset L_{\mathrm{cusp}}^{2} (\Gamma_0(N) \backslash \mathbb{H}^3)\) we let \(\mu_{\pi}\) denote the spectral parameter of \(\pi\), normalized such that \(\mathrm{Re}(\mu_{\pi})=0\) if the Ramanujan conjecture is true. We also fix a compact subset \(\Omega\) of the complexified dual of the Lie algebra of the maximal torus in \(\mathrm{PGL}_3 (\mathbb{R})\), assuming that the number of \(\mu_{\pi} \in \Omega\) is of size \(\asymp_{\Omega} N^{2+ o(1)}\). In their main result, they state and prove a Kuznetsov type formula for \(\Gamma_0(N)\) (Theorem 6, Section 2). The proof of this result follows the classical path of computing the inner product of two level \(N\) Poincaré series in two different ways, by unfolding and by spectral decomposition, and comparing the two expressions. To make their trace formula useful and attractive, the authors deduce explicit applications for the following three problems. First, they prove a mean-Lindelöf estimate in the level aspect for the sixth moment of the \(L\)-function of \(\pi\) twisted with a fixed primitive character. That is, for \(N\) large prime level and \(\chi\) a primitive character mod \(p\) of order \(>2\) they prove the estimate \[ \sum_{\overset{\pi \subset L_{\mathrm{cusp}}^{2} (\Gamma_0(N) \backslash \mathbb{H}^3)}{\mu_{\pi} \in \Omega}} |L(1/2, \pi \times \chi)|^6 \ll_{p, \Omega, \varepsilon} N^{2+\varepsilon}. \] Secondly, they prove a large sieve inequality for the Fourier coefficients of \(\pi\), which for the unramified Hecke eigenvalues \(\lambda_{\pi}(n)\) with \((n, N)=1\) gives \[ \begin{aligned} \sum_{\overset{\pi \subset L_{\mathrm{cusp}}^{2} (\Gamma_0(N) \backslash \mathbb{H}^3)}{ \mu_{\pi} \in \Omega} } \left| \sum_{\overset{X \leq n \leq 2X}{(n, N) =1 } } \lambda_{\pi}(n) a(n) \right|^2 \ll_{\Omega, \varepsilon} ( NX)^{\varepsilon} (N^2 + X^2 N^{1/2} ) \| a \|^2.\end{aligned} \] For \(\mathrm{SL}_3( {\mathbb Z})\), this sieve formula is optimal for \(X \leq N^{3/4}\) and improves on a result of \textit{A. Venkatesh} [Adv. Math. 200, No. 2, 336--356 (2006; Zbl 1139.11026)] which was optimal in the range \(X \leq N^{1/4}\). This improvement can be understood as a sign of the power of the Kuznetsov formula in explicit applications. The third application presented in the work under review is on the Ramanujan conjecture, both at the Archimedean place and at finite places. For the Archimedean place, their density theorem for exceptional eigenvalues of \(\Gamma_0(N) \backslash \mathbb{H}^3\) reads: \[ \begin{aligned} \sum_{\overset{\pi \subset L_{\mathrm{cusp}}^{2} (\Gamma_0(N) \backslash \mathbb{H}^3)}{ \mu_{\pi} \in \Omega} } N^{4 \| \mathrm{Re}(\mu_{\pi})\|} \ll_{\Omega, \varepsilon} N^{2+\varepsilon}. \end{aligned} \] Notice that conjecturally \(\| \mathrm{Re}(\mu_{\pi})\| = 0\), whereas the best known bound \(\| \mathrm{Re}(\mu_{\pi})\| \leq 5/14\) is due to Kim and Sarnak.