Transfer operators and Hankel transforms between relative trace formulas. II: Rankin-Selberg theory (Q2064474)

In a broad sense, the idea of ``Beyond endoscopy'' in Langlands' program could be understood as comparison of trace formulas that encompasses the (stable) Arthur-Selberg trace formula as well as the relative trace formulas such as Kuznetsov trace formula: different trace formulas are to be compared via certain transfer operators. The work under review studies an enlightening example using Rankin-Selberg theory, the protagonists being the symmetric square \(L\)-function, Kuznetsov trace formula for \(\mathrm{GL}_2\) and Hankel transforms. This yields a trace formula-theoretic approach to the functional equation for \(L(\mathrm{Sym}^2)\) for \(\mathrm{GL}_2\). Specifically, let \((V, \omega)\) be a two-dimensional symplectic space. The Rankin-Selberg variety \(\bar{X}\) is defined as \(V \times \mathrm{SL}(V)\), on which \(\tilde{G} = \mathbb{G}_m \times \mathrm{SL}(V)^2\) acts by \[ (v, g) (a, g_1, g_2) = (avg_1, g_1^{-1} g g_2); \] it can also be identified with the homogeneous vector bundle \(V \overset{\mathrm{SL}(V)}{\times} \mathrm{SL}(V)^2\) over \(\mathrm{SL}(V)\) where \(\mathrm{SL}(V)\) embeds diagonally. This is an affine spherical \(\tilde{G}\)-variety, and the complement \(X\) of the zero section of the bundle \(\bar{X}\) is the open \(\tilde{G}\)-orbit. By fixing a symplectic basis for \(V\), we have \[ X \simeq \text{diag}\bigl( \begin{smallmatrix} 1 & * \\ & 1 \end{smallmatrix} \bigr) \big\backslash \mathrm{SL}(2)^2. \] The strategy is ultimately based on a space of non-standard test measures \(\mathcal{S}(\bar{X} / \mathrm{SL}_2)^\circ\) over local fields and its endomorphism \(\mathcal{H}_X^\circ\). In Section 8, one defines \(\mathcal{S}(\bar{X} / \mathrm{SL}_2)^\circ\) over a local field. It lies between \(\mathcal{S}(X / \mathrm{SL}_2)\) (the naive Schwartz space) and \(\mathcal{S}(\bar{X} / \mathrm{SL}_2)\) (which produces the \(L\)-values \(L(\mathrm{Sym}^2)\) and \(L(\mathrm{triv})\) in relative trace formula). It is designed to yield the \(L\)-value \(L(\mathrm{Sym}^2)\) only. These spaces are described in terms of Mellin transforms, and the relevant tools for this are given in Section 7. Fix an additive character \(\psi\). The endomorphism \(\mathcal{H}_X^\circ\) is a factor of the fiberwise Fourier transform on \(\bar{X}\): it produces the symmetric square \(\gamma\)-factor when applied to relative characters (Theorem 8.3.5). To describe such Fourier transforms, it is best to consider spaces of half-densities \(\mathcal{D}(\cdots)\) instead of measures (i.e.,\ \(1\)-densities), although it is routine to pass between them. In Section 9, these spaces are used to construct a space \(\mathcal{S}^-_{L(\mathrm{Sym}^2, 1)}(N, \psi \backslash G / N, \psi)\) of test measures for the Kuznetsov trace formula for \(G = \mathbb{G}_m \times \mathrm{SL}_2\). The basic tool is the unfolding map \(\mathcal{U}\) from \(\bar{X}\) to \(N, \psi \backslash \tilde{G}\), a sophisticated paraphrase of the unfolding technique in Rankin-Selberg theory. One also obtains a Hankel transform \[ \mathcal{H}_{\mathrm{Sym}^2}: \mathcal{D}^-_{L(\mathrm{Sym}^2, 1/2)}(N, \psi \backslash G / N, \psi) \xrightarrow{\sim} \mathcal{D}^-_{L(\mathrm{Sym}^{2, \vee}, 1/2)}(N, \psi \backslash G / N, \psi). \] Its effect on the relative character of \(\pi\) is the multiplication by \(\gamma(\pi, \mathrm{Sym}^2, \frac{1}{2}, \psi)\), when the irreducible generic representation \(\pi\) varies in a family \(\pi \otimes |\cdot|^s\); see Theorem 9.0.1. Following the general philosophy of comparison of trace formula, all these spaces contain a ``basic vector'' in the unramified setting, preserved by Hankel transforms, and the relation between unramified Hecke action and Hankel transforms has a transparent description. Furthermore, \(\mathcal{H}_{\mathrm{Sym}^2}\) is compatible with boundary degeneration (Theorem 9.6.1). Such phenomena also pertain to the Hankel transform \(\mathcal{H}_{\mathrm{Std}}\) for the standard \(L\)-factor considered by Jacquet, and suggest that Hankel transform is a deformation of its abelian analogue. In Section 10, one constructs a transfer operator \(\mathcal{T}_T\) from \(\mathcal{S}^-_{L(\mathrm{Sym}^2, 1)}(N, \psi \backslash G /N, \psi)\) to \(\mathcal{S}(T)\) with many properties (Theorem 10.1.1), where \(T\) is the torus associated with some quadratic (possibly split) extension of the base field \(F\). It is constructed by first passing to \(\kappa\)-orbital integrals on \(\mathrm{SL}_2\) (Theorem 10.3.2) and then to \(T\) by Labesse-Langlands theory, suggesting that beyond endoscopy is not entirely disjoint from endoscopy. As an application, one obtains a simple expression for the stable characters on \(\mathrm{SL}_2\) lifted from \(T\), known as Gelfand-Graev-Piatetski-Shapiro formula (see 10.29). The relation to Venkatesh's thesis is discussed at length in 10.1.