Relative functoriality and functional equations via trace formulas (Q2000812)

In his paper ``Beyond endoscopy'' [in: Contributions to automorphic forms, geometry, and number theory. Papers from the conference in honor of Joseph Shalika on the occasion of his 60th birthday, Johns Hopkins University, Baltimore, MD, USA, May 14--17, 2002. Baltimore, MD: Johns Hopkins University Press. 611--697 (2004; Zbl 1078.11033)], \textit{R. P. Langlands} proposed to study functoriality via poles of \(L\)-functions comparing new forms of trace formulas. The main purpose of the paper under review is to present a uniform framework for carrying out this study in the broadened setting of spherical varieties (of which reductive groups are special cases). The article is divided into an introduction and two parts. The introduction is mainly concerned with the presentation of the general formalism. Let \(X\) a spherical variety under the action of a reductive group \(G\) defined over a global field \(k\). Firstly, the author defines, at each place \(v\) of \(k\), a Schwartz space \(\mathcal S(k_{v})\) of measures over the quotient space \(X\times X/\!\!/G\) (\(G\) acting diagonaly on \(X\times X\)) and, at (almost) all non-archimedean place, endows it with a basic vector (typically the characteristic function of the integer points of \(X\) times a \(G\)-invariant measure). Then making up the adelic space \(\mathcal S(\mathbb{A}_{k})\), he defines the relative trace formula as the spectral and geometric decomposition of a distribution on \(\mathcal S(\mathbb A_{k})\otimes \mathcal S(\mathbb A_{k})\), the former in terms of relative characters associated to automorphic representations of \(G(k)\) and the latter, rougly, in terms of \(G(k)\)-orbits on \(X\times X(k)\). Finally, let \(Y\) an other spherical variety under a reductive group \(G'\) defined over \(k\), \(\check{G}_{X}\) and \(\check{G}_{Y}\) the \(L\)-groups of \(X\) and \(Y\) respectively as defined by \textit{F. Knop} and \textit{B. Schalke} [Trans. Mosc. Math. Soc. 2017, 187--216 (2017; Zbl 1433.14046)]. Given a morphism of \(L\)-groups \(\check{G}_{Y}\rightarrow \check{G}_{X}\), the comparison of the trace formulas for \(X\) and \(Y\) requires a linear operator \(\mathcal T\) between their Schwartz spaces satisfying three conditions, two about his local components (among which a fundamental lemma) plus a global ``commutativity condition'' with trace formula distributions (unstated in full generality). To comprehend the nature of the transfert operator \(\mathcal T\) by way of examples is the goal of the paper. The first part deals with technical considerations with the examples in mind: equivariant Fourier transforms, Kutnetsov trace formula and test measures or densities. It concludes with the study of two general local examples, the lifting from a Cartan subgroup and the lifting from the boundary degeneration.The second part deals with several special local examples, most of which have appeared in the literature. All these examples are presented in the previous setup and in each one, the transfer operator is defined as generalized Fourier transform (times innocuous scalar factors). Moreover, these examples make known an important property of the transfer operators, namely they are some deformations of the corresponding transfer operators for the boundary degenerations. This paper is dense, without proofs of new results (which appear in other papers) and ``skims through'' its subject refering to numerous previous works.