Generic representations for symmetric spaces. With an appendix by Yiannis Sakellaridis. (Q2631942)

The paper under review is concerned with the so-called distinction problem in representation theory, which is an very important notion, say, in the study of (relative) Langlands program and also the study of periods in number theory. Let \(G\) be a quasi-split reductive group over either a finite or non-Archimedean local field \(k\) of characteristic not 2, and \(\theta\) be an involution of \(G\). Let \(K=G^\theta\) and \(K^1=[K^0, K^0]\), where \(K^0\) is the identity component of \(K\). The main result of the paper provides two equivalent necessary conditions for \(G\) to possess a generic representation \(\pi\) distinguished by \(K^1(k)\), i.e., \(\text{Hom}_{K^1(k)}(\pi, \mathbb{C})\neq 0\). The first condition is easy to state, which says that the symmetric space \((G,\theta)\) is quasi-split over \(\bar{k}\), i.e., there is a Borel subgroup \(B\) of \(G(\overline{k})\) such that \(B\cap \theta(B)\) is a maximal torus of \(G\). The second condition, which is equivalent to the first one, is based on the \textit{universality} of reductive groups with involutions over general algebraically closed fields of characteristic not 2. In brief, the condition requires that if \((G,\theta)\) over \(\bar{k}\) is associated to \((G',\theta')\) over \(\mathbb{C}\) in a certain sense, then the real reductive group \(G'_{\theta'}(\mathbb{R})\) is quasi-split over \(\mathbb{R}\). The author applies his criterion to many familiar examples of symmetric spaces \((G,\theta)\) which have no generic representations distinguished by \(K\). These results were known usually only for the context of general linear groups and proved by different methods. To name a few examples, there are such symmetric spaces \((\text{GL}_{n+m}, \text{GL}_n\times \text{GL}_m)\) (with \(|m-n|>1\)), \((\text{GL}_{2n}, \text{Sp}_{2n})\) and their twisted analogs in the setting of unitary groups, and also other classical groups such as \((\text{Sp}_{2m+2n}, \text{Sp}_{2m}\times \text{Sp}_{2n})\). The appendix by Y. Sakellaridis applies a theorem of \textit{F. Knop} [Invent. Math. 99, No. 1, 1--23 (1990; Zbl 0726.20031)] to give another proof of a sufficient condition for \((G,\theta)\) to be quasi-split (which is Theorem 2 of the paper), and it applies to more general \(G\)-varieties in characteristic zero.