Local unitary periods and relative discrete series (Q1799060)

Let \(F\) be a \(p\)-adic field, \(p\neq2\), and \(E\) be a quadratic extension of \(F\). Let \(G=\mathrm{GL}_{2n}(E)\), \(Z_G\) be the center of \(G\), \(H\subset G\) be the \(F\)-points of a quasi-split unitary group, and \(L^2(Z_GH\backslash G)\) denote the space of complex-valued functions on the \(p\)-adic symmetric space \(H\backslash G\) that are square integrable modulo \(Z_G\) and transform under \(Z_G\) by a unitary character. This paper constructs relative discrete series representations of \(H\backslash G\), i.e.\ subrepresentations of \(G\) that intertwine with an irreducible subrepresentation of the right regular representation of \(G\) on \(L^2(Z_GH\backslash G)\). By work of \textit{S.-I. Kato} and \textit{K. Takano} [J. Funct. Anal. 258, No. 5, 1427--1451 (2010; Zbl 1185.22014)], the \(H\)-distinguished discrete series representations of \(G\) satisfy this condition. This paper constructs relative discrete series that do not occur in the discrete spectrum for \(G\) itself. Let \(\sigma\) generate \(\text{Gal}(E/F)\) and \(\tau\) be a discrete series representation of \(\mathrm{GL}_n(E)\) which is not \(\sigma\)-invariant. Then the author shows that inducing \(\tau\,\otimes \,^\sigma\!\tau\) to \(G\) gives a relative discrete series representation \(\pi\) that does not occur in the discrete series for \(G\). Combining this with work of \textit{R. Beuzart-Plessis} [``Plancherel formula for \(\mathrm{GL}_n(F)\backslash\mathrm{GL}_n(E)\) and applications to the Ichino-Ikeda and formal degree conjectures for unitary groups'', Preprint, \url{arXiv:1812.00047}], in fact this construction exhausts all relative discrete series that lie outside \(L^2_{\text{disc}}(G)\). The proof of the main result is modeled on the author's prior work on the \(p\)-adic symmetric spaces \((\mathrm{GL}_n(F)\times \mathrm{GL}_n(F))\backslash \mathrm{GL}_{2n}(F)\) and \(\mathrm{GL}_n(F)\backslash \mathrm{GL}_n(E)\) [``Relative discrete series representations for two quotients of \(p\)-adic \(\mathrm{GL}_n\)'', Canad. J. Math. 70, 1339--1372 (2018; \url{doi:10.4153/CJM-2017-047-7})]. Key steps include verifying the relative version of Casselman's criterion for square-integrability, due to \textit{S.-I. Kato} and \textit{K. Takano} [Int. Math. Res. Not. 2008, Article ID rnn028, 40 p. (2008; Zbl 1197.22007)] and \textit{N. Lagier} [J. Funct. Anal. 254, No. 4, 1088--1145 (2008; Zbl 1194.22010)], observing that the representation \(\pi\) is \(H\)-distinguished by a result of \textit{B. Feigon} et al. [Publ. Math., Inst. Hautes Étud. Sci. 115, 185--323 (2012; Zbl 1329.11053)] (these authors also study the analogous global periods), and deducing from \textit{A. V. Zelevinsky}'s classification [Ann. Sci. Éc. Norm. Supér. (4) 13, 165--210 (1980; Zbl 0441.22014)] that \(\pi\) does not occur in the discrete series for \(G\). This paper complements the study of harmonic analysis on \(p\)-adic spherical varieties of \textit{Y. Sakellaridis} and \textit{A. Venkatesh} [Periods and harmonic analysis on spherical varieties. Paris: Société Mathématique de France (SMF) (2017; Zbl 1479.22016)].