Supercuspidal representations of \(\mathrm{GL}_n(F)\) distinguished by an orthogonal involution (Q2167488)

Let \(F\) be a nonarchimedean local field of odd residue characteristic, let \(G=\mathrm{GL}_n(F)\) and let \(H\) be an orthogonal subgroup of \(G\). Let \(\pi\) be a complex irreducible supercuspidal representation of \(G\). The paper under view studies the distinction of \(\pi\) by \(H\), that is, the space \(\mathrm{Hom}_H(\pi, 1)\). This problem fits into the framework of ``relative Langlands program''. The main results characterize the nonvanishing and dimension of this space in terms of the simple type theory of Bushnell-Kutzko and the endo-class theory of Bushnell-Henniart. We refer to the paper for the full statements. As a particular case, if \(H\) is split, then \(\pi\) is distinguished by \(H\) if and only if \(\omega_\pi(-1)=1\) where \(\omega_\pi\) is the central character of \(\pi\). In this case \(\mathrm{Hom}_H(\pi, 1)\) may have dimension 1, 2 or 3. The author also proves that \(\pi\) is distinguished by a certain orthogonal subgroup if and only if \(\omega_\pi(-1)=1\), in which case it holds that \[ \dim_{\mathbb{C}}\mathrm{Hom}_G(\pi, \mathcal{C}^\infty(\mathcal{S}))=4, \] where \(\mathcal{S}\) is the set of symmetric matrices in \(G\).