Epsilon dichotomy for linear models (Q2656320)

The paper under review studies the existence of linear models for smooth representations of the inner forms of general linear groups over non-Archimedean local fields. It confirms a conjecture of \textit{D. Prasad} and \textit{R. Takloo-Bighash} [J. Reine Angew. Math. 655, 189--243 (2011; Zbl 1228.11070)] in many cases, which characterizes the existence of linear models using the local root numbers. Let \(E/F\) be a quadratic extension of non-Archimedean local fields of characteristic zero and \(\eta\) the associated quadratic character of \(F^{\times}\). Let \(A\) be a central simple algebra over \(F\) of dimension \(4n^2\) with a fixed embedding of \(E\) in it. Let \(B\) be the centralizer of \(E\) in \(A\), then \(B\) is a central simple algebra over \(E\) of dimension \(n^2\). Let \(G = A^{\times}\) and \(H = B^{\times}\). The author proves that if an irreducible admissible representation \(\pi\) of \(G\) is \(H\)-distinguished, i.e., \(\mathrm{Hom}_{H}(\pi, \mathbb{C}) \neq 0\), then its Jacquet-Langlands transfer \(\pi'\) to \(G' := \mathrm{GL}_{2n}(F)\) is self-dual of symplectic type and \( \epsilon(\pi')\epsilon(\pi' \otimes \eta)\eta(-1)^n = (-1)^r, \) where \(\epsilon(\pi'), \epsilon(\pi'\otimes \eta)\) are the local root numbers and \(r\) is the split rank of \(G\). The author also proves the opposite direction when \(\pi'\) is supercuspidal or \(\pi\) is superecuspidal and \(G = \mathrm{GL}_{n}(D)\), where \(D\) is a quaternion algebra over \(F\) (split or not). The strategy of the proof is to relate \(\mathrm{Hom}_{H}(\pi, \mathbb{C})\) with \(\mathrm{Hom}_{H'}(\pi', \mathbb{C})\) and \(\mathrm{Hom}_{H'}(\pi' \otimes \eta, \mathbb{C})\) for \(H' = \mathrm{GL}_{n}(F) \times \mathrm{GL}_{n}(F)\) via the relative trace formula of \textit{J. Guo} [Can. J. Math. 48, No. 1, 105--142 (1996; Zbl 0852.11026)]. For generic \(\pi'\), it is proven in [\textit{E. Lapid} and \textit{Z. Mao}, Am. J. Math. 139, No. 1, 1--55 (2017; Zbl 1367.11053)] that \(\pi'\) is \(H'\)-distinguished, i.e., \(\mathrm{Hom}_{H'}(\pi', \mathbb{C}) \neq 0\) if and only if \(\pi'\) is self-dual of symplectic type. In order to obtain the condition on the local root numbers, the author introduces the involution method, which explores the behaviors of the relative characters of \(G'\) and the smooth transfer between \(C^{\infty}_c(G)\) and \(C^{\infty}_c(G')\) under the involution \(f' \mapsto f'^{w}\) on \(C^{\infty}_c(G')\) given by \( f'^{w}(g) = f'(wgw), \) where \(w\) is the longest Weyl group element in \(G'\).