Cuspidal representations associated to \((\text{GL}(n),O(n))\) over finite fields and \(p\)-adic fields (Q1283665)

Let \(G\) be a finite group and \(H\) a proper subgroup. \((G,H)\) is called a Gelfand pair if \(\text{Ind}^G_H1\) is a multiplicity free representation. Gelfand pairs of real or \(p\)-adic groups are similarly defined. If \({\mathbf G}\) is a reductive group defined over a finite field or a local field \(F\) and \({\mathbf H}\) an algebraic subgroup then \({\mathbf h}\) is called spherical [cf. \textit{D. Luna} and \textit{T. Vust}, Comment. Math. Helv. 58, 186--245 (1983; Zbl 0545.14010)] if there exists a Borel subgroup \({\mathbf B}\) such that \({\mathbf B}{\mathbf H}\) is open in \({\mathbf G}\). If \({\mathbf H}\) is a spherical subgroup then the groups of rational points, \(G= {\mathbf G}(F)\) and \(H={\mathbf H}(F)\), are expected to form a Gelfand pair or almost a Gelfand pair. Here ``almost'' is not a precise term, but taken to mean that ``most'' representations occur with multiplicity one or with small multiplicity in \(\text{Ind}^G_H1\). In the paper under review, the authors study the spherical pair \((\text{GL}(n),O(n))\), where \(O(n)\) is an orthogonal group associated to a symmetric form for the case, where \(F\) is a finite field or a \(p\)-adic field. They also study the related spherical pair \((\text{GL}(n),\text{GO}(n))\), where \(\text{GO}(n)\) is the similitude group. In the case, where \(F= {\mathbb R}\) and \(O(n)\) is the standard compact group, then \((\text{GL}(n,{\mathbb R}),O(n,{\mathbb R}))\) is a Gelfand pair [cf. \textit{I. G. Macdonald}, Symmetric functions and Hall polynomials, 2nd ed. Oxford (1995; Zbl 0824.05059)]. However, when \(F\) is a finite field or a \(p\)-adic field this is no longer true. The authors study the multiplicities of cuspidal representations in \(\text{Ind}^G_H1\) for the finite field case and of ``level zero'' supercuspidal representations for the \(p\)-adic field case. Level zero supercuspidals are those supercuspidals which are obtained from cuspidal representations on the finite group via induction from a compact open mod center subgroup. In the finite field case, this multiplicity is the same as the dimension of the space of \(H\) fixed vectors in a supercuspidal representation. In the finite field case, the authors use a formula of \textit{G. Lusztig} [The Grothendieck Festschrift, Vol. III, Prog. Math. 88, 57--81 (1990; Zbl 0773.20012)] to give a simple criterion for a cuspidal representation of \(\text{GL}(n)\) to have an \(O(n)\) fixed vector. They show that the dimension of the space of \(O(n)\) fixed vectors in this case is at most one (giving credence to the assumption that \((\text{GL}(n),O(n))\) is ``almost'' a Gelfand pair). In the \(p\)-adic field case, the authors use a delicate analysis of a level zero supercuspidal representation and their result from the finite field case in order to give a precise formula for the multiplicity in question. They derive from their formula that for \(H= O(n)\) the multiplicity is at most two and that multiplicity two can occur, while for \(H= \text{GO}(n)\), the similitude group, the multiplicity is at most one. The results in this paper are used by the authors [Pac. J. Math. 185, No. 1, 149--162 (1998; Zbl 0921.22014)] in their study of supercuspidal representations of \(\text{GL}(n)\) which are distinguished by a unitary group.