Generalized Whittaker vectors for holomorphic and quaternionic representations (Q1405750)

Let \(G\) be a real reductive group and let \(P= MN\) be a parabolic subalgebra of \(G\). Let \(\psi: N\to S^1\) denote a unitary character of \(N\). Let \((\pi,H)\) be an admissible unitary representation of \(G\) and let \(H^\infty\) denote the subspace of \(C^\infty\) vectors. Let \(Wh^\infty_\psi(H)\) denote the space of all continuous linear functionals, \(\nu\), on \(H^\infty\) such that \(\nu(\pi(n)w)= \psi^{-1}(n)\nu(w)\) for all \(n\in\mathbb{N}\) and all \(w\in H^\infty\). This is called the space of generalized continuous Whittaker functionals (or vectors) on \(H^\infty\). In this paper, the author computes the dimension of \(Wh^\infty_\psi(H)\) in a number of situations listed below. (1) The maximal compact subgroup of \(G\) has a one-dimensional center and its Lie algebra is of tube type. The Harish-Chandra module of \(H\) is a lowest weight Verma module. This includes the holomorphic discrete series representations. The parabolic subgroup \(P= MN\) is a parabolic subgroup where \(N\) is abelian. (2) The group \(G\) is of quaternionic type as in [\textit{B. H. Gross} and \textit{N. R. Wallach}, J. Reine Angew. Math. 481, 73-123 (1996; Zbl 0857.22012)] not of type \(C_n\). The parabolic subgroup \(P= MN\) is a parabolic subgroup such that \(N\) is a Heisenberg group. The unitary character \(\psi\) is a non-degenerate unitary character of \(N\). The representation \(H\) is a quaternionic discrete series representation. (3) The same as in (2) except that \(H\) is a continuation of a family of quaternionic discrete series representations as in the paper by \textit{B. Gross} and \textit{N. Wallach} [op. cit.]. The Harish-Chandra module of \(H\) is of the form \(A_{\mathfrak q}(\lambda)\) where \(\lambda\) is in the weakly fair range and \({\mathfrak q}\) is a theta stable Heisenberg parabolic subalgebra. (4) The same as in (2) except that \(H\) is a degenerate principal series representation induced from a finite-dimensional \(V\) representation of \(M\). Then \(\dim Wh^\infty_\psi(H)= \dim V\). It is too lengthy to describe the precise statements on the dimensions of \(Wh^\infty_\psi(H)\) in the first three cases in this review. In short, the dimension is nonzero if and only if the character \(\psi\) satisfies certain admissible conditions. If the dimension is nonzero, then it is determined by the dimension of the lowest \(K\)-type of \(H\). Central to the proofs of the existences of the Whittaker vectors are a translation principle in [\textit{N. R. Wallach}, Adv. Stud. Pure Math. 14, 123-151, Academic Press (1988; Zbl 0714.17016)] and the convergence of the integral \(\int_N\psi(n) f(n) dn\) where \(f(n)\) is a matrix coefficient. For Case 1, M. Hashizume and H. Yamashita have studied the related problem of finding Whittaker models for holomorphic discrete series representations. For Case 4, the author has extended the result in a preprint to all degenerate principal series representations of all real reductive groups.