Whittaker functions for the large discrete series representations of \(\mathrm{SU}(2,1)\) and related zeta integrals (Q1918526)

The authors study the smooth Whittaker functions for certain discrete series representations of the real semisimple group \(\mathrm{SU}(2,1)\). This group has three types of discrete series: holomorphic, anti-holomorphic, and neither. It is the third type which is considered here. These discrete series representations are those of Gelfand-Kirillov dimension 3, the dimension of the maximal unipotent subgroup; hence they are large in the sense of Vogan. By a theorem of Kostant, they possess a unique smooth Whittaker model. In this paper the authors give explicit formulas for these Whittaker functions and for their Mellin transforms. The authors first describe the Harish-Chandra parameters associated to the large discrete series representations, and compute their \(K\)-types. They compute the Whittaker functions by first identifying the relevant intertwining space with the solution space to a certain partial differential operator. In the case of minimal \(K\)-type, they further reduce this to a system of differential-difference equations, which they solve. This gives an explicit formula for (the radial part) of the minimal \(K\)-type Whittaker function in terms of classical Bessel functions. Next the authors give recurrence relations for the weight vectors in all \(K\)-type subspaces of the Whittaker model. This step uses the observation that the restriction of the Whittaker model of a given large discrete series representation of \(\mathrm{SU}(2,1)\) to \(K\) is multiplicity free. Using these formulas, the authors compute the Mellin transforms of the \(K\)-finite Whittaker functions in terms of polynomials and Gamma factors. The authors apply their results above to carry out the evaluation of certain local zeta integrals in the archimedean case. Namely, \textit{S. Gelbart} and \textit{I. Piatetski-Shapiro} [in: Lie group representations II, Lect. Notes Math. 1041, Springer-Verlag, 141--184 (1984; Zbl 0531.10034)] attached certain zeta-integrals to a generic representation of the quasi-split unitary group in 3 variables, defined over a number field. These integrals factorize, and they evaluated them in the unramified local case as degree 3 local \(L\)-factors, corresponding to a degree 3 representation of the relevant \(L\)-group. The authors here evaluate these integrals when the base field is real and the representation belongs to the large discrete series of \(\mathrm{U}(2,1)\). The greatest common divisor of the integrals is shown to be a product of 3 Gamma factors. They note that their method also applies to the case of the principal series. These last results may be put in the following context. There are many Rankin-Selberg integrals which establish the meromorphicity and functional equation for particular partial automorphic \(L\)-functions. However, it is frequently the case that the archimedean local integrals are not evaluable by standard methods. The authors give an instance where a detailed study of the Whittaker functions gives enough information to complete the evaluation. Work in a similar direction for the general linear group has been carried out by \textit{E. Stade} [Duke Math. J. 60, No. 2, 313--362 (1990; Zbl 0731.11027) and Manuscr. Math. 87, No. 4, 511--526 (1995; Zbl 0842.11049)], and for the \(4\times 4\) real symplectic group by \textit{S. Niwa} [Proc. Japan Acad., Ser. A 71, No. 8, 189-191 (1995)].