The unitary dual for complex classical Lie groups (Q583393)

The paper under review gives a complete classification of the unitary dual for the complex classical Lie groups. Recall that there are two well known ways of constructing unitary representations from lower groups: unitary induction from a parabolic subgroup and complementary series. A reasonable way to describe the unitary dual is to find a distinguished set of representations for each group such that every other unitary representation is obtained from this set by the above constructions from strictly smaller parabolic subgroups. For integral infinitesimal character such a set is provided by the special unipotent representations introduced by \textit{D. Barbasch} and \textit{D. Vogan} [Ann. Math., II. Ser. 121, 41-110 (1985; Zbl 0582.22007)]. They were shown to have many interesting properties such as the fact that their annihilator is a maximal primitive ideal. From this character formulas were obtained for them. It turns out that in general this set is not large enough to describe the entire unitary dual. The definition of special unipotent representations suggests that one should consider representations with maximal primitive ideal and infinitesimal character as small as possible. However it is not clear which of them should be unitary. This leads to the task of explicitly computing the \({\mathfrak g}\)-invariant form and ruling out the cases when it is indefinite. For the classical groups various techniques introduced here simplify the calculations significantly. A first step reduces the question of computing the signature on isolated representations (the most difficult case) to the similar question for a simpler representation but on a higher rank group. Using the ``bottom layer K-type'' idea by \textit{B. Speh} and \textit{D. Vogan} [Acta Math. 145, 227-299 (1980; Zbl 0457.22011)] one comes down to the question of computing the signature of simpler representations on a higher rank group. In particular the finite set consists of K-types which are very close to the ones with fundamental extremal weight. These computations are then performed to obtain the complete classification. Some low rank examples are given. The case \(GL_ n\) was previously dealt with by \textit{D. Vogan} [Invent. Math. 83, 449-505 (1986; Zbl 0598.22008)]. The case of regular integral infinitesimal character was done by \textit{T. Enright} [Duke Math. J. 46, 513-525 (1979; Zbl 0427.22010)].