Associated varieties and Howe's \(N\)-spectrum (Q953126)

Let \(G\) be a real semisimple Lie group, let \(N\) be a nilpotent subgroup of \(G\), and let \(\pi\) be an irreducible unitary representation of \(G\). Howe has defined a notion of \(N\)-rank of \(\pi\); it is the rank of the support of the projection valued measure on the unitary dual of \(N\) obtained by restriction of \(\pi\). The paper's main result asserts that this rank can be redefined in terms of the rank of the associated variety of the annihilator of \(\pi\) in \(\mathcal U(\mathfrak g)\). For the type I classical groups, the author gives a table of the Howe rank in such terms. In order to obtain the main result, the author proves some results about the relationship between the associated variety and the \(N\)-associated variety (i.e., the associated variety of the annihilator in \(\mathcal U(\mathfrak n)\)), as well as between the \(N\)-associated variety and Howe's \(N\)-spectrum, in the case where \(N\) is abelian. The proof is more self-contained than the one due to \textit{J.-S. Huang} and \textit{J.-S. Li} [Am. J. Math. 121, No.~3, 497--517 (1999; Zbl 0929.22009)], and does not use the stable range Howe correspondence.