Convexity and unitary representations of nilpotent Lie groups (Q1825293)

Let G be a connected, simply connected, nilpotent Lie group with Lie algebra \({\mathfrak G}\) and Lie algebra dual \({\mathfrak G}^*\). Let \(\rho\) be an irreducible, unitary representation of G in a Hilbert space H. For any \(v\in H^{\infty}\), the subspace of \(C^{\infty}\) vectors in H, define \(\psi\) (v)\(\in {\mathfrak G}^*\) by the formula \[ \psi (v)(X)=\frac{1}{i}\frac{<d\rho (X)\cdot v,v>}{<v,v>} \] for \(X\in {\mathfrak G}\). \(\psi\) is called the moment map and the closure of the set \(\{\) \(\psi\) (v)\(|\) \(v\in H^{\infty}\}\) is called the moment set, \(I_{\rho}\). The main point of this paper is to relate the moment set of \(\rho\), which is a subset of \({\mathfrak G}^*\), to the Kirillov orbit picture. Indeed, the relationship is strong; the main theorem says that \(I_{\rho}\) is the closure of the convex hull of the Kirillov orbit associated with \(\rho\). The proof involves coordinatizing the Kirillov orbits by a variation of the procedure of geometric quantization.