On the convexity of the moment mapping for unitary highest weight representations (Q1891788)

Let \(\pi\) be a unitary representation of a Lie group \(G\) on a Hilbert space \(H\). A vector \(v \in H\) is said to be smooth if the orbit mapping \(G \to H\), \(g \to \pi(g)v\) is smooth. One writes \(H^\infty\) for the space of smooth vectors, and notes that \(d \pi (X)v := (d/dt)|_{t=0} \pi (\text{exp } tX)v\) defines the structure of a \(g\)-module on \(H^\infty\) which extends to a \(g_C\)-module structure via \(d\pi(X + iY) := d\pi(X) + id\pi(Y)\) because \(H^\infty\) is a complex vector space. For each non-zero vector \(v \in H^\infty\) a linear functional on \(g^*\) is defined by \[ \Psi_\pi' : H^\infty \setminus \{0\} \to g^*,\quad \Psi_\pi'(v) (X) := {1\over i} {\langle d\pi(X)v,v \rangle\over \langle v,v\rangle} , \] that is called the moment mapping associated to \(\pi\). To each continuous unitary representation of a Lie group \(G\) in a Hilbert space \(H\) the author associates a moment map from the projective space of smooth vectors \(H^\infty\) to \(g^*\). For unitary highest weight representations one obtains a characterization of those highest weights for which the closure of the image of this map is convex, i.e. equal to the convex hull of the highest weight orbit. This result generalizes the corresponding result for representations of compact groups and holomorphic discrete series representations.