Separation of representations with quadratic overgroups (Q630618)

Let \(\pi\) be a unitary irreducible representation of a Lie group \(G\) on a Hilbert space \((H,\langle\cdot,\cdot\rangle)\). Let \({\mathfrak g}\) be the Lie algebra of \(G\) and \(H^\infty\) the space of \(C^\infty\) vectors for \(\pi\). The moment set for \(\pi\) is defined by \[ I_\pi=\overline{\{ l\in{\mathfrak g}^*, \exists v\in H^\infty, v\neq 0, l(X)=\frac{1}{i}\frac{\langle\pi(X)v,v\rangle}{\|v\|^2}\}}; \] see \textit{N. J. Wildberger} [Invent. Math. 98, No.~2, 281--292 (1989; Zbl 0684.22005)] (the image of the moment map associated to the Hamiltonian action of \(G\) on the projective space of \(H^\infty\)). The range \(I_\pi\) of the moment map does not characterize the representation \(\pi\). In order to separate representations, \textit{D. Arnal} and \textit{M. Selmi} [Bull. Sci. Math. 132, No. 1, 54--69 (2008; Zbl 1138.22006)] considered an exponential Lie group \(G\) (the exponential mapping is a diffeomorphism) and {\parindent=5mm \begin{itemize}\item[1.] an overgroup \(G^+\) for \(G\); \item[2.] a non-linear mapping from \({\mathfrak g}^*\) into \(({\mathfrak g}^+)^*\), such that \(p\circ \varphi=id_{{\mathfrak g}^*}\), where \(p\) is the canonical restriction \(p: ({\mathfrak g}^+)^*\to{\mathfrak g}^*\); \item[3.] a mapping \(\Phi: \widehat{G}\to \widehat{G^+}\) such that \({\mathcal O}_{\Phi(\pi)}=\varphi({\mathcal O}_\pi)\) and \(I_{\Phi(\pi)}=I_{\Phi(\pi')}\) iff \(\pi\simeq \pi'\) (\(\widehat{G}\) -- the set of equivalence classes of irreducible unitary representations of \(G\), \({\mathcal O}_\pi\) - the coadjoint orbit associated to \(\pi\)). \end{itemize}} Here, the mapping \(\varphi\) is not regular. In this paper, the authors generalize the above procedure with a quadratic polynomial mapping \(\varphi\) to several classes of groups. For example, the existence of quadratic overgroups \(G^+\) and mappings \(\Phi\) is proved for a class of special exponential Lie groups \(G\) (\({\mathfrak g}\) contains an abelian ideal with codimension the half of the dimension of a generic coadjoint orbit) and a set of 6-dimensional connected, simply connected nilpotent Lie groups \(G\).