Entrelacement des restrictions des représentations unitaires des groupes de Lie nilpotents. (Intertwining of restriction of unitary representations of nilpotent Lie groups) (Q5928782)

Let \(G=\exp {\mathfrak g}\) be a connected, simply connected nilpotent Lie group with Lie algebra \({\mathfrak g},H= \exp{\mathfrak h}\) its analytic subgroup with Lie algebra \({\mathfrak h}\). We denote by \(\widehat G\) the unitary dual of \(G\) and consider an irreducible unitary representation \(\pi\) of \(G\), namely \(\pi\in\widehat G\) under the identification of each irreducible unitary representation with its equivalence class. Now the restriction \(\pi|_H\) of \(\pi\) onto \(H\) is decomposed into irreducibles as follows. Let \(\Omega(\pi)\) be the coadjoint orbit of \(G\) corresponding to \(\pi\in \widehat G\) and \(\mu\) be a finite measure on \(\Omega(\pi)\) equivalent to a \(G\)-invariant measure. Composing two mappings, the projection \(p:{\mathfrak g}^* \to{\mathfrak h}^*\) and the Kirillov mapping \(\theta: {\mathfrak h}^* \to\widehat H\), we take the image \(\nu\) of \(\mu\), i.e. \(\nu=(\theta \circ p)_*(\mu)\) on the unitary dual \(\widehat H\) of \(H\). For \(\sigma\in \widehat H\), we denote by \(\omega(\sigma)\) the corresponding coadjoint orbit of \(H\) and by \(m(\sigma)\) the number of \(H\)-orbits contained in \(\Omega (\pi) \cap p^{-1} (\omega(\sigma))\). Then it is well known [\textit{L. Corwin} and \textit{F. P. Greenleaf}, Pac. J. Math. 135, 233-267 (1988; Zbl 0628.22007)] that \(\pi|_H \simeq\int^\oplus_{\widehat H}m(\sigma) \sigma d\nu(\sigma)\). This paper begins by making concrete this decomposition through the determination of a base space of decomposition. Then, using this result the authors give an explicit intertwining operator between the restriction \(\pi|_H\) and its concrete decomposition into the irreducibles just obtained. The whole work is pursued in the framework of the orbit method [cf. \textit{A. A. Kirillov}, Usp. Mat. Nauk 17, 57-110 (1962; Zbl 0106.25001); \textit{L. Corwin} and \textit{F. P. Greenleaf}, Representations of nilpotent Lie groups and their applications, Part I, Cambridge/New York (1990; Zbl 0704.22007)].