Intertwining operators of irreducible representations for exponential solvable Lie groups (Q2353623)

Let \(G\) be a real solvable exponential Lie group with Lie algebra \(\mathfrak{g}\). This means that the exponential map \(\exp: \mathfrak{g}\to G\) is a \(\mathscr{C}^\infty\)-diffeomorphism. According to the orbit method, for each irreducible unitary representation \(\pi\) of \(G\), there exist a linear form \(l\in \mathfrak{g}^*\) and a Puk-polarization \(\mathfrak p\) at \(l\) such that the representation \(\pi\) is realized as the induced representation \(\pi_{l, \mathfrak{p}} {\buildrel\mathrm{def}\over=} ~\mathrm{ind}_P^G \chi_l\) on the Hilbert space \(\mathrm{L}^2(G/P, \chi_l)\) from the unitary character \(\chi_l\) of \(P=\exp(\mathfrak{p})\) defined by \(\chi_l(\exp(X))=e^{i l(X)}\), for any \(X\in \mathfrak{p}\). The representation \(\pi_{l, \mathfrak{p}}\) depends only on the orbit of \(l\) under the coadjoint action of \(G\), and not on the choice of the Puk-polarization \(\mathfrak p\) at \(l\). This means that for any other Puk-polarization \(P'=\exp(\mathfrak p')\) at \(l\), the monomial representations \(\pi_{l, \mathfrak{p}}\) and \(\pi_{l, \mathfrak{p'}}\) are unitarily equivalent. In the paper under review, the authors are interested in the following question raised by \textit{M. Vergne} [Ann. Sci. Éc. Norm. Supér. (4) 3, 353--384 (1970; Zbl 0221.22014)]: Determine an explicit unitary intertwining operator between \(\pi_{l, \mathfrak{p}}\) and \(\pi_{l, \mathfrak{p'}}\). In [Mem. Fac. Sci., Kyushu Univ.Ser. A 36, 13--72 (1982; Zbl 0513.22006)], the second author of the present paper suggests a formal condidate to the intertwining operator, denoted by \(T_{P', P}\) and defined for any \(g\in G\) by \[ T_{P', P}\xi(g) = \oint_{P'/P'\cap P} \xi(gh) \chi_l(h) \Delta_{P', G}^{-1/2}(h)\;d\nu(h),\hfill (HF). \] The function \(T_{P', P}\xi\) verifies the covariance condition to belong to \(\mathrm{L}^2(G/P', \chi_l)\) and that the operator \(T_{P', P}\) commutes with the action of \(G\) by left translation. It remains only to study the convergence of the integral (HF). The authors of the paper under review introduce the property \((\mathrm{CP})\) as follows: Let \(\mathfrak{p}, \mathfrak{p}'\) be two Puk-polarizations at \(l\in \mathfrak{g}^*\). We say that the pair \((\mathfrak{p}, \mathfrak{p}')\) has property \((\mathrm{CP})\), provided that the integral (HF) converges absolutely on a \(G\) and \(\mathcal{U}(\mathfrak{g})\)-invariant dense subspace of the space of \(\mathscr{C}^\infty\)-vectors of \(\pi_{l, \mathfrak{p}}\). If the product \(PP'\) is closed in \(G\), which is true when \(G\) is nilpotent [\textit{G. Lion}, Lect. Notes Math. 587, 160--176 (1977; Zbl 0391.22008)] or one of the two polarizations is of Vergne type [\textit{D. Arnal} et al., Am. J. Math. 118, No. 4, 839--878 (1996; Zbl 0866.22012)], \((\mathfrak{p}, \mathfrak{p}')\) meets property \((\mathrm{CP})\). The main result is Theorem 3.1, which asserts that if the pair \((\mathfrak{p}, \mathfrak{p}')\) has property \((\mathrm{CP})\) then the operator (HF) can be extended to a unitary intertwining operator. Furthermore, the paper establishes: if \(P_i=\exp(\mathfrak{p}_i)\), \(1\leq i\leq 3\), are three Puk-polarizations at \(l\in \mathfrak g^*\) such that the pair \((\mathfrak{p}_i, \mathfrak{p}_j)\), \(1\leq i\neq j\leq 3\), meets property \((\mathrm{CP})\), then the composition formula holds: \(T_{P_1, P_3} \circ T_{P_3, P_2} \circ T_{P_2, P_1}= e^{\frac{i \pi}{4}\tau(\mathfrak{p}_3, \mathfrak{p}_2, \mathfrak{p}_1)} \mathbb{I}\), where \(\tau(\mathfrak{p}_3, \mathfrak{p}_2, \mathfrak{p}_1)\) denotes the Maslov index of the spaces \(\mathfrak{p}_i\), \(1\leq i\leq 3\).