Semiinvariant vectors associated to decompositions of monomial representations of exponential Lie groups (Q1127007)

Let \(G=\exp{\mathfrak g}\) be an exponential solvable Lie group with Lie algebra \({\mathfrak g}\), and let \(H=\exp{\mathfrak h}\) be an analytic subgroup of \(G\) with Lie algebra \({\mathfrak h}\). Let \(\Delta_G\) (resp. \(\Delta_H)\) denote the modular function of \(G\) (resp. \(H)\), and put \(\Delta^{1/2}_{H,G}(h)=(\Delta_H(h)/\Delta_G(h))^{1/2}\) for \(h\in H\). For a unitary representation \((\pi,{\mathcal H}_\pi)\) of \(G\), \({\mathcal H}^{-\infty}_\pi\) denotes the antidual space of the space \({\mathcal H}_\pi^\infty\) of \(C^\infty\)-vectors. Given a unitary character \(\chi\) of \(H\), we are interested to know the dimension of the space \[ ({\mathcal H}_\pi^{-\infty})^{H,\chi\Delta^{1/2}_{H,G}}=\bigl\{a\in{\mathcal H}_\pi^{-\infty};\pi(h)a=\chi(h)\Delta^{1/2}_{H,G}(h)a,\;\forall h\in H\bigr\}, \] in particular when \(\pi\) is irreducible. More precisely, we construct the induced representation \(\sigma=\text{ind}^G_H\chi\) of \(G\) and take its canonical central decomposition: \(\sigma=\int^\oplus_{\widehat G}m(\pi)\pi d\mu(\pi)\) with a Borel measure on the unitary dual \(\widehat G\) of \(G\). Here the multiplicity \(m(\pi)\) of \(\pi\in\widehat G\) is obtained as follows [cf. \textit{H. Fujiwara}, The orbit method in representation theory, Proc. Conf., Copenhagen 1988, Prog. Math. 82, 61-84 (1990; Zbl 0744.22010)]: Take a \(f\in{\mathfrak g}^*\) satisfying \(\chi(\exp X)=e^{\sqrt{-1}f(X)}\) \((\forall X\in{\mathfrak h})\) ad put \(\Gamma_\sigma=\{\ell\in{\mathfrak g}^*;\ell(X)=f(X),\;\forall X\in{\mathfrak h}\}=f+{\mathfrak h}^\perp\). Then \(m(\pi)\) is the number of \(H\)-orbits contained in \(\Omega(\pi)\cap\Gamma_\sigma\), where \(\Omega(\pi)\) denotes the coadjoint orbit corresponding to \(\pi\in\widehat G\). Does the equality \(m(\pi)=\dim({\mathcal H}_\pi^{-\infty})^{H,\chi\Delta^{1/2}_{H,G}}\) (a kind of Frobenius reciprocity) hold \(\mu\)-almost everywhere? It is well known [cf. \textit{R. Penney}, J. Funct. Anal. 18, 177-190 (1975; Zbl 0305.22016)] that the inequality \(m(\pi)\leq\dim({\mathcal H}_\pi^{-\infty})^{H,\chi\Delta^{1/2}_{H,G}}\) holds. In this paper the author considers \(\pi\in\widehat G\) with the following condition: There exists an ideal \({\mathfrak p}\) of \({\mathfrak g}\) such that \(\Omega(\pi)+{\mathfrak p}^\perp=\Omega(\pi)\) and \(\ell([{\mathfrak p},{\mathfrak p}])=\{0\}\) for \(\ell\in\Omega(\pi)\). Under this condition she proves: (1) If \(\Omega(\pi)\cap\Gamma_\sigma=\emptyset\), then \(({\mathcal H}_\pi^{-\infty})^{H,\chi\Delta^{1/2}_{H,G}}=\{0\}\). (2) Suppose that \(\Omega(\pi)\cap\Gamma_\sigma\neq\emptyset\) and each connected component of this intersection is a single \(H\)-orbit and that the number \(m(\pi)\) of \(H\)-orbits contained in this intersection is finite. Then \(m(\pi)\geq\dim({\mathcal H}_\pi^{-\infty})^{H,\chi\Delta^{1/2}_{H,G}}\).