Invariant differential operators on certain nilpotent homogeneous spaces (Q5956991)

Let \(G=\exp {\mathfrak g}\) be a connected, simply connected real nilpotent Lie group with Lie algebra \({\mathfrak g}\), \(H=\exp {\mathfrak h}\) be an analytic subgroup of \(G\) with Lie algebra \({\mathfrak h}\). Given a unitary character \(\chi= \chi_f\), \(\chi(\exp X)=e^{-\sqrt {-1}f(X)}\) \((\forall X\in {\mathfrak h})\) with \(f\in {\mathfrak g}^*\) satisfying \(f([{\mathfrak h}, {\mathfrak h}])=\{0\}\), we construct the induced representation \(\tau=\text{Ind}^G_H\chi_f\) of \(G\). In this paper, the authors study the algebra \(D_\tau (G/H)\) of \(G\)-invariant differential operators on the line bundle with basis \(G/H\) associated to these data. Let \(\widehat G\) be the unitary dual of \(G\), then we have the canonical central decomposition of \(\tau:\tau\cong\int^\oplus_{\widehat G}m (\pi)\pi d\mu(\pi)\) with multiplicity function \(m(\pi)\) and a certain positive Borel measure \(\mu\) on \(\widehat G\). We have the following alternative: either \(m(\pi)= \infty\) \(\mu\)-a.e. or \(m(\pi) <c\) \(\mu\)-a.e. for some constant \(c\) [cf. \textit{L. Corwin}, \textit{F. P. Greenleaf} and \textit{G. Grélaud}: Trans. Am. Math. Soc. 304, 549-583 (1987; Zbl 0629.22005)]. We distinguish these two situations by saying that \(\tau\) is of infinite (resp. finite) multiplicities. There are two well-known conjectures from \textit{L. Corwin} and \textit{F. P. Greenleaf} [Comm. Pure Appl. Math. 45, 681-748 (1992; Zbl 0812.43004)]. Polynomial conjecture: when \(\tau\) is of finite multiplicities, the algebra \(D_\tau (G/H)\) is isomorphic to the algebra \(\mathbb{C} [\Gamma_\tau]^H\) of \(H\)-invariant polynomial functions on \(\Gamma_\tau= f+{\mathfrak h}^\perp\subset {\mathfrak g}^*\). Commutativity conjecture: \(\tau\) is of finite multiplicities if and only if \(D_\tau (G/H)\) is commutative. The authors prove here the polynomial conjecture under the condition that there exists a subalgebra which polarizes all generic elements in \(\Gamma_\tau\), and also the commutativity conjecture when \({\mathfrak h}\) is an ideal of \({\mathfrak g}\).