On the analytic structure of commutative nilmanifolds (Q282804)

Let \(G\) be a connected Lie group and \(K\) be a compact Lie subgroup in \(G\). A homogeneous space \(X = G/K\), or equivalently a Gelfand pair \((G,K)\), is called commutative if the algebra of \(G\)-invariant differential operators on \(G/K\) is commutative. The commutative space \(G/K\) is a commutative nilmanifold if there is some closed nilpotent analytic subgroup \(N\) of \(G\) which acts on \(X\) transitively. It is known, that if \(G/K\) is connected and simply connected, then \(N\) is the nilradical of \(G\), \(N\) acts simply transitively on \(G/K\) and \(G\) is the semidirect product \(G=K \cdot N\). Also it is known that such \(N\) is abelian or 2-step nilpotent. There is the Vinberg classification of maximal irreducible commutative nilmanifolds. With three exceptions the commutative nilmanifolds in this classification have the property that \(N\) has square integrable representations. In this paper it is proved, that each of these three ``exceptional'' cases are stepwise (in fact 2-step) square integrable. Then explicit Plancherel and Fourier inversion theorems for commutative nilmanifolds are obtained. Finally, these results are specialized to weakly symmetric Riemannian nilmanifolds, also an extension of this specialization to weakly symmetric Finsler nilmanifolds is given.