A geometric criterion for Gelfand pairs associated with nilpotent Lie groups (Q5940316)

Let \(G= K\ltimes N\), where \(N\) is a connected and simply connected nilpotent Lie group and \(K\) is a compact Lie group acting smoothly on \(N\) as automorphisms. The pair \((K,N)\) is called a Gelfand pair if the algebra \(L^1_K(N)\) of \(K\)-invariant integrable functions on \(N\) is a commutative subalgebra of the convolution algebra \(L^1(N)\). \textit{C. Benson, J. Jenkins} and \textit{G. Ratcliff} proved [Trans. Am. Math. Soc. 321, 85-116 (1990; Zbl 0704.22006)] that \((K,N)\) is a Gelfand pair only if \(N\) is at most two step nilpotent. They collaborated with \textit{R. L. Lipsman} [Pac. J. Math. 178, 1-36 (1997; Zbl 0868.22015)] to formulate a geometric criterion for \((K,N)\) to be a Gelfand pair in terms of coadjoint orbits and to show its validity in the case where \(N\) is a Heisenberg group, and then proved [J. Geom. Anal. 9, 569-582 (1999)] that the orbit condition of the criterion is necessary for \((K,N)\) to be a Gelfand pair, leaving its sufficiency as a conjecture. Let \(\mathfrak g\) (resp. \(\mathfrak k\)) be the Lie algebra of \(G\) (resp. \(K\)). Denoting by \({\mathfrak k}^\perp\) the annihilator of \(\mathfrak k\) in \({\mathfrak g}^*\), the orbit condition in question is stated as follows: \(G\cdot\xi\cap {\mathfrak k}^\perp=K\cdot \xi\) for every \(\xi\in{\mathfrak k}^\perp\). Let \(\mathfrak n\) be the Lie algebra of \(N\), and take a \(K\)-invariant inner product \(\langle\cdot,\cdot \rangle\) on \(\mathfrak n\). Let \(\mathfrak z\) be the center of \(\mathfrak n\) and \(V\) the orthogonal complement of \(\mathfrak z\) in \(\mathfrak n\). For each \(z\in{\mathfrak z}\), define an operator \(J_z: V\to V\) by \(\langle J_z v,w\rangle=\langle z, [v,w]\rangle\) \((v, w\in V)\). Using essentially these operators, this paper presents a proof of the conjecture and thus makes the criterion complete. As an application of the criterion, the author takes up a certain class of two step nilpotent Lie groups considered in a paper of \textit{J. Lauret} [Manuscripta Math. 99, 287-309 (1999; Zbl 0931.22005)] and notes that the equivalence of (iii) and (iv) in Theorem 4.2 of the paper does not hold.