Radon transform on nilpotent Lie groups (Q690230)

Let \(N\) be a simply connected Lie group and \({\mathfrak n}\) be its Lie algebra. Let \(\rho\) be a unipotent representation of \(N\) on a finite dimensional vector space \(V\). The adjoint representation is an example of such a representation. For fixed \(\xi \in V^*\) one defines subspaces \({\mathfrak m} \subset {\mathfrak n}\) and \(E \subset V\) such that \({\mathfrak n} = \{X| d\rho^*(X)\xi = 0\}\oplus{\mathfrak m}\), \(V^*=d\rho^*({\mathfrak n}) \xi\oplus E^ \perp\). Let \(U\) be a complementary subspace to \(E\) in \(V\). One defines the Radon transform of \(f\in S(V)\) as \[ {\mathcal R}(Y,u)=\int_ Ef \bigl( \rho(\text{Exp} Y)(u+z)\bigr)dz\;(Y \in{\mathfrak m},\;u \in U), \] and the dual Radon transform of a function \(\varphi\) defined on \({\mathfrak m}\times U\) by \[ {\mathcal R}^*(x)=\int_{\mathfrak m} \varphi\Biggl(Y,\pi_ U \biggl( \rho\bigl( \text{Exp}(-Y) \bigr)x \biggr)\Bigr)dY\;(x\in V), \] where \(\pi_ U\) is the projection onto \(U\) parallel to \(E\). One proves an inversion formula of the form \({\mathcal R}^*(F(D) ({\mathcal R}f))=f\), where \(F(D)\) is a Fourier multiplier. In the case of the adjoint representation \(F(D)\) is a differential operator, and one proves a Plancherel formula for the Radon transform. The special case of a two-step nilpotent Lie group is studied in some details.