The Green function of model step two hypoelliptic operators and the analysis of certain tangential Cauchy Riemann complexes (Q1924004)

The authors consider second order hypoelliptic operators of the form \(\Delta = {1\over 2} \sum^n_{j=1} X^2_j + 1^{\text{st}}\) order terms on the nilpotent Lie group \(\mathbb{R}^{n+p}\) where the \(X_j\) together with their first brackets generate the corresponding Lie algebra. For such operators they give explicit formulae for the fundamental solution and for the heat kernel. The work extends the special case of the Heisenberg group \((n\) even, \(p=1)\) which has been treated before in the first and third author's monograph [Calculus on Heisenberg manifolds, Ann. Math. Studies 119, Princeton Univ. Press (1988; Zbl 0654.58033)]. The main strategy to obtain the fundamental solution consists in using a complex Hamiltonian formalism and to integrate the Hamiltonian equations and a generalized transport equation. As in the afore-mentioned special situation the present model case can be extended to general 2-step hypoelliptic operators on manifolds. This is promised to be treated elsewhere. The results are used to invert \(\square_b\) and \(\overline \partial_b\) on boundaries of Siegel upper half spaces of general codimension showing in particular that solvability of \(\square_b\) is equivalent to convergence of the integral in which the fundamental solution can be expressed.