A criterion for analytic hypoellipticity of a class of differential operators with polynomial coefficients (Q794197)

The authors study microlocal analytic hypoellipticity and analytic hypoellipticity for various classes of pseudodifferential operators with ''polynomial coefficients'' and multiple characteristics. For the first result, we consider operators \(P(t,D_ t,D_ y)\) acting on \({\mathcal D}'({\mathbb{R}}_{t,y}^{n_ 1+n_ 2}).\) It is assumed that P is transversally elliptic on the characteristic variety. If \(\eta\) is the space variable, we write \(\omega =\eta /| \eta |\), \(z=| \eta |^{-1/2}\), \(t'=z^{-1}t\) and (using the Grušin decomposition for pseudodifferential operators with polynomial coefficients into homogeneous components) \[ z^{-2\mu -m}\quad P(\eta)=A_{(z,\omega)}(t',D_ t)=\sum_{j}z^ j\quad A_{m- j}(1,\omega)(t',D_{t'}) \] \[ A_{(0,\omega)}(t',D_{t'})=A_ m(1,\omega)(t',D_{t'})=\sum_{| \alpha +\beta | \leq m}C^ m_{\alpha \beta}(\omega)t^{\prime\alpha}D^{\beta}_{t'}=P_ m(\eta /| \eta |). \] For \(| z|<\delta\), \(| \omega - \omega_ 0|<\delta\), the spaces \(E_ 1(z,w)\), \(E_ 2(z,w)\) are the perturbations respectively, of the kernels of \(A(0,\omega_ 0)\) and \(A^*(0,\omega_ 0)\). Let M(z,\(\omega)\) be the matrix of \(A(z,\omega):\quad E_ 1(z,\omega)\to E_ 2(z,\omega)\) with respect to orthonormal bases \(\{h^ i_ j(t',z,\omega)\}\) for these spaces. Finally let \(M(\eta)=M(| \eta |^{-1/2},\eta /| \eta |)\) with respect to the bases \(\{h^ i_ j(t| \eta |^{1/2},| \eta |^{-1/2},\eta /| \eta |)\}.\) Theorem: The operator \(P(t,D_ t,D_ y)\) is microlocally analytic hypoelliptic (resp. \(C^{\infty}\) hypoelliptic) at \((0,y_ 0;0,\eta_ 0)\) if and only if \(M(D_ y)\) is microlocally analytic hypoelliptic (resp. \(C^{\infty}\) hypoelliptic) at \((y_ 0,\eta_ 0).\) Corollary: If the index of \(P(\eta)\) is positive, then \(P(t,D_ t,D_ y)\) is microlocally neither \(C^{\infty}\) nor analytic hypoelliptic at \((0,y_ 0\); \(0,\eta_ 0).\) Theorem: If the index of \(P(\eta)\) is zero, then the following are equivalent: \((i)\quad P(t,D_ t,D_ y)\) is analytic hypoelliptic at \((0,y_ 0;0,\eta_ 0). (ii)\quad P^*P(t,D_ t,D_ y)\) is analytic hypoelliptic at \((0,y_ 0,0,\eta_ 0).\) (iii) There exists \(\delta>0\) such that \(P(\eta)\) is injective for all \(\eta \in {\mathbb{C}}^{\eta_ 2}\), \(| \eta |>\delta^{-2}\), \(| \eta /| \eta | - \eta_ 0/| \eta_ 0| |<\delta.\) (iv) The product of the small eigenvalues of \(P^*P(\eta)\) is elliptic at \(\eta_ 0.\) Results are also obtained for pseudodifferential operators on nilpotent Lie groups of step two. - The exposition in this paper is of exceptionally high quality.