Analytic regularity for an operator with Trèves curves (Q1876244)

The author studies a partial differential operator \(\mathcal{H}\) with analytic coefficients in \(\mathbb{R}^{3},\) which is of the form \[ \mathcal{H}=-\frac{\partial^2}{\partial t^2}-t^2\left( \frac {\partial^2}{\partial x_1^2}+\frac{\partial^2}{\partial x_2^2 }\right) -\left( x_1\frac{\partial}{\partial x_2}-x_2\frac{\partial }{\partial x_1}\right) ^2.\tag{1} \] A classical result of Hörmander guarantees the hypoellipticity of \(\mathcal{H}\) on any open set \(\Omega\subset\mathbb{R}^{3}.\) The main results of the paper deal with the analytic hypoellipticity of the operator \(\mathcal{H}\) . The essential theorems proved by the author are: Theorem 1. Let \(\Omega\subset\mathbb{R}^{3}\) be open. If \(\Omega\cap\{t=0\}\neq \emptyset,\) then \(\mathcal{H}\) is not analytic hypoelliptic on \(\Omega.\) Theorem 2. The operator \(\mathcal{H}\) is globally analytic hypoelliptic on \(\Omega_{p,q}\), for every \(p,q>0,\) where \[ \Omega_{p,q}=\{(x_1,x_2,t):x_1^2+x_2^2<p^2,t^2<q^2\}. \] Theorem 1 is consistent with the \textit{F. Trèves}' conjectures [Proc. Symp. Pure Math. 65, 201--219 (1999; Zbl 0938.35038)], while as a corollary of Theorem 2 the operator \(\mathcal{H}\) is analytic hypoelliptic at the origin, i. e., if \(u\) is a distribution near the origin and \(\mathcal{H}u\) is analytic near the origin, then \(u\) is analytic near the origin.