Analytic hypoellipticity at non-symplectic Poisson-Treves strata for certain sums of squares of vector fields (Q950691)

The purpose of this paper is to study analytic hypoellipticity for certain sums of squares of vector fields having non-trivial Poisson-Treves stratification, i.e., the stratification has non-trivial strata at depth larger than one. In all examples known so far, analytic hypoellipticity holds in the sense of germs when the characteristic manifold is non-symplectic and is the only stratum of the stratification, which implies that the symplectic form \(\sigma= d\xi\wedge dx\) has constant rank on the characteristic manifold. Furthermore, the bicharacteristic curves are closed curves foliating a given neighborhood of a characteristic point on which hypoellipticity in the sense of germs is obtained. The authors study two different cases. The first is a sum of two squares of vector fields for which the stratification has a symplectic ``surface'' (i.e., of depth 1) stratum and a deeper non-symplectic stratum \(\Sigma_2\): \[ P(t,x,D_t, D_x)= D^2_t+ (x_1 D_2- x_2 D_1+ t^k(x_1 D_1+ x_2 D_2))^2,\;k\geq 2, \] \[ \Sigma_2= \{\tau= 0,\,t= 0,\,x_1\xi_2- x_2\xi_1= 0,\,\xi= (\xi_1,\xi_2)\neq 0\}. \] They prove that if \(U\) is an open subset of \(\mathbb{R}^3\) in the variables \((t,x_1, x_2)\) projecting on an annulus of the form \(r_1<|x|<r_2\) and containing points where \(t= 0\), then \(P\) is analytic hypoelliptic (in the sense of germs) at points in \(U\cap\Sigma_2\). The second model operator does not have closed orbits, but have closed limit sets: \(P= D^2_t+X^2_2\), where \[ X_2= g_1(x) g_2(x) (x_1 D_2- x_2 D_1+ \mu(x_1 D_1+ r_2 D_2+ t^k(x_1 D_1+ x_2 D_2)), \] \[ g_1(x)= |x|^2- a^2,\;g_2(x)= b^2- |x|^2,\quad 0< a< b,\;\mu> 0. \] The authors prove the following propagation of regularity result: Let \(U\) be an open set in \(\mathbb{R}^3\) projecting onto an annulus of the form \(a<|x|< b\) in the \(x\)-variables. Let \(u\) be a distribution such that \(P_u\in C^\omega(U)\). Then if both circles \(g_i(x)= 0\), \(i= 1,2\), do not intersect \(WF_a(u)\), we have \(\mu\in C^\omega(U)\).