Local \(p\)-convexity (Q1580487)

It was shown in the 50's by Malgrange, that a differential equation \(P(D)u=f\), \(D=-i\partial/\partial x\), with constant coefficients in an open set \(X\subset {\mathbb {R}}^n\) has a solution \(u\in C^{\infty}(X)\) for all \(f \in C^{\infty}(X)\) if and only if for every compact set \(K\subset X\), there exists a compact set \(K' \subset X\) such that \[ \varphi \in C^{\infty}_0(X),\;{\text{supp}} P(-D)\varphi\subset K\Rightarrow {\text{supp}} \varphi \subset K'.\tag{1} \] The fact that this condition is true for every convex set \(X\), led the author to call an open set satisfying \((1)\) a \(P\)-convex set. This paper starts by recalling standard arguments used to study \(P\)-convexity and showing that they lead to a notion of \(P\)-convexity with respect to vectors \(\xi\in{\mathbb {R}}^n\setminus\{0\}\) with \(p(\xi)=0\), where \(p\) is the principal symbol of \(P\). A set is \(P\)-convex if and only if it is \(P\)-convex with respect to every such \(\xi.\) By constructing null solutions \(P(-D) u =0\) with certain properties, the author gives necessary conditions for the \(P\)-convexity with respect to \(\xi\). Sufficiency conditions are then discussed in various cases. In particular, if \(0 \neq \xi \in {\mathbb {R}}^n\) is an isolated characteristic of \(P\), then \(X\) is \(P\)-convex with respect to \(\xi\) and \(- \xi\) if and only if the boundary distance \(d(x, CX)\) satisfies the minimum principle in every hyperplane with conormal \(\xi.\) In addition, the author addresses the geometrical interpretation of the necessary conditions for the case when the curvatures of the characteristic surface at \(\xi\) transversally to the radial direction are different from zero but not all of the same sign (as in the case of the ultrahyperbolic operator in \({\mathbb {R}}^4\)). Note that one of the final sections focuses on the exterior conormal set of the parallel sets of a closed set and related versions of Holmgren's uniqueness theorem.