On the Hopf-Oleinik lemma for degenerate-elliptic equations at characteristic points (Q342989)

Let \(\Omega\) be a bounded smooth open set in \(\mathbb{R}^N\) and let \(\mathcal{L}= \sum_{i,j=1}^Na_{ij}(p)\partial^2_{ij}+\sum_{k=1}^Nb_k(p)\partial_k\) be an elliptic operator, where the coefficients \(a_{ij},b_k\) are continuous functions on a neighborhood of \(\overline{\Omega}\) and the matrix \(A_{ij}(p)=(a_{ij}(p))_{i,j=1}^N\) is symmetric nonnegative definite and not identically vanishing. When a point \(y\in \partial \Omega\) is such that the unit inner normal \(\nu\) to \(\partial \Omega\) at \(y\) does not belong to the kernel of \(A(y)\), then an \(\mathcal{L}\)-barrier function for \(\Omega\) to \(y\) can be constructed, thus ensuring the validity of the Hopf Lemma for \(\mathcal{L}\) in \(\Omega\) at the point \(y\). In this paper, the authors investigate the validity of the Hopf Lemma at \(y\) in the case in which the unit normal \(\nu\) belongs to the kernel of \(A(y)\) (the point \(y\), in this case, is called characteristic for \((\Omega,\mathcal{L})\)). Several results and counterexamples are given for various classes of operators \(\mathcal{L}\) including the Kohn Laplacian on the Heisenberg group, sub-Laplacians in general homogeneous Carnot groups, and operators having non-trivial first order terms along a characteristic direction. The validity of the Hopf Lemma for nonlinear operators \(\mathcal{L}\), expressed in terms of a comparison principle for a couple of functions \((u,v)\), is also discussed. In particular, the authors prove that the Hopf Lemma holds if a point \(y\) is not characteristic for at least one of the linearized operators \(\mathcal{L}_u\), \(\mathcal{L}_v\) and that this could be not true if \(y\) is a characteristic point for both \(\mathcal{L}_u\), \(\mathcal{L}_v\), as shown by a counterexample involving the nonlinear Levi operator.