Remarks on critical exponents for Hessian operators (Q752313)

Let \(S_ k(\nabla^ 2u)\), \(k=1,...,n\), be the k-th Hessian operator, i.e., the k-th elementary symmetric function of the Hessian matrix of u, where \(n\geq 2\) is the dimension. Thus, for \(k=1\) we have the Laplacian operator and for \(k=n\) the determinant of the Hessian matrix (Monge- Ampère operator). Consider the Dirichlet problem \[ S_ k(\nabla^ 2u)=(-u)^ p\text{ in } \Omega,\quad u=0\text{ on } \partial \Omega. \] The author proves the following result: Let \(\Omega\) be a ball of \({\mathbb{R}}^ n\) and let \[ \gamma (k)=(n+2k)/(n-2k)\quad if\quad 1\leq k<n/2,\quad \gamma (k)=\infty \quad if\quad n/2\leq k\leq n. \] Then (I) The above problem has no negative solution in \(C^ 1({\bar \Omega})\cap C^ 4(\Omega)\) when \(p\geq \gamma (k)\); and (II) It admits a negative solution which is radially symmetric and is in \(C^ 2({\bar \Omega})\) when \(0<p<\gamma (k)\) and \(p\neq k.\) Hence, the critical exponent of the k-th Hessian operator is determined. Actually, \((-u)^ p\) is replaced throughout the paper by a more general nonlinearity of the form f(x,-u) and the nonexistence result is proved for any bounded star-shaped \(C^ 2\)-domain \(\Omega\). (Negative solutions assure that the k-th Hessian operators are elliptic for \(k\geq 2).\) By using the Newtonian tensor, the above equation is written as the Euler-Lagrange equation for a certain functional. Then a minimax method and the mountain pass lemma yield the existence result (for the ball). Under stronger hypotheses on f(x,-u) the solution is also an absolute minimum of the functional. Nonexistence is established by means of a Pohozaev-type argument.