Stability of hypersurfaces of constant mean curvature in Riemannian manifolds (Q1107857)

Let \(M_ 0\) be a compact oriented hypersurface of constant mean curvature \(H_ 0\) in an \((n+1)\)-dimensional Riemannian manifold \(\bar M.\) Such hypersurfaces arise as the stationary points of a constrained variational problem: minimize the (hypersurface) area functional A subject to fixed volume V. The method of Lagrange multipliers then realizes \(M_ 0\) as a stationary point for the unconstrained functional \(J=A+H_ 0V.\) The authors say that \(M_ 0\) is stable if the second variation \(\delta^ 2A\) is nonnegative for all volume-preserving deformations of \(M_ 0\); equivalently, if \(\delta^ 2J\geq 0\) for all such deformations. (In a review of several related papers [MR 85k:58021] R. Osserman refers to the latter formulation as `weak stability'.) The paper has three main results: (A): Suppose that \(\bar M\) is a simply- connected space form of curvature c. Then \(M_ 0\) is stable if and only if it is a geodesic sphere in M. (The case \(c=0\) was actually done in an earlier paper of the first two authors [ibid. 185, 339-353 (1984; Zbl 0513.53002)].) (B): Suppose that \(\bar M\) is a projective space, with the usual metric, over one of the fields \({\mathbb{R}}\), \({\mathbb{C}}\) or \({\mathbb{H}}\). If \(M_ 0\) is the boundary of a geodesic tube of radius \(\rho\) over a totally geodesic subspace of \(\bar M\) then \(M_ 0\) is stable if and only if \(\rho\) is sufficiently constricted (the paper provides precise bounds). In particular, there exist geodesic hyperspheres which are not stable, and there exist stable hypersurfaces which are not geodesic hyperspheres. (C): Suppose that \(\bar M\) is the complex hyperbolic space (i.e., the noncompact dual of the complex projective space). Then every geodesic hypersphere is stable. \{Reviewer's Comments: (a) The authors do not make clear why they prefer the formulation in terms of the functional J to the more natural formulation in terms of A. The usual reason for introducing Lagrange multipliers is to eliminate the constraints on the allowed deformations; but this does not carry through to the second variation, and indeed the authors impose the same constraints on J as they would on A: volume preserving deformations. (b) \textit{E. Heinze} [Math. Ann. 280, No.3, 389- 402 (1988; Zbl 0628.53044)] proves Theorem (A) in a different way.\}