Homoclinic, heteroclinic, and periodic orbits for a class of indefinite Hamiltonian systems (Q1060877)

Consider the Hamiltonian system in Euclidian n-space E with inner product \((, )\) given by the Hamiltonian function on \(E\times E:\) \(H(p,q)=1/2(Sp,p)+V(q)\), where S is a symmetric linear operator with one negative and n-1 positive eigenvalues. The potential V is assumed to satisfy the following conditions: \(V\in C^ 3(E,{\mathbb{R}})\) and there is a closed, bounded convex set \(C\subset E\) which is the closure of a component of the set \(\{q\in E:V(q)>0\}\), and which has the following property: if \(q\in \partial C\), and V'(q)\(\neq 0\) and \((SV'(q),V'(q))=0\) then \((V''(q)SV'(q),SV'(q))<0\). Moreover, if \(q\in \partial C\) and \(V'(q)=0\), \(\Sigma =\{q\in E:\) \((S^{-1}q,q)<0\}\) then \(C-\{q\}\subset q+\Sigma\), and \(V(q)>0\) for all \(q\in int(C).\) It is shown: (I) If V'(q)\(\neq 0\) for \(q\in \partial C\) then there exists a periodic orbit, (II) If there exists a unique point \(a\in \partial C\) with \(V'(a)=0\) then there exists a homoclinic orbit with q(t)\(\to a\), p(t)\(\to 0\) (\(| t| \to \infty)\). (III) If there exist exactly two points a,b\(\in \partial C\) with \(V'(a)=V'(b)=0\) then there exists a heteroclinic orbit with q(t)\(\to a\) \((t\to -\infty),\) q(t)\(\to b\) \((t\to +\infty),\) p(t)\(\to 0\) (\(| t| \to \infty).\) The main idea of the proofs is a monotonicity property of the orbits. The set \(\Sigma\) defined above is the disjoint union of two open cones, the closure of one of them is taken to define a partial ordering in E. Then for any orbit in the energy surface \(H=0\) with q(t)\(\in int(C)\) this projection q(t) is strictly monotonic. Such indefinite Hamiltonian systems are observed in some nonlinear mechanics arising from problems in modelling nonlinear water waves.