On the structure of an invertible system with a trajectory homoclinic to a saddle-center (Q942005)

The paper considers an analytic invertible vector field \(f\) in the four-dimensional space \(\mathbb R^4\) with a trajectory \(\Gamma\) homoclinic to a singular point \(\mathcal O\) of saddle-center type, i.e. the linearization operator \(\mathcal Df(O)\) has nonzero eigenvalues \(\pm\lambda\), \(\pm i\omega\). The invertibility means that there exists a linear involution \(R:\mathbb R^4\to\mathbb R^4\), \(R^2=\text{id}\), \(\dim\text{Fix}(R)=2\), such that \(f(Rx)=-Rf(x)\) for any \(x\in\mathbb R^4\). It is also assumed that the singular point \(\mathcal O\) is symmetric, i.e. \(R(\mathcal O)=\mathcal O\). Such a singular point has a local two-dimensional singular submanifold (center manifold) \(W^c\) filled by symmetric periodic trajectories (Lyapunov family) surrounding \(\mathcal O\). There exist local three-dimensional center stable and center unstable submanifolds \(W^{cs}\) and \(W^{cu}\) that are the union of stable and unstable submanifolds of periodic trajectories from \(W^c\). Also, there exist local one-dimensional stable and unstable submanifolds \(W^s\) and \(W^u\) which are related by the symmetry \(R\). Thus, the homoclinic trajectory \(\Gamma\) is either symmetric or there exist two homoclinic trajectories related by the symmetry and composing a ``figure-eight'' curve. The primary aim of the work is to study the dynamics in a neighborhood of a symmetric homoclinic trajectory \(\Gamma\). In particular, it is shown that under some conditions there exist trajectories homoclinic to periodic solutions on the center manifold and also periodic trajectories accumulating to a loop.