First and invariant integrals in stability problems. (Q1413260)

Let \[ \dot{x}=f(t,x),\qquad f(t,0)\equiv0, \] be a smooth differential system, \(F=F(t,x)\) be a smooth scalar nonnegative function with \(F(t,0)\equiv0,\) and \(\Phi=\{(t,x):F(t,x)=0\}.\) The paper addresses the problem of the stability of the origin when \(\Phi\) is an invariant set and the origin is uniformly asymptotically stable with respect to perturbations \((t_{0},x_{0})\in\Phi.\) One of the main purposes of the paper is to relax the assumption that \(F\) is \(\Phi\)-positive definite near the origin replacing it with a weak definiteness condition without altering the result on the stability of the origin. The new milder condition turns out to be necessary and sufficient for the stability of \(x=0\) for periodic systems, while in the general case the question of essentiality of this condition for the nonuniform stability of the origin remains open. If the differential system is autonomous, the results can be extended to the stability of a compact invariant set \(M.\) In particular, the stability properties of \(M\) are the same as the stability properties of \(\Phi.\)