Approaching nonsmooth nonconvex optimization problems through first order dynamical systems with hidden acceleration and Hessian driven damping terms (Q1653320)

In this paper the authors carry out an asymptotic analysis of the proximal-gradient dynamical system \[ \begin{cases} \dot x(t)+x(t)=prox_{\gamma f}[x(t)-\gamma\nabla \phi(x(t))-ax(t)-by(t)],\\ \dot y(t)+ax(t)+by(t)=0 \end{cases} \] where \(f\) is a proper, convex and lower semicontinuous function, \(\phi\) a possibly nonconvex smooth function and \(\gamma\), \(a\) and \(b\) are positive real numbers. They show that the generated trajectories approach the set of critical points of \(f+\phi\), here understood as zeros of its limiting subdifferential, under the premise that a regularization of this sum function satisfies the Kurdyka-Łojasiewicz property. The authors also establish convergence rates for the trajectories, formulated in terms of the Łojasiewicz exponent of the considered regularization function.