Extremal curves in nilpotent Lie groups (Q359553)

The existence of \textit{abnormal geodesics} (length minimizing curves which do not arise as projections of bicharacteristic solutions of Hamilton's equations) is a characteristic feature of sub-Riemannian geometry which distinguishes it from its Riemannian counterpart. The existence of strictly abnormal geodesics in sub-Riemannian geometry was first confirmed by Montgomery. The book [\textit{R. Montgomery}, A tour of subriemannian geometries, their geodesics and applications. Providence, RI: American Mathematical Society (2002; Zbl 1044.53022)] contains an informative summary of the history and development of the subject. In this paper, the authors characterize the abnormal geodesics in arbitrary free nilpotent Lie groups (free Carnot groups). The characterization identifies such curves as precisely the horizontal curves contained in certain specific algebraic subvarieties defined by canonical polynomial mappings obtained from the structure constants of the associated graded nilpotent Lie algebra. The characterization is thus purely algebraic, indicating that techniques of real algebraic geometry may be of use in further investigation of the structure and regularity of sub-Riemannian geodesics. As applications of their general theorem, the authors (i) provide a short new proof of a recent result of Tan and Yang on the \(C^\infty\) regularity of geodesics in step-three Carnot groups, (ii) give an alternate proof of the smoothness of generic length-minimizing curves in rank-two Carnot groups (this is a special case of more general results of Liu and Sussmann), and (iii) rederive the well-known example of Golé and Karidi of a strictly abnormal geodesic in the free Carnot group of rank two and step at least four.