Extremals of a left-invariant sub-Finsler metric on the Engel group (Q2203436)

The Engel group is the four-dimensional connected simply connected nilpotent Lie group \(G\) whose Lie algebra \(\mathfrak{g}\) is spanned by vectors \(X,Y,Z,V\) satisfying the bracket relations \([X,Y]=Z\), \([X,Z]=V\), and all other brackets vanishing. Its usual left-invariant bracket-generating horizontal distribution \(D\) is given at the identity \(e\) by \(D(e) = \operatorname{span}\{X,Y\}\). A left-invariant sub-Finsler structure on \(G\) corresponds to a choice of norm \(F\) on the two-dimensional vector space \(D(e)\), and induces a left-invariant sub-Finsler metric (or distance) \(d\) on \(G\). The length-minimizing arcs of this metric are, by the Pontryagin maximum principle, projections of arcs \(\psi\) in the cotangent bundle which solve an appropriate maximization problem for the Hamiltonian; the solutions of this problem (not all of which necessarily project to globally length-minimizing arcs) are called extremals. In this paper, the authors explicitly compute all the extremals for arbitrary sub-Finsler metrics, in terms of the equation \(r = r(\theta)\) in polar coordinates for the unit sphere of the norm \(F\). Some simplifications are given in the special case of a norm \(F_\alpha\) whose unit sphere is a square rotated by some angle \(\alpha\), such as the \(\ell^1\) and \(\ell^\infty\) norms. It is also remarked that the results extend easily to sub-Finsler quasimetrics, for which the homogeneity condition of the norm \(F\) is relaxed to require only \(F(tx) = t F(x)\) for scalars \(t>0\), so that the unit ball of \(F\) is still convex but not necessarily symmetric. Previous work on this problem was done in [\textit{J. N. Clelland} et al., Asian J. Math. 11, No. 4, 699--726 (2007; Zbl 1141.53026)], which studied geodesics on so-called Engel manifolds, of which the Engel group is the prototypical example. However, they imposed the additional assumption that \(F^2\) should be strongly convex, which would exclude norms such as \(\ell^1\) whose unit sphere is a polygon. The present paper allows for the norm \(F\) to be completely general; moreover, considering of only the Engel group itself allows the development to be more elementary.