Anatomy of the shape Hessian via Lie brackets (Q1817934)

The shape derivatives used in shape optimization problems are introduced through the flow generated by an autonomous vector field. The second order differential, or the Hessian, is naturally introduced through a pair of vector fields. Clearly, the second order shape derivative is not necessarily symmetric unless additional conditions regarding the vector fields are imposed. A structure theorem is given where the second shape derivative is decomposed into three parts. The first one is given by a bilinear form acting on the components of the vector fields normal to the boundary of the domain. The second part is symmetric with respect to the pair of vector fields, and the last one involves half of Lie bracket of the fields. It is obtained that in order the mixed derivative of a smooth functional be commutative with respect to the vector fields, it suffices that the normal component of the Lie bracket of the pair of fields vanishes on the boundary of the domain. The main theorem of the paper relates the symmetry of the linear operator, that generates the bilinear form in the first term, to the smoothness of the functional. The results are illustrated on an example of shape optimization problem with constraints in the form of a Dirichlet boundary-value problem for Poisson equation.