Differential properties of the Moreau envelope (Q2452447)

The infimal convolution \((f\square g)(x)=\inf \{f(x)+g(x-y) : y\in X\}\) of two extended real-valued functions \(f,g\) defined on a Banach space \(X\) was defined by J. J. Moreau in 1962. As it was shown by Moreau, of particular interest is the function \(g= 2^{-1}\|\cdot\|^2\), in which case the convolution \(f\square g\) is of class \(C^1,\) provided \(X\) is a Hilbert space. Later (1971) H. Brézis considered the more general case of \(f\square (2\lambda)^{-1}\|\cdot\|^2\), for some \(\lambda >0,\) emphasizing its role in the study of some differential inclusions. The aim of the present paper is to study the differentiability and subdifferentiability properties of the Moreau envelope (a term proposed by \textit{R.~T.~Rockafellar}, \textit{R.~J.~B.~Wets} [''Variational analysis'', Grundlehren der Mathematischen Wissenschaften. 317. Berlin: Springer. xiii, 733 p. (1998; Zbl 0888.49001)] \(e_\lambda f =f \square (2\lambda)^{-1}\|\cdot\|^2,\) with applications to the distance function \(d_S(x)=\inf\{\|x-s\| : s\in S\},\) \(S\) a nonempty subset of \(X\). The authors consider various subdifferentials of non-smooth functions: Mordukhovich limiting subdifferential, Clarke subdifferential and Dini subdifferential. Some of the results hold in the more general framework of ''presubdifferentials'', a class of set-valued mappings defined axiomatically. A key role is played by the class of generalized essentially directionally smooth functions, defined and studied by \textit{L. Thibault} and \textit{D. Zagrodny} [SIAM J. Optim. 20, No. 5, 2300--2326 (2010; Zbl 1208.49011)]. The authors prove, among other results, that the directional subregularity of the Moreau envelope of every \(lsc\) function on \(X\) minorized by a negative quadratic function is equivalent to the uniform Gâteaux differentiability of the norm of \(X\). These results are applied in Section 4 to the study of differentiability and subdifferentiability properties of the distance function. As a consequence of the obtained results one shows that a closed subset \(S\) of a Banach space \(X\) is convex if and only if the following two conditions hold: (i)\; \(\limsup_{h\to 0}\|h\|^{-1}\,[d_S(x+h)-d_S(x)]=0,\,\) and \, (ii)\; \(\liminf_{\|x\|\to \infty}[d_{[a,b]}(x)-d_S(x)]\geq 0,\, \) for every pair \(a,b\) of distinct points in \(S\). In particular, every Chebyshev subset of a reflexive Banach space with uniformly Gâteaux differentiable norm is convex.