Generalized differential calculus for nonsmooth and set-valued mappings (Q1329327)

Some generalized differentiability concepts for multifunctions and nonsmooth mappings in \(\mathbb{R}^ n\) are studied. In this paper the most important one is the so-called coderivative of multifunctions introduced earlier [the author, Dokl. Akad. Nauk SSSR 254, 1072-1076 (1980; Zbl 0491.49011)] using the normal cone to the graph in the sense given by the author in Prikl. Mat. Mekh. 40, 1014-1023 (1976; Zbl 0362.49017). This normal cone is not convex. The corresponding nonconvex coderivative appears to be useful in control theory to obtain necessary optimality conditions in differential inclusions. In this paper a rich calculus for the coderivative and related subdifferential constructions is developed using a variational approach, namely an extremal generalization of the separability theorem for nonconvex sets. The proof of this result is based on a special smooth approximation procedure using proximal points (the so-called metric approximation method).