Variational analysis of a composite function: A formula for the lower second order epi-derivative (Q1192534)

The author considers the real functional \(f(x):=g(F(x))\) on the real Banach space \(X\), where \(F\) is a \(C^ 1\)-map in some neighbourhood of a point \(\bar x\in X\) into a Banach space \(Y\) and twice Fréchet differentiable at \(\bar x\) and \(g\) is a proper convex function on \(Y\) being finite and l.s.c. at \(\bar y=F(\bar x)\). In a very nice manner he derived an exact formula for the lower second order epi-derivative of such a composite function. He generalized considerations made by Rockafellar (1988, 1989) and Chaney (1987). Under the regularity condition (H): Im \(F'(\bar x)+\text{cl(Lin(dom} g-\bar y))=Y\), he proved the following main result. Theorem 1. If the regularity hypothesis (H) is satisfied, then for a given \(x^*\) we have the following alternative with \(A=F'(\bar x)\): (a) either \(f''_{-}(\bar x,x^*;h)=-\infty\) for some \(h\), (b) or \(\{y^*\in\partial g(\bar y): A^* y^*=x^*\}\neq\emptyset\) and for any \(h\), \[ \begin{aligned} f''_{-}(\bar x,x^*;h) & =\lim_{{{{h'\to h}\atop{t\to+0}}\atop{w\to w(h)}}}\inf\bigl[g(\bar y+tAh'+t^ 2 w)- g(\bar y)-t\langle x^*,h'\rangle\bigr]/t^ 2\\ & =\lim_{{{{h'\to h}\atop {t\to +0}}\atop {w\to w(h)}}}\inf\bigl[g(\bar y+tAh'+t^ 2 w)- g(\bar y)-t\langle x^*,h'\rangle\bigr]/t^ 2.\end{aligned} \] If, in addition, \(f''_{-}(\bar x,x^*;h)<\infty\) for some \(h\), then \(\langle y^*,Ah\rangle\leq \langle x^*,h\rangle\) for all \(y^*\in\partial g(\bar y)\). In a few corollaries estimations and more specific formulas are given. The complete proof of this Theorem 1 is presented. The main tool is a certain Lyusternik like Lemma. Theorem 1 is applied to the function \(f(x)=\max_{q\in Q} f(q,x)\), where \(Q\) is a compact metrizable space and \(f\), \(f_ x\) and \(f_{xx}\) are continuous with respect to \((x,q)\).