Two steps at a time-taking GAN training in stride with Tseng's method
From MaRDI portal
Publication:5089720
Abstract: Motivated by the training of Generative Adversarial Networks (GANs), we study methods for solving minimax problems with additional nonsmooth regularizers. We do so by employing emph{monotone operator} theory, in particular the emph{Forward-Backward-Forward (FBF)} method, which avoids the known issue of limit cycling by correcting each update by a second gradient evaluation. Furthermore, we propose a seemingly new scheme which recycles old gradients to mitigate the additional computational cost. In doing so we rediscover a known method, related to emph{Optimistic Gradient Descent Ascent (OGDA)}. For both schemes we prove novel convergence rates for convex-concave minimax problems via a unifying approach. The derived error bounds are in terms of the gap function for the ergodic iterates. For the deterministic and the stochastic problem we show a convergence rate of and , respectively. We complement our theoretical results with empirical improvements in the training of Wasserstein GANs on the CIFAR10 dataset.
Recommendations
- Alternating Proximal-Gradient Steps for (Stochastic) Nonconvex-Concave Minimax Problems
- Optimality Conditions for Nonsmooth Nonconvex-Nonconcave Min-Max Problems and Generative Adversarial Networks
- Training GANs with centripetal acceleration
- Efficient second-order optimization with predictions in differential games
- Convergence rate of \(\mathcal{O}(1/k)\) for optimistic gradient and extragradient methods in smooth convex-concave saddle point problems
Cites work
- scientific article; zbMATH DE number 3534286 (Why is no real title available?)
- scientific article; zbMATH DE number 845714 (Why is no real title available?)
- A Stochastic Approximation Method
- A forward-backward splitting method for monotone inclusions without cocoercivity
- A primal-dual algorithm with line search for general convex-concave saddle point problems
- Applications of a Splitting Algorithm to Decomposition in Convex Programming and Variational Inequalities
- Breaking the curse of dimensionality with convex neural networks
- Convergence rate of \(\mathcal{O}(1/k)\) for optimistic gradient and extragradient methods in smooth convex-concave saddle point problems
- Convex Optimization in Signal Processing and Communications
- Convex analysis and monotone operator theory in Hilbert spaces
- Dual extrapolation and its applications to solving variational inequalities and related problems
- Extragradient Method with Variance Reduction for Stochastic Variational Inequalities
- Minibatch forward-backward-forward methods for solving stochastic variational inequalities
- Nonlinear total variation based noise removal algorithms
- Projected reflected gradient methods for monotone variational inequalities
- Prox-Method with Rate of Convergence O(1/t) for Variational Inequalities with Lipschitz Continuous Monotone Operators and Smooth Convex-Concave Saddle Point Problems
- Shadow Douglas-Rachford splitting for monotone inclusions
- Solving variational inequalities with stochastic mirror-prox algorithm
- The complexity of constrained min-max optimization
- ℓ1 Regularization in Infinite Dimensional Feature Spaces
Cited in
(15)- Variable sample-size operator extrapolation algorithm for stochastic mixed variational inequalities
- Optimality Conditions for Nonsmooth Nonconvex-Nonconcave Min-Max Problems and Generative Adversarial Networks
- Alleviating limit cycling in training GANs with an optimization technique
- The backtrack Hölder gradient method with application to min-max and min-min problems
- An accelerated minimax algorithm for convex-concave saddle point problems with nonsmooth coupling function
- Training GANs with centripetal acceleration
- Tseng’s Algorithm with Extrapolation from the past Endowed with Variable Metrics and Error Terms
- Alternating Proximal-Gradient Steps for (Stochastic) Nonconvex-Concave Minimax Problems
- A modified Tseng's algorithm with extrapolation from the past for pseudo-monotone variational inequalities
- GANs training: A game and stochastic control approach
- On adversarial robustness and the use of Wasserstein ascent-descent dynamics to enforce it
- A mirror inertial forward-reflected-backward splitting: convergence analysis beyond convexity and Lipschitz smoothness
- Variable sample-size optimistic mirror descent algorithm for stochastic mixed variational inequalities
- Efficient second-order optimization with predictions in differential games
- Fast convergence of the primal-dual dynamical system and corresponding algorithms for a nonsmooth bilinearly coupled saddle point problem
This page was built for publication: Two steps at a time-taking GAN training in stride with Tseng's method
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5089720)
