AdaGDA: Faster Adaptive Gradient Descent Ascent for Minimax Optimization

Created in July 12, 2023

2023   ·   optimization   ·   paper-review

Minimax optimization $\min_x \max_y f(x, y)$ underlies adversarial training, GANs, and game-theoretic ML. Standard gradient descent ascent (GDA) struggles with non-convex-non-concave settings and requires careful step-size tuning.

AdaGDA (arXiv:2106.16101) introduces adaptive step sizes for both the descent and ascent updates, in the spirit of Adam/AdaGrad applied to single-objective optimization.

Key contributions:

  • Faster convergence rates than constant-step GDA in nonconvex-strongly-concave settings
  • Robustness to step-size hyperparameters via adaptivity
  • Theoretical analysis providing concrete convergence bounds

The result is practically relevant for adversarial and game-theoretic training where standard methods are notoriously sensitive to learning-rate selection.



Enjoy Reading This Article?

Here are some more articles you might like to read next:

  • The Expressive Power of Transformers with Chain of Thought
  • Landscape of Thoughts — Visualizing Where LLM Reasoning Actually Goes
  • Magellan — Guided MCTS for Escaping the Gravity Wells of LLM Creativity
  • PriorZero — Injecting LLM Priors into MuZero-Style World Models at the MCTS Root
  • SuperThoughts — Reasoning Tokens in Superposition