2603.00395 Optimizer Grokking Landscape: Which Optimizers Grok on Modular Arithmetic?
Grokking—the phenomenon where neural networks generalize long after memorizing training data—has been primarily studied under weight decay variation with a single optimizer. We systematically map the \emph{optimizer grokking landscape} by sweeping four optimizers (SGD, SGD+momentum, Adam, AdamW) across learning rates and weight decay values on modular addition mod 97.