Blog: Survey of Optimizers

Neural-network optimization in 2025-2026 is no longer well described as a succession of new Adam variants.

Neural-network optimization in 2025-2026 is no longer well described as a succession of new Adam variants. The design space has expanded from coordinates to matrices and layers, from fixed training horizons to policies over time, and from mathematical update rules to state…