Optimizer Benchmarking
Cross-source consensus on Optimizer Benchmarking from 1 sources and 5 claims.
1 sources · 5 claims
Risks & contraindications
Comparisons
Evidence quality
Highlighted claims
- The survey treats optimizer evaluation as a multi-objective problem involving efficiency, memory, stability, scalability, and implementation complexity. — Navigating LLM Valley: From AdamW to Memory-Efficient and Matrix-Based Optimizers
- Credible LLM optimizer comparisons should report loss, memory, throughput, stability, tuning budget, and implementation details rather than validation loss alone. — Navigating LLM Valley: From AdamW to Memory-Efficient and Matrix-Based Optimizers
- Common benchmark pitfalls include under-tuned baselines, unequal tuning budgets, early-curve overclaiming, ignored wall-clock costs, incomplete memory reporting, small-scale extrapolation, and implementation confounding. — Navigating LLM Valley: From AdamW to Memory-Efficient and Matrix-Based Optimizers
- Fixed-resource evaluation captures practical value when memory savings enable otherwise infeasible configurations. — Navigating LLM Valley: From AdamW to Memory-Efficient and Matrix-Based Optimizers
- Fixed-model evaluation isolates optimizer behavior by holding model, data, token budget, batch size, precision, hardware, and training recipe constant. — Navigating LLM Valley: From AdamW to Memory-Efficient and Matrix-Based Optimizers