Researchers evaluated whether LLM agents can compete with classical HPO algorithms like CMA-ES and TPE. Classical methods outperformed LLMs in fixed search spaces, but allowing code editing narrowed the gap. The hybrid Centaur method combined strengths of both to achieve the best results.
A study systematically compared LLM-based agents against classical hyperparameter optimization (HPO) algorithms like CMA-ES and TPE. Using the autoresearch repository, LLMs were allowed to edit training code directly to tune hyperparameters.
HPO is critical for ML model performance, with methods ranging from grid search to Bayesian optimization. Recent advances in LLM code generation have sparked interest in whether LLMs can replace traditional HPO.
Classical methods still dominate in fixed search spaces, but LLMs show promise when allowed to flexibly edit code. The hybrid Centaur approach achieved the best results, suggesting LLMs are best used as complements to classical optimizers rather than replacements.
Comments agree with the paper's core claim that combining LLM and classical hyperparameter optimization is better than using either alone, citing real research cases (e.g., AI-PROPELLER) and personal experiences. Some users shared findings where LLMs outperformed classical optimizers in specific budget or HPC code tuning domains, while others noted cases where LLMs failed to suggest optimal algorithms due to training data limitations. Overall, interest in the 'Centaur approach' is high, with discussions on specific tools (e.g., pycma) and methodologies for implementation.