LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code

TL;DR

LiveCodeBench is a benchmark that continuously collects problems from competitive coding platforms to provide contamination-free, holistic evaluation of LLMs' code capabilities.

Problem

Existing code benchmarks (HumanEval, MBPP) use static data, increasing contamination risk as new LLMs emerge, and fail to evaluate diverse capabilities beyond code generation (e.g., self-repair, execution, test prediction).

Approach

Collect 400 high-quality problems from LeetCode, AtCoder, and CodeForces contests between May 2023 and May 2024. Evaluate four scenarios: code generation, self-repair, code execution, and test output prediction. Release all prompts and model completions.

Results & Contribution

Evaluate 18 base LLMs and 34 instruction-tuned LLMs, presenting contamination analysis, holistic performance comparisons, and potential overfitting in existing benchmarks. Continuous updates and open-source toolkit support community expansion and reproducibility.