Jacky Kwok, Shulu Li, Pranav Atreya, Yuejiang Liu, Yixing Jiang, Chelsea Finn, Marco Pavone, Ion Stoica et al.
A general-purpose verification framework that leverages LLMs' verification ability as a new scaling axis to evaluate and improve the correctness of agentic tasks.
Existing verification methods for agentic tasks, which rely on discrete scoring, lack fine-grained feedback and are difficult to scale for improved verification performance.
LLM-as-a-Verifier computes continuous scores by taking the expectation over the distribution of scoring token logits. This allows verification to scale along dimensions like score granularity, repeated evaluation, and criteria decomposition. A cost-efficient ranking algorithm is also introduced for solution selection.
Achieves state-of-the-art performance on benchmarks like Terminal-Bench V2 and SWE-Bench Verified. Beyond verification, the fine-grained signals serve as a proxy for task progress and as dense feedback for reinforcement learning, improving sample efficiency.