Natural-Language Temporal Grounding in Hour-Long Videos is a Search Problem: A Benchmark and Empirical Decomposition

TL;DR

For hour-long videos, temporal grounding is bottlenecked by search, not recognition; a retrieve-then-ground hybrid significantly outperforms monolithic Video-LLMs.

Problem

Existing temporal grounding research focuses on short videos (minutes), leaving hour-scale natural-language grounding unexplored. In long videos, query-relevant events are scattered, causing Video-LLMs to fail without explicit search.

Approach

The authors frame hour-scale temporal grounding as a search problem and construct ExtremeWhenBench, the first open benchmark for hour-scale grounding, with 2,273 queries over 194 videos (mean 75.7 min, max 9 hr). They compare monolithic Video-LLMs, a frame-level retrieval baseline, and a retrieve-then-ground hybrid, and analyze failure causes via a taxonomy.

Results & Contribution

All open Video-LLMs collapse on long videos, while a frame-level retrieval baseline outperforms them. A failure taxonomy attributes 85% of failures to search; the retrieve-then-ground hybrid achieves a 6.7x improvement over monolithic Video-LLMs, mirroring the retrieve-then-read paradigm in open-domain QA and highlighting the importance of search in long-video understanding.