Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith et al.
Blink is a benchmark covering 14 visual perception tasks (e.g., relative depth, visual correspondence, forensics detection, multi-view reasoning) that humans solve easily but multimodal LLMs struggle with. GPT-4V and Gemini achieve 51.26% and 45.72% accuracy respectively, far below human performance (95.70%).
Existing multimodal LLM benchmarks focus on high-level reasoning that can be described via language, such as object recognition and scene understanding. However, basic visual perception abilities—like relative depth estimation, visual correspondence, and forgery detection—that humans solve 'within a blink' are not evaluated. These abilities resist linguistic mediation and are essential for verifying whether multimodal LLMs achieve true visual understanding.
The authors selected 14 classic computer vision tasks and converted them into 3,807 multiple-choice questions. Each question includes single or multiple images with visual prompts (e.g., red dots marking locations), designed so that models must 'see' the images to choose the correct answer. 100 human evaluators solved the same questions to establish baseline performance, and state-of-the-art multimodal LLMs (GPT-4V, Gemini, Qwen-VL) were evaluated.
Humans achieved an average accuracy of 95.70%, while the best-performing model (GPT-4V) reached only 51.26%, just 13.17% above random guessing (38.09%). Performance was particularly low on tasks like relative depth, visual correspondence, and forgery detection. In contrast, specialized CV models (e.g., DINOv2, Depth Anything) performed much better on the same tasks, suggesting that multimodal LLMs need integration of CV models or new learning paradigms to acquire visual perception abilities. Blink is released as a public benchmark to stimulate future research.