Rohit Gandikota, David Bau
VLMs develop a small set of attention heads (gaze heads) in their language model backbone that track the image region being described. Manipulating these heads can redirect the description to a desired region without retraining.
How a VLM internally links image regions to language generation is unknown. There is a need for inference-time control of model attention to specific regions without retraining.
Using comic strips as a controlled testbed, we analyze attention head patterns in the VLM's language model backbone. We identify a small set of gaze heads via a simple correlation score from a few forward passes. We then perform a single attention-mask intervention that redirects these heads' attention to a chosen region.
Intervening on the top-100 gaze heads (<9% of all heads) achieves 83.1% accuracy in selecting the correct comic panel. Random head intervention fails, and intervening on all heads destroys generation. Switching the gaze target mid-generation shifts the description within a few tokens. The method works on natural COCO images and generalizes across model sizes (2B–32B) and other VLM architectures. This demonstrates that targeted edits from mechanistic analysis can serve as practical inference-time levers for steering multimodal model behavior without retraining.