Timing Yang, Predrag Neskovic, Jansen Seheult, Wenchao Han, Anand Bhattad, Alan Yuille, Feng Wang
RATS introduces register tokens and a three-step attention mechanism, allowing each register to spontaneously learn meaningful object parts without supervision.
Existing vision transformers use a single classification token to represent the entire image, failing to learn part-level structure and lacking interpretability. A method is needed to autonomously discover compositional object parts.
Replace the classification token with N learnable register tokens, routing patch information through an L→N→N→L bottleneck via compress-communicate-broadcast attention. The N registers are partitioned across H attention heads, so registers from different heads do not interact. Without auxiliary losses or part annotations, each register specializes into a proto-semantic region.
RATS surpasses all baselines by +12 mIoU on average across five segmentation benchmarks, with consistent gains on ADE20K (+1.11 mIoU) and COCO (+0.2 AP^m). Its register dictionary exhibits part-level consistency and semantic proximity across related categories, providing a useful architectural prior for structured and interpretable visual representation learning.