Learning to Move Before Learning to Do: Task-Agnostic pretraining for VLAs

TL;DR

Task-Agnostic Pretraining (TAP) is a two-stage framework that dramatically improves data efficiency and robustness in Vision-Language-Action (VLA) models by first learning physical movement skills from cheap, unlabeled data and then aligning them to language with minimal expert demonstrations.

Problem

Vision-Language-Action (VLA) models are bottlenecked by the high cost of collecting large-scale expert demonstrations (observation-instruction-action triplets). Existing approaches conflate two distinct learning objectives—acquiring physical competence (how to move) and semantic alignment (what to do)—leading to data inefficiency.

Approach

Based on a Decomposition Hypothesis, the method separates these objectives. The first stage uses a self-supervised Inverse Dynamics objective to learn transferable motor priors from cheap, unlabeled interaction data, such as discarded off-task trajectories and autonomous robot play. The second stage is a lightweight grounding step that uses minimal expert data to align these priors to language instructions.

Results & Contribution

On the SIMPLER benchmark, TAP matches models trained on over 1M expert trajectories while using orders of magnitude less labeled data, yielding a 10% absolute gain over standard behavior cloning. On a real-world WidowX platform, TAP retains 25% success under camera perturbations where internet-scale baselines collapse to 0%, demonstrating that task-agnostic pretraining produces robust, transferable physical representations and offers a scalable path for Embodied AI.