Learning What to Say to Your VLA: Mostly Harmless Vision Language Action Model Steering

TL;DR

A framework that optimizes and safely calibrates language commands for VLA models, significantly improving performance without modifying the original model.

Problem

VLA models are sensitive to language commands, exhibiting different behaviors for similar instructions, and desired capabilities may not be elicitable through prompting alone. Human instructions or zero-shot language models often fail to reliably steer VLAs toward successful task execution.

Approach

Results & Contribution

Achieves 24.7% improvement in simulation and 65.0% on hardware, with harmlessness guarantees under visual and semantic perturbations. Also produces recovery behaviors not observed with open-loop prompting.