TL;DR
A zero-shot framework for long-horizon dexterous manipulation using multi-view RGB images and VLM to generate 3D task plans.
Problem
Existing robot manipulation methods rely on training data or single-view inputs, lacking 3D understanding and generalization for long-horizon tasks.
Approach
Extract 2D keypoints and task grounding from multi-view RGB images using VLM
Lift to 3D via triangulation and reference-view ray voting
Generate motions for pick-and-place and tool-use using 3D keypoints
Support long-horizon tasks with closed-loop status verification and replanning
Results & Contribution
Improved 3D grounding accuracy and execution reliability over single-view RGB-D methods
Outperforms fine-tuned VLA baselines
Enables zero-shot execution on novel objects and tool-use tasks