Zhao-Heng Yin, Guanya Shi, Pieter Abbeel, C. Karen Liu
A framework that introduces keyframe concepts from computer animation for dexterous manipulation of articulated tools. It procedurally generates grasp poses in simulation and fine-tunes them with reinforcement learning, enabling a real robot hand to zero-shot handle various articulated tools.
Existing robot manipulation research has largely focused on rigid objects, while manipulation of articulated tools (e.g., scissors, pliers, spray guns) has been rarely addressed due to complex internal degrees of freedom and contact dynamics. In particular, learning functional grasps and in-hand manipulation simultaneously is a very difficult problem.
Mana consists of two main stages. First, procedural generation creates grasp keyframes tailored to the functional affordances of the tool (e.g., blades and handles of scissors) with just a few mouse clicks. Second, using these keyframes as starting points, a pipeline combining motion planning and reinforcement learning (RL) learns smooth manipulation trajectories. The entire data generation process is largely automated, requiring less than one minute of manual work per tool.
For four articulated tools (scissors, pliers, spray gun, folding knife), policies learned in simulation were zero-shot transferred to a real robot hand (e.g., Allegro Hand), demonstrating successful grasping and in-hand manipulation. This is the first sim-to-real success in articulated tool manipulation, providing a practical framework that combines scalable data generation with robust policy learning.