Kevin Yuanbo Wu, Tianxing Zhou, Isaac Tu, Billy Yan, Irmak Guzey, David Fouhey, Dandan Shan, Lerrel Pinto
We collect large-scale human grasping data in daily life and train a flow-matching model to generate diverse human grasps, which are retargeted to robot hands for zero-shot universal grasping.
Multi-fingered robot hands cannot grasp a wide variety of objects as naturally as humans. Existing robot grasping methods are limited to specific objects or environments and fail to capture the distribution of natural human grasps.
Using smart glasses, we collect 1M-HUGs, an egocentric dataset of human grasps spanning 1M frames (27.8 hrs) and 6,707 object instances across 41 buildings. We train a flow-matching model that takes RGB-D images as input and outputs wrist translation, wrist rotation, and MANO hand pose. Predicted grasps can be retargeted to various robot hands. For evaluation, we build HUG-Bench, a simulated benchmark with 90 unseen objects.
HUG outperforms state-of-the-art grasping baselines by 23-34% in success rate and demonstrates zero-shot grasping of diverse objects in real-world robots. We release a large-scale dataset, benchmark, and code to advance universal grasping research.