Video
DexGanGrasp is a model that uses Conditional Generative Adversarial Networks (cGANs) to generate dexterous grasps (via a DexGenerator) and assess their stability (via a DexEvaluator). The method was tested through extensive simulations and real-world experiments, showing an 18.57% higher success rate compared to the baseline FFHNet.
Abstract
We introduce DexGANGrasp, a dexterous grasping synthesis method that generates and evaluates grasps with single view in real time. DexGanGrasp comprises a Conditional Generative Adversarial Networks (cGANs)-based DexGenerator to generate dexterous grasps and a discriminator-like DexEvalautor to assess the stability of these grasps. Extensive simulation and real-world expriments showcases the effectiveness of our proposed method, outperforming the baseline FFHNet with an 18.57% higher success rate in real-world evaluation. We further extend DexGanGrasp to DexAfford-Prompt, an openvocabulary affordance grounding pipeline for dexterous grasping leveraging Multimodal Large Language Models (MLLMs) and Vision Language Models (VLMs), to achieve task-oriented grasping with successful real-world deployments.
Model
DexGANGrasp Model Architecture
DexAfford-Prompt Pipeline
Results
Grasping in simulation with the Robotiq-3F gripper
Grasping with DexGANGrasp
Task Oriented Grasping with DexAfford-Prompt
Summary
- We propose a real-time dexterous grasp synthesis and evaluation pipeline using conditional GANs for unknown objects with a dexterous hand from a single view.
- New synthetic grasping dataset for Robotiq-3F hand: 115 objects, 2.1 million grasps, 5750 different scenes.
- Extension of DexGanGrasp to DexAfford-Prompt, a task-oriented grasping pipeline leveraging MLLMs and VLMs for open-vocabulary affordance grounding.
- Extensive experiments in simulation and real-world settings, with a detailed ablation study, showing our method outperforms the cVAE baseline and is effective in DexAfford-Prompt for task-oriented grasping with user-initiated affordance.
BibTeX
@misc{feng2024dexgangraspdexterousgenerativeadversarial,
title={DexGANGrasp: Dexterous Generative Adversarial Grasping Synthesis for Task-Oriented Manipulation},
author={Qian Feng and David S. Martinez Lema and Mohammadhossein Malmir and Hang Li and Jianxiang Feng and Zhaopeng Chen and Alois Knoll},
year={2024},
eprint={2407.17348},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2407.17348},
}