TL;DR
This paper proposes ToolCAD, a framework that trains LLMs as tool-using agents to automatically generate professional 3D CAD models from text descriptions using reinforcement learning.
Problem
CAD modeling is an expert task requiring long-horizon reasoning and coherent modeling actions. There has been a lack of research on how tool-using LLMs can optimally interact with CAD engines, hindering the development of LLM-based agentic text-to-CAD systems.
Approach
ToolCAD Framework: Proposes a novel agentic framework deploying LLMs as tool-using agents that interact with CAD engines.
Interactive CAD Modeling Environment: Builds a simulation environment to rollout reasoning and tool-augmented interaction trajectories with the CAD engine, incorporating hybrid feedback and human supervision.
RL-based Training Strategy: Presents an end-to-end post-training strategy using online curriculum reinforcement learning to enable the LLM agent to elicit refined CAD Modeling Chain of Thought (CAD-CoT) and evolve into proficient CAD tool-using agents.
Results & Contribution
Fills the gap in adopting and training open-source LLMs for CAD tool-using agents.
Demonstrates that open-source LLMs can perform comparably to proprietary models using the proposed method.
Paves the way for more accessible and robust autonomous text-to-CAD modeling systems.