RAPID: Robot Agentic Programming from Demonstrations
RAPID：只需單次視覺示範，AI 代理即可自動生成與優化機器人操控程式
Traditional robot programming is labor-intensive and lacks flexibility. RAPID bypasses this by learning from a single visual demonstration. It automatically extracts task specifications, action primitives, and a matching simulation environment. It then deploys an agentic loop to write, test, and iteratively refine control programs. By representing movements through object-centric geometric constraints rather than hardcoded paths, RAPID successfully generalized to various shapes, materials, and poses across simulations and a physical Franka robot arm.
Key points
One-Shot Inference
Automatically extracts testable task specifications, action primitives, and interactive simulation setups from just one visual demonstration.
Agentic Code Refinement
Employs an iterative LLM coding agent loop to execute, verify, and automatically debug robot programs in simulated trials.
Object-Centric Generalization
Focuses on high-level geometric relationships and motion effects rather than raw trajectories, allowing seamless adaptation to new object shapes and poses.
How it works
Why it matters
This research bridges the gap between LLM program synthesis and physical control. By framing skills as structured code rather than static trajectories, RAPID demonstrates how robots can autonomously acquire complex, contact-rich manipulation skills (like pushing and sliding) and safely deploy them to real-world hardware, drastically reducing manual engineering overhead.
Who it affects
- AI Developer
- AI Researcher
- Student & Learner
How to use it
- 1Contact-rich nonprehensile manipulation, such as pushing, sliding, or flipping irregular objects.
- 2Cross-scenario skill transfer, executing tasks successfully even when object shapes, materials, or initial poses change.
Limitations & caveats
- Highly dependent on the quality of the visual demonstration; severe physical occlusions can cause incorrect task spec or relationship inferences.
- The sim-to-real gap between the generated simulation and physical setup can still affect the precision of highly delicate contact dynamics.
Related
AD-WM: Action-Discriminative World Models for Counterfactual MPC
AD-WM：專為反事實預測控制設計的動作辨識世界模型
AD-WM enhances latent world models by learning to distinguish alternative actions from the same state, significantly improving counterfactual control and zero-shot transfer in robotics.
Rolling-WAM: Accelerating Robotic World Action Models via Rolling Imagination
Rolling-WAM：利用滾動想像實現高效控制的機器人世界動作模型
Rolling-WAM accelerates world action models by distributing the joint video-action denoising process across sliding windows over successive replanning cycles, achieving a 4.5x speedup.
Coding Agents Outperform Hand-Engineered Planners in Task and Motion Planning
程式碼生成 Agent 突破機器人規劃瓶頸：自動編寫高泛化性的 TAMP 解決方案
Researchers demonstrate that coding agents can interactively synthesize highly generalizable programs for task and motion planning (TAMP), achieving up to 95% success and outperforming hand-engineered baselines.