Observe Then Act: Asynchronous Active Vision-Action Model for Robotic Manipulation

作     者：Wang, Guokang Li, Hang Zhang, Shuyuan Guo, Di Liu, Yanhong Liu, Huaping 

作者机构：Zhengzhou Univ Sch Elect & Informat Engn Zhengzhou 450001 Peoples R China Beijing Univ Posts & Telecommun Sch Artificial Intelligence Beijing 100876 Peoples R China Beijing Univ Posts & Telecommun Sch Comp Sci Beijing 100876 Peoples R China Tsinghua Univ Dept Comp Sci & Technol Beijing 100084 Peoples R China 

出 版 物：《IEEE ROBOTICS AND AUTOMATION LETTERS》 (IEEE Robot. Autom.)

年 卷 期：2025年第10卷第4期

页      面：3422-3429页

核心收录：

学科分类：0808[工学-电气工程] 08[工学] 0811[工学-控制科学与工程] 

基　　金：China's National Natural Science Fund for Key International Collaboration 

主　　题：Grippers Robots Robot sensing systems Three-dimensional displays Visualization Training Trajectory Robot kinematics Cameras Motors Manipulation planning reinforcement learning perception for grasping and manipulation 

摘      要：In real-world scenarios, many robotic manipulation tasks are hindered by occlusions and limited fields of view, posing significant challenges for passive observation-based models that rely on fixed or wrist-mounted cameras. In this letter, we investigate the problem of robotic manipulation under limited visual observation and propose a task-driven asynchronous active vision-action model. Our model serially connects a camera Next-Best-View (NBV) policy with a gripper Next-Best-Pose (NBP) policy, and trains them in a sensor-motor coordination framework using few-shot reinforcement learning. This approach enables the agent to reposition a third-person camera to actively observe the environment based on the task goal, and subsequently determine the appropriate manipulation actions. We trained and evaluated our model on 8 viewpoint-constrained tasks in RLBench. The results demonstrate that our model consistently outperforms baseline algorithms, showcasing its effectiveness in handling visual constraints in manipulation tasks.