An empirical evaluation of a hierarchical reinforcement learning method towards modular robot control

作     者：Takeda, Sho Yamamori, Satoshi Yagi, Satoshi Morimoto, Jun 

作者机构：Kyoto Univ Grad Sch Informat Kyoto Japan ATR Dept Brain Robot Interface Computat Neurosci Labs Kyoto Japan 

出 版 物：《ARTIFICIAL LIFE AND ROBOTICS》 (Artif. Life Rob.)

年 卷 期：2025年第30卷第2期

页      面：245-251页

核心收录：

学科分类：12[管理学] 1201[管理学-管理科学与工程(可授管理学、工学学位)] 08[工学] 0811[工学-控制科学与工程] 

基　　金：JST Moonshot RD program [JPMJMS223B-3, JPNP20006] NEDO JSPS KAKENHI [22H03669] 

主　　题：Hierarchical reinforcement learning Modular reinforcement learning Multitask manipulation Modular robot 

摘      要：There is a growing expectation that deep reinforcement learning will enable multi-degree-of-freedom robots to acquire policies suitable for real-world applications. However, a robot system with a variety of components requires many learning trials for each different combination of robot modules. In this study, we propose a hierarchical policy design to segment tasks according to different robot components. The tasks of the multi-module robot are performed by skill sets trained on a component-by-component basis. In our learning approach, each module learns reusable skills, which are then integrated to control the whole robotic system. By adopting component-based learning and reusing previously acquired policies, we transform the action space from continuous to discrete. This transformation reduces the complexity of exploration across the entire robotic system. We validated our proposed method by applying it to a valve rotation task using a combination of a robotic arm and a robotic gripper. Evaluation based on physical simulations showed that hierarchical policy construction improved sample efficiency, achieving performance comparable to the baseline with 46.3% fewer samples.