Relaxed Actor-Critic With Convergence Guarantees for Continuous-Time Optimal Control of Nonlinear Systems

作     者：Duan, Jingliang Li, Jie Ge, Qiang Li, Shengbo Eben Bujarbaruah, Monimoy Ma, Fei Zhang, Dezhao 

作者机构：Univ Sci & Technol Beijing Sch Mech Engn Beijing 100083 Peoples R China Tsinghua Univ Sch Vehicle & Mobil Beijing 100084 Peoples R China Univ Calif Berkeley Dept Mech Engn Berkeley CA 94720 USA Beijing Idriverplus Technol Co Ltd Beijing 100192 Peoples R China 

出 版 物：《IEEE TRANSACTIONS ON INTELLIGENT VEHICLES》 (IEEE Trans. Intell. Veh.)

年 卷 期：2023年第8卷第5期

页      面：3299-3311页

核心收录：

学科分类：0808[工学-电气工程] 08[工学] 0823[工学-交通运输工程] 0812[工学-计算机科学与技术（可授工学、理学学位）] 

基　　金：N S F China State Key Laboratory of Automotive Safety and Energy, China under Grant [KFY2212] Tsinghua University Initiative Scientific Research Program 

主　　题：Heuristic algorithms Convergence Vehicle dynamics Nonlinear dynamical systems Mathematical models Approximation algorithms Infinite horizon Reinforcement learning continuous-time optimal control nonlinear systems 

摘      要：This paper presents the Relaxed Continuous-Time Actor-critic (RCTAC) algorithm, a method for finding the nearly optimal policy for nonlinear continuous-time (CT) systems with known dynamics and infinite horizon, such as the path-tracking control of vehicles. RCTAC has several advantages over existing adaptive dynamic programming algorithms for CT systems. It does not require the admissibility of the initialized policy or the input-affine nature of controlled systems for convergence. Instead, given any initial policy, RCTAC can converge to an admissible, and subsequently nearly optimal policy for a general nonlinear system with a saturated controller. RCTAC consists of two phases: a warm-up phase and a generalized policy iteration phase. The warm-up phase minimizes the square of the Hamiltonian to achieve admissibility, while the generalized policy iteration phase relaxes the update termination conditions for faster convergence. The convergence and optimality of the algorithm are proven through Lyapunov analysis, and its effectiveness is demonstrated through simulations and real-world path-tracking tasks.