Adaptive temporal-difference learning via deep neural network function approximation: a non-asymptotic analysis

作     者：Wang, Guoyong Fu, Tiange Zheng, Ruijuan Zhao, Xuhui Zhu, Junlong Zhang, Mingchuan 

作者机构：Luoyang Inst Sci & Technol Sch Informat Engn Luoyang 471023 Peoples R China Longmen Lab Luoyang 471023 Peoples R China Henan Univ Sci & Technol Sch Informat Engn Luoyang 471023 Peoples R China 

出 版 物：《COMPLEX & INTELLIGENT SYSTEMS》 (Complex Intell. Syst.)

年 卷 期：2025年第11卷第2期

页      面：1-19页

核心收录：

基　　金：National Natural Science Foundation of China [62176113, 62172142] Science and Technology Development Plan of Henan Province of China [231100220600, 231111222600] Scientific and Tech- nological Innovation Teams of Colleges and Universities in Henan Province of China [24IRTSTHN022] 

主　　题：Adaptive methods Non-asymptotic convergence Nonlinear function approximation Reinforcement learning Temporal-difference learning 

摘      要：Although deep reinforcement learning has achieved notable practical achievements, its theoretical foundations have been scarcely explored until recent times. Nonetheless, the rate of convergence for current neural temporal-difference (TD) learning algorithms is constrained, largely due to their high sensitivity to stepsize choices. In order to mitigate this issue, we propose an adaptive neural TD algorithm (AdaBNTD) inspired by the superior performance of adaptive gradient techniques in training deep neural networks. Simultaneously, we derive non-asymptotic bounds for AdaBNTD within the Markovian observation framework. In particular, AdaBNTD is capable of converging to the global optimum of the mean square projection Bellman error (MSPBE) with a convergence rate of O(1/K)\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\mathcal {O}}}(1/\sqrt{K})$$\end{document}, where K denotes the iteration count. Besides, the effectiveness AdaBNTD is also verified through several reinforcement learning benchmark domains.