Cooperative multi-agent actor-critic control of traffic network flow based on edge computing

作     者：Zhang, Yongnan Zhou, Yonghua Lu, Huapu Fujita, Hamido 

作者机构：Beijing Jiaotong Univ Sch Elect & Informat Engn Beijing Peoples R China Tsinghua Univ Inst Transportat Engn Beijing Peoples R China Iwate Prefectural Univ Fac Software & Informat Sci Takizawa Iwate Japan Univ Granada Andalusian Res Inst Data Sci & Computat Intellige Granada Spain 

出 版 物：《FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE》 (下代计算机系统)

年 卷 期：2021年第123卷

页      面：128-141页

核心收录：

学科分类：08[工学] 0812[工学-计算机科学与技术（可授工学、理学学位）] 

基　　金：Beijing Natural Science Foundation, China [L191017] National Natural Science Foundation of China Science and Technology Research Program of Beijing, China [Z161100001116093] 

主　　题：Distributed deep reinforcement learning Edge computing Traffic network flow control Cooperative multi-agent actor-critic framework 

摘      要：Most of the existing traffic signal control strategies are hard to satisfy the real-time requirements of traffic big data analysis, knowledge reasoning and decision making for sophisticated traffic dynamics and heterogeneous intersection structures in the context of Internet of Vehicles (IoV). In this paper, we attempt to propose a cooperative multi-agent actor-critic (CMAC) deep reinforcement learning (DRL) approach with value decomposition based on edge computing architecture. The intuition behind CMAC is to decompose the global actor-critic learning tasks into several local actor-critic sub-problems with respect to each intersection. Each agent searches the local optimal decision by actor-critic network that takes the discrete state encoding about several consecutive frames of image-like traffic states as the inputs of the network. Among them, the green ratio output strategy considering multiple constraints is formulated in the output layer of the actor network, so that the continuous control of traffic signals using multi-agent DRL (MADRL) can be realized. Furthermore, a cooperative mechanism that considers contribution weight distributions of local agents to the global traffic pattern is proposed to coordinate multiple local agents to evolve toward global optimization. Especially, some parallel training tasks of CMAC with a large number of computing loads are deployed on the cloud side in the edge computing architecture to accelerate learning and reconstructing knowledge. The well-trained multi-agent model is downloaded from the cloud side into the edge side for real-time decision making of traffic network flow adaptive control. Simulation results with regard to a realistic traffic network demonstrate that the proposed CMAC approach under edge computing architecture outperforms the value-decomposition based multi-agent actor-critic (VMAC), independent multi-agent actor-critic (IMAC), and the fixed timing control (FTC) in terms of alleviating traffic congestion.