Towards Accurate and Compact Architectures via Neural Architecture Transformer

作     者：Guo, Yong Zheng, Yin Tan, Mingkui Chen, Qi Li, Zhipeng Chen, Jian Zhao, Peilin Huang, Junzhou 

作者机构：South China Univ Technol Sch Software Engn Guangzhou 510006 Peoples R China Peng Cheng Lab Shenzhen 518066 Peoples R China Tencent Weixin Grp Shenzhen 518054 Peoples R China South China Univ Technol Key Lab Big Data & Intelligent Robot Minist Educ Guangzhou 510006 Peoples R China Tencent Tencent AI Lab Shenzhen 518054 Peoples R China 

出 版 物：《IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE》 (IEEE Trans Pattern Anal Mach Intell)

年 卷 期：2022年第44卷第10期

页      面：6501-6516页

核心收录：

学科分类：0808[工学-电气工程] 08[工学] 0812[工学-计算机科学与技术（可授工学、理学学位）] 

基　　金：Key Realm R&D Program of Guangzhou Key-Area Research and Development Program of Guangdong Province National Natural Science Foundation of China Program for Guangdong Introducing Innovative and Enterpreneurial Teams Tencent AI Lab Rhino-Bird Focused Research Program Fundamental Research Funds for the Central Universities Ministry of Science and Technology Foundation Guangdong Basic and Applied Basic Research Foundation Guangzhou Science and Technology Planning Project Opening Project of Guangdong Key Laboratory of Big Data Analysis and Processing 

主　　题：Architecture optimization neural architecture search compact architecture design operation transition 

摘      要：Designing effective architectures is one of the key factors behind the success of deep neural networks. Existing deep architectures are either manually designed or automatically searched by some Neural Architecture Search (NAS) methods. However, even a well-designed/searched architecture may still contain many nonsignificant or redundant modules/operations (e.g., some intermediate convolution or pooling layers). Such redundancy may not only incur substantial memory consumption and computational cost but also deteriorate the performance. Thus, it is necessary to optimize the operations inside an architecture to improve the performance without introducing extra computational cost. To this end, we have proposed a Neural Architecture Transformer (NAT) method which casts the optimization problem into a Markov Decision Process (MDP) and seeks to replace the redundant operations with more efficient operations, such as skip or null connection. Note that NAT only considers a small number of possible replacements/transitions and thus comes with a limited search space. As a result, such a small search space may hamper the performance of architecture optimization. To address this issue, we propose a Neural Architecture Transformer++ (NAT++) method which further enlarges the set of candidate transitions to improve the performance of architecture optimization. Specifically, we present a two-level transition rule to obtain valid transitions, i.e., allowing operations to have more efficient types (e.g., convolution - separable convolution) or smaller kernel sizes (e.g., 5x5 - 3x3). Note that different operations may have different valid transitions. We further propose a Binary-Masked Softmax (BMSoftmax) layer to omit the possible invalid transitions. Last, based on the MDP formulation, we apply policy gradient to learn an optimal policy, which will be used to infer the optimized architectures. Extensive experiments show that the transformed architectures significantly outperform