Parameter-free dynamic fidelity selection for automated machine learning

作     者：Takenaga, Shintaro Ozaki, Yoshihiko Onishi, Masaki 

作者机构：Univ Tsukuba Tsukuba Ibaraki 3058573 Japan AIST AI Res Ctr Tsukuba Ibaraki 3058561 Japan Preferred Networks Inc Tokyo Tokyo 1000004 Japan 

出 版 物：《APPLIED SOFT COMPUTING》 (Appl. Soft Comput.)

年 卷 期：2025年第171卷

核心收录：

学科分类：08[工学] 0812[工学-计算机科学与技术（可授工学、理学学位）] 

基　　金：New Energy and Industrial Technology Development Organization (NEDO) [JPNP18002] 

主　　题：Automated machine learning Hyperparameter optimization Neural architecture search Evolutionary algorithms Multi-fidelity optimization 

摘      要：Hyperparameter optimization (HPO) and neural architecture search (NAS) have developed dramatically with the evolution of deep neural networks (DNNs). HPO and NAS require DNN training when evaluating a set of hyperparameters and a network architecture, respectively. Therefore, these tasks impose expensive computational time because the objective function is repeatedly evaluated during the search process. To expedite this process, multi-fidelity optimization algorithms have been developed for efficient optimization by exploiting a cheap-to-evaluate low-fidelity objective function. Furthermore, population-based algorithms, such as genetic algorithms and evolution strategies, that leverage parallel processing are also employed to hasten optimization. In this paper, we propose anew algorithm, referred to as parameter-free dynamic fidelity selection (PF-DFS), for efficiently performing HPO and NAS when using the ranking-based evolutionary algorithms. We present an evaluation of the effectiveness of PF-DFS with two prominent ranking-based evolutionary algorithms on 38 multi-fidelity optimization problems of HPO and NAS. Our experimental results demonstrate that PF-DFS accelerates the search speed by 2.5%-24.9% while maintaining the quality of the obtained solutions, as compared to the optimizers without PF-DFS. Furthermore, we demonstrate that CMA-ES with PF-DFS outperforms Hyperband and DEHB (combining differential evolution with Hyperband), widelyused/state-of-the-art multi-fidelity optimization algorithms, in HPO for classification and segmentation tasks of real-world scenarios.