检索结果-内蒙古大学图书馆

Scaling Up Optuna: P2P Distributed hyperparameters optimization

CONCURRENCY AND COMPUTATION-PRACTICE & EXPERIENCE 2025年第4-5期37卷

作者： Cudennec, Loic Minist Armed Forces Ministerial Agcy Def Artificial Intelligence AMIAD Bruz France

In machine learning (ML), hyperparameter optimization (HPO) is the process of choosing a tuple of values that ensures an efficient deployment and training of an AI model. In practice, HPO not only applies to ML tuning but can also be used to tune complex numerical simulations. In this context, a numerical model of a given object is created to be used in realistic simulations. This model is defined by a set of values describing properties such as the geometry of the object or other unknown parameters related to physical quantities. While HPO for ML usually requires finding a few parameters, a numerical model can involve the tuning of more than a hundred parameters. As a consequence, a large number of tuples have to be explored and evaluated before finding a relevant solution, offering new challenges in high-performance computing for efficiently driving the optimization. In this work we rely on the Optuna HPO framework, primarily designed for ML tasks and including state-of-the-art sampling and pruning algorithms. We report on its use to optimize a complex numerical model onto a 1024-core machine. We suggest 1.5M tuples and evaluate 5M simulations using different Optuna-distributed layouts to build several tradeoffs between performance and energy consumption metrics. In order to further scale up the optimization process onto resources, we introduce OptunaP2P, an extension of Optuna based on the peer-to-peer paradigm. This allows to remove any bottleneck in the management of the shared knowledge between optimization processes. With OptunaP2P, we were able to compute up to 3 times faster compared to the regular Optuna-distributed implementation and to obtain close-to-similar results in terms of quality in this reduced time-frame.

关键词： distributed computing energy consumption high-performance computing hyperparameters optimization machine learning operational research optuna peer-to-peer

来源：评论

学校读者我要写书评

暂无评论

hyperparameters optimization of convolutional neural network based on local autonomous competition harmony search algorithm

引用

JOURNAL OF COMPUTATIONAL DESIGN AND ENGINEERING 2023年第4期10卷 1280-1297页

作者： Liu, Dongmei Ouyang, Haibin Li, Steven Zhang, Chunliang Zhan, Zhi-Hui Guangzhou Univ Sch Mech & Elect Engn Guangzhou 510006 Peoples R China South China Univ Technol Sch Comp Sci & Engn Guangzhou 510006 Peoples R China RMIT Univ Grad Sch Business & Law Melbourne 3000 Australia

Because of the good performance of convolutional neural network (CNN), it has been extensively used in many fields, such as image, speech, text, etc. However, it is easily affected by hyperparameters. How to effectively configure hyperparameters at a reasonable time to improve the performance of CNNs has always been a complex problem. To solve this problem, this paper proposes a method to automatically optimize CNN hyperparameters based on the local autonomous competitive harmony search (LACHS) algorithm. To avoid the influence of complicated parameter adjustment of LACHS algorithm on its performance, a parameter dynamic adjustment strategy is adopted, which makes the pitch adjustment probability PAR and step factor BW dynamically adjust according to the actual situation. To strengthen the fine search of neighborhood space and reduce the possibility of falling into local optima for a long time, an autonomous decision-making search strategy based on the optimal state is designed. To help the algorithm jump out of the local fitting situation, this paper proposes a local competition mechanism to make the new sound competes with the worst harmonic progression of local selection. In addition, an evaluation function is proposed, which integrates the training times and recognition accuracy. To achieve the purpose of saving the calculation cost without affecting the search result, it makes the training time for each model depending on the learning rate and batch size. In order to prove the feasibility of LACHS algorithm in configuring CNN superparameters, the classification of the Fashion-MNIST dataset and CIFAR10 dataset is tested. The comparison is made between CNN based on empirical configuration and CNN based on classical algorithms to optimize hyperparameters automatically. The results show that the performance of CNN based on the LACHS algorithm has been improved effectively, so this algorithm has certain advantages in hyperparametric optimization. In addition, this p

关键词： harmony search algorithm convolutional neural network optimization speed hyperparameters optimization

来源：评论

学校读者我要写书评

暂无评论

hyperparameters optimization for Federated Learning System: Speech Emotion Recognition Case Study 8

Hyperparameters Optimization for Federated Learning System: ...

引用

8th IEEE International Conference on Fog and Mobile Edge Computing (FMEC)

作者： Mishchenko, Kateryna Mohammadi, Samaneh Mohammadi, Mohammadreza Sinaei, Sima RISE Res Inst Sweden Stockholm Sweden

ISBN: (纸本)9798350316971

Context: Federated Learning (FL) has emerged as a promising, massively distributed way to train a joint deep model across numerous edge devices, ensuring user data privacy by retaining it on the device. In FL, hyperparameters (HP) significantly affect the training overhead regarding computation and transmission time, computation and transmission load, as well as model accuracy. This paper presents a novel approach where hyperparameters optimization (HPO) is used to optimize the performance of the FL model for Speech Emotion Recognition (SER) application. To solve this problem, both Single-Objective optimization (SOO) and Multi-Objective optimization (MOO) models are developed and evaluated. The optimization model includes two objectives: accuracy and total execution time. Numerical results show that optimal hyperparameters (HP) settings allow for improving both the accuracy of the model and its computation time. The proposed method assists FL system designers in finding optimal parameters setup, allowing them to carry out model design and development efficiently depending on their goals.

关键词： Federated Learning hyperparameters optimization Speech Emotion Recognition

来源：评论

学校读者我要写书评

暂无评论

A new hyperparameters optimization method for convolutional neural networks

引用

PATTERN RECOGNITION LETTERS 2019年 125卷 828-834页

作者： Cui, Hua Bai, Jie Yulin Normal Univ 1303 Jiaoyudong Rd Yulin 537000 Peoples R China Tongji Univ 1239 Siping Rd Shanghai 200092 Peoples R China

The use of convolutional neural networks involves hyperparameters optimization. Gaussian process based Bayesian optimization (GPEI) has proven to be an effective algorithm to optimize several hyperparameters. Then deep networks for global optimization algorithm (DNGO) that used neural network as an alternative to Gaussian process was proposed to optimize more hyperparameters. This paper presents a new algorithm that combines multiscale and multilevel evolutionary optimization (MSMLEO) with GPEI to optimize dozens of hyperparameters. These hyperparameters are divided into two groups. The first group related with the sizes of layers and kernels are discrete integers. The second group related with learning rates and so on is continuous floating-point numbers. All combinations of the first group are corresponding to the combinations of grid points on multi-scale grids and MSMLEO launches GPEI to optimize the second group of hyperparameters while the first group keeps fixed. The output of convolutional networks configured with above two groups of optimized hyperparameters is used as the fitness of MSMLEO. MSMLEO alternates with GPEI to search the optimal hyperparameters from coarsest scale to finest scale. Experimental results show that our algorithm has better performance and adaptability on optimizing dozens of hyperparameters of neural networks with a variety of numerical types. (C) 2019 Published by Elsevier B.V.

关键词： Convolutional neural networks hyperparameters optimization Multilevel evolutionary optimization Bayesian optimization

来源：评论

学校读者我要写书评

暂无评论

Efficient hyperparameters optimization through model-based reinforcement learning with experience exploiting and meta-learning

引用

SOFT COMPUTING 2023年第13期27卷 8661-8678页

作者： Liu, Xiyuan Wu, Jia Chen, Senpeng Univ Elect Sci & Technol China Chengdu Peoples R China

Hyperparameter optimization plays a significant role in the overall performance of machine learning algorithms. However, the computational cost of algorithm evaluation can be extremely high for complex algorithm or large dataset. In this paper, we propose a model-based reinforcement learning with experience variable and meta-learning optimization method to speed up the training process of hyperparameter optimization. Specifically, an RL agent is employed to select hyperparameters and treat the k-fold cross-validation result as a reward signal to update the agent. To guide the agent's policy update, we design an embedding representation called "experience variable" and dynamically update it during the training process. Besides, we employ a predictive model to predict the performance of machine learning algorithm with the selected hyperparameters and limit the model rollout in short horizon to reduce the impact of the inaccuracy of the model. Finally, we use the meta-learning technique to pre-train the model for fast adapting to a new task. To prove the advantages of our method, we conduct experiments on 25 real HPO tasks and the experimental results show that with the limited computational resources, the proposed method outperforms the state-of-the-art Bayesian methods and evolution method.

关键词： hyperparameters optimization Reinforcement learning Meta-learning Deep learning

来源：评论

学校读者我要写书评

暂无评论

Surrogate-Assisted Hybrid-Model Estimation of Distribution Algorithm for Mixed-Variable hyperparameters optimization in Convolutional Neural Networks

引用

IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2023年第5期34卷 2338-2352页

作者： Li, Jian-Yu Zhan, Zhi-Hui Xu, Jin Kwong, Sam Zhang, Jun South China Univ Technol Sch Comp Sci & Engn Guangzhou 510006 Peoples R China Pazhou Lab Guangzhou 510330 Peoples R China South China Univ Technol Guangdong Prov Key Lab Computat Intelligence & Cy Guangzhou 510006 Peoples R China Tencent Inc WeChat Data Qual Team Shenzhen 518052 Peoples R China City Univ Hong Kong Dept Comp Sci Hong Kong Peoples R China Hanyang Univ Ansan 15588 South Korea Zhejiang Normal Univ Jinhua 321004 Zhejiang Peoples R China Chaoyang Univ Technol Taichung 413310 Taiwan

The performance of a convolutional neural network (CNN) heavily depends on its hyperparameters. However, finding a suitable hyperparameters configuration is difficult, challenging, and computationally expensive due to three issues, which are 1) the mixed-variable problem of different types of hyperparameters;2) the large-scale search space of finding optimal hyperparameters;and 3) the expensive computational cost for evaluating candidate hyperparameters configuration. Therefore, this article focuses on these three issues and proposes a novel estimation of distribution algorithm (EDA) for efficient hyperparameters optimization, with three major contributions in the algorithm design. First, a hybrid-model EDA is proposed to efficiently deal with the mixed-variable difficulty. The proposed algorithm uses a mixed-variable encoding scheme to encode the mixed-variable hyperparameters and adopts an adaptive hybrid-model learning (AHL) strategy to efficiently optimize the mixed-variables. Second, an orthogonal initialization (OI) strategy is proposed to efficiently deal with the challenge of large-scale search space. Third, a surrogate-assisted multi-level evaluation (SME) method is proposed to reduce the expensive computational cost. Based on the above, the proposed algorithm is named surrogate-assisted hybrid-model EDA (SHEDA). For experimental studies, the proposed SHEDA is verified on widely used classification benchmark problems, and is compared with various state-of-the-art methods. Moreover, a case study on aortic dissection (AD) diagnosis is carried out to evaluate its performance. Experimental results show that the proposed SHEDA is very effective and efficient for hyperparameters optimization, which can find a satisfactory hyperparameters configuration for the CIFAR10, CIFAR100, and AD diagnosis with only 0.58, 0.97, and 1.18 GPU days, respectively.

关键词： optimization Convolutional neural networks Estimation Computational modeling Brain modeling Probabilistic logic Feature extraction Aortic dissection (AD) diagnosis convolutional neural network (CNN) deep learning estimation of distribution algorithm (EDA) evolutionary computation (EC) hybrid model hyperparameters optimization mixed variable

来源：评论

学校读者我要写书评

暂无评论

Theoretical Aspects in Penalty hyperparameters optimization

引用

MEDITERRANEAN JOURNAL OF MATHEMATICS 2023年第6期20卷 1-13页

作者： Esposito, Flavia Selicato, Laura Sportelli, Caterina Univ Bari Aldo Moro Dept Math Via Orabona 4 I-70125 Bari Italy Univ Western Australia Dept Math & Stat 35 Stirling Highway Crawley WA 6009 Australia

Learning processes play an important role in enhancing understanding and analyzing real phenomena. Most of these methodologies revolve around solving penalized optimization problems. A significant challenge arises in the choice of the penalty hyperparameter, which is typically user -specified or determined through Grid search approaches. There is a lack of automated tuning procedures for the estimation of these hyperparameters, particularly in unsupervised learning scenarios. In this paper, we focus on the unsupervised context and propose a bilevel strategy to address the issue of tuning the penalty hyperparameter. We establish suitable conditions for the existence of a minimizer in an infinite -dimensional Hilbert space, along with presenting some theoretical considerations. These results can be applied in situations where obtaining an exact minimizer is unfeasible. Working on the estimation of the hyperparameter with the gradient-based method, we also introduce a modified version of Ekeland's principle as a stopping criterion for these methods. Our approach distinguishes from conventional techniques by reducing reliance on random or black-box strategies, resulting in stronger mathematical generalization.

关键词： hyperparameters optimization learning approaches existence results

来源：评论

学校读者我要写书评

暂无评论

On the use of Metaheuristics in hyperparameters optimization of Gaussian Processes 19

On the use of Metaheuristics in Hyperparameters Optimization...

引用

Genetic and Evolutionary Computation Conference (GECCO)

作者： Palar, Pramudita Satria Zuhal, Lavi Rizki Shimoyama, Koji Inst Teknol Bandung Fac Mech & Aerosp Engn Bandung Indonesia Tohoku Univ Inst Fluid Sci Sendai Miyagi Japan

ISBN: (纸本)9781450367486

Due to difficulties such as multiple local optima and flat landscape, it is suggested to use global optimization techniques to discover the global optimum of the auxiliary optimization problem of finding good Gaussian Processes (GP) hyperparameters. We investigated the performance of genetic algorithms (GA), particle swarm optimization (PSO), differential evolution (DE), and covariance matrix adaptation evolution strategy (CMA-ES) for optimizing hyperparameters of GP. The study was performed on two artificial problems and also one real-world problem. From the results, we observe that PSO, CMA-ES, and DE/local-to-best/1 consistently outperformed two variants of GA and DE/rand/1 with per-generation-dither on all problems. In particular, CMA-ES is an attractive method since it is quasi-parameter free and it also demonstrates good exploitative and explorative power on optimizing the hyperparameters.

关键词： Gaussian Process Regression hyperparameters optimization Metaheuristics Likelihood function

来源：评论

学校读者我要写书评

暂无评论

A hyperparameters automatic optimization method of time graph convolution network model for traffic prediction

引用

WIRELESS NETWORKS 2021年第7期27卷 4411-4419页

作者： Chen, Lei Bei, Lulu An, Yuan Zhang, Kailiang Cui, Ping Xuzhou Univ Technol Jiangsu Prov Key Lab Intelligent Ind Control Tech Xuzhou 221018 Jiangsu Peoples R China

Smart transportation is an essential component of the smart city. Traffic prediction is an important issue in smart transportation. The convolutional neural networks (GCN) are an effective approach for traffic prediction. However, the GCN meets some challenges, such as stability of prediction precision and computation cost, in traffic prediction. The hyperparameters significantly affect the performance of GCN. We conduct a regression analysis between hyperparameters and GCN performance. Our simulation results show that there is the obvious optimal point of hyperparameters. Some empirical suggestion is given to adjust the hyperparameters based on the simulation results.

关键词： GCN Machine learning hyperparameters optimization Traffic prediction

来源：评论

学校读者我要写书评

暂无评论

MoAR-CNN: Multi-Objective Adversarially Robust Convolutional Neural Network for SAR Image Classification

引用

IEEE TRANSACTIONS ON EMERGING TOPICS IN COMPUTATIONAL INTELLIGENCE 2025年第1期9卷 57-74页

作者： Wei, Hai-Nan Zeng, Guo-Qiang Lu, Kang-Di Geng, Guang-Gang Weng, Jian Jinan Univ Coll Cyber Secur Guangzhou 510632 Peoples R China Jinan Univ Natl Joint Engn Res Ctr Network Secur Detect & Pro Guangzhou 510632 Peoples R China Wenzhou Univ Natl Local Joint Engn Res Ctr Digitalized Elect De Wenzhou 325035 Peoples R China Donghua Univ Coll Informat Sci & Technol Shanghai 201620 Peoples R China Zhejiang Univ Inst Cyber Syst & Control Hangzhou Peoples R China

Deep neural networks (DNNs) have been widely applied to the synthetic aperture radar (SAR) images detection and classification recently while different kinds of adversarial attacks from malicious adversary and the hidden vulnerability of DNNs may lead to serious security threats. The state-of-the-art DNNs-based SAR image detection models are designed manually by only considering the test accuracy performance on clean datasets but neglecting the models' adversarial robustness under various types of adversarial attacks. In order to obtain the best trade-off between the clean accuracy and adversarial robustness in robust convolutional neural networks (CNNs)-based SAR image classification models, this work makes the first attempt to develop a multi-objective adversarially robust CNN, called MoAR-CNN. In the MoAR-CNN, we propose a multi-objective automatic design method of the cells-based neural architectures and some critical hyperparameters such as the optimizer type and learning rate of CNNs. A Squeeze-and-Excitation (SE) layer is introduced after each cell to improve the computational efficiency and robustness. The experiments on FUSAR-Ship and OpenSARShip datasets against seven types of adversarial attacks have been implemented to demonstrate the superiority of the proposed MoAR-CNN to six classical manually designed CNNs and four robust neural architectures search methods in terms of clean accuracy, adversarial accuracy, and model size. Furthermore, we also demonstrate the advantages of using SE layer in MoAR-CNN, the transferability of MoAR-CNN, search costs, adversarial training, and the developed NSGA-II in MoAR-CNN through experiments.

关键词： Synthetic aperture radar (SAR) adversarial robustness convolutional neural networks multi-objective evolutionary algorithms hyperparameters optimization neural architectures search Synthetic aperture radar (SAR) adversarial robustness convolutional neural networks multi-objective evolutionary algorithms hyperparameters optimization neural architectures search

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：