检索结果-内蒙古大学图书馆

Leader population learning rate schedule

INFORMATION SCIENCES 2023年第1期623卷 455-468页

作者： Wei, Jia Zhang, Xingjun Zhuo, Zhimin Ji, Zeyu Wei, Zheng Li, Jingbo Li, Qianyang Xi An Jiao Tong Univ Sch Comp Sci & Technol Xian 710049 Peoples R China Beijing Inst Elect Syst Engn Beijing 100854 Peoples R China

The successful application of modern deep neural network models is deeply influenced by the choice of hyperparameters. As one of the most important hyperparameters, the Learning Rate (LR) needs to be fine-tuned. In order to train models that can perform better, finding a suitable LR schedule quickly has become a pressing problem. Current research often relies on manual use of parallel search methods such as grid search, or serial methods such as Bayesian. However, the variety and number of models and datasets result in a very large LR space, and these methods consume significant resources to find only sub-optimal static LR schedules within a given time frame. It was found that in the currently widely used Distributed Data Parallel (DDP) deep learning, where multiple nodes themselves con-stitute the population, it is possible to use population algorithms to optimise the searched LR schedule dynamically throughout the training process with little additional training time. To maximise the potential of both the searched LR schedule and the population opti-misation of the participating multi-nodes, this paper proposes a Leader population Learning Rate Schedule (LPLRS) for the DDP deep learning environment, which continu-ously explores for better learning rates in the neighbourhood of the searched LR schedule and guides the subsequent training process. LPLRS achieves higher model classification accuracy on the Cifar 10&100 test datasets on the state-of-the-art Wide Residual Network with Sharpness-Aware Minimization (WRN(SAM)) model compared to the latest SGDR, CLR, and StepLR learning rate schedules.(c) 2022 Elsevier Inc. All rights reserved.

关键词： Deep learning Distributed training Hyperparameter search Data parallel population algorithm Heuristic algorithms

来源：评论

学校读者我要写书评

暂无评论

PANDA: population Automatic Neural Distributed algorithm for Deep Leaning 19

PANDA: Population Automatic Neural Distributed Algorithm for...

引用

19th IEEE International Symposium on Parallel and Distributed Processing with Applications (IEEE ISPA)

作者： Wei, Jia Zhang, Xingjun Ji, Zeyu Li, Jingbo Wei, Zheng Xi An Jiao Tong Univ High Peiformance Comp Lab Xian Peoples R China

ISBN: (纸本)9781665435741

Deep neural network models perform very brightly in the field of artificial intelligence, but their success is affected by hyperparameters, and the learning rate schedule is one of the most important hyperparameters, and the search for the learning rate schedule is often time-consuming and computationally resource-intensive. In this paper, we propose a population Automatic Neural Distributed algorithm (PANDA) based on population joint optimization, which uses distributed data parallel deep neural network training to implement a dynamic learning rate schedule optimization strategy based on the population idea, with almost no loss of test accuracy, PANDA is able to dynamically refine the learning rate schedule during model training instead of following the usual suboptimal strategy. We conducted experiments on typical AlexNet, VGG16, and ResNet18 using the Tianhe-3 supercomputing prototype, and the results show that using PANDA to dynamically update the learning rate greatly reduces the learning rate schedule search time while ensuring close performance with the latest population hyperparameter algorithm, and in our experiments, PANDA lead to at max 123.85x speedup, and the experimental results prove the effectiveness and robustness of PANDA.

关键词： Deep Learning Distributed Training Hyperparameter Search Data Parallel population algorithm Learning Rate Search

来源：评论

学校读者我要写书评

暂无评论

DPLRS: Distributed population Learning Rate Schedule

引用

FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE 2022年 132卷 40-50页

作者： Wei, Jia Zhang, Xingjun Ji, Zeyu Wei, Zheng Li, Jingbo Xi An Jiao Tong Univ Sch Comp Sci & Technol Xian 710049 Peoples R China

Deep neural network models perform very brightly in the field of artificial intelligence, but their success is affected by hyperparameters, and the learning rate schedule is one of the most important hyperparameters, while the search for the learning rate schedule is often time-consuming and compu-tationally resource-intensive. In this paper, we proposed Distributed population Learning Rate Schedule (DPLRS) based on population joint optimization, which uses distributed data parallel deep neural network training to implement a dynamic learning rate schedule optimization strategy based on the population idea, with almost no loss of test accuracy. DPLRS is able to dynamically refine the learning rate schedule during model training instead of following the usual suboptimal strategy. We conducted experiments on typical AlexNet, VGG16, and ResNet18 using the Tianhe-3 supercomputing prototype. The results illustrate that using DPLRS to dynamically update the learning rate can greatly reduce the searching time of the learning rate schedule and meanwhile, can ensure the close performance with the latest population hyperparameter algorithm. Also, In our experiments, DPLRS lead to 123.85x speedup maximum, which prove the effectiveness and robustness of DPLRS. (c) 2022 Elsevier B.V. All rights reserved.

关键词： Deep learning Distributed training Hyperparameter search Data parallel population algorithm

来源：评论

学校读者我要写书评

暂无评论

Prediction of Sinter Quality Based on population Genetic algorithms and Fuzzy Reasoning

Prediction of Sinter Quality Based on Population Genetic Alg...

引用

2019 2nd International Conference on Computer Science and Advanced Materials(CSAM 2019)

作者： Liu Guangyue Hu Qinghe Zhang Shuang College of Information Science and Engineering Northeastern University

In view of the characteristics of non-linearity, strong coupling and large time delay in sintering process, the sintering process was analyzed from the point of view of process parameter control, and the evaluation index of sinter performance and its main influencing parameters were determined. On this basis, a sinter quality prediction method based on population genetic algorithm and fuzzy reasoning was proposed. Firstly, the pheromone-based ant colony algorithm is used to generate an optimal solution as the initial population of genetic algorithm, which is conducive to improving the convergence performance. Then, by making full use of random numbers and increasing the number of cycles, the roulette-based selection algorithm in traditional genetic algorithm is improved, so as to ensure the diversity of the next generation population and improve the chance of the best chromosome being selected. The method is applied to the prediction of sinter quality. The experimental results show that this method can predict the iron content of ore by quantifying the sinter quality, which has a positive effect on the beneficiation and output prediction of related enterprises.

关键词： population algorithm genetic algorithm fuzzy reasoning sinter quality prediction

来源：评论

学校读者我要写书评

暂无评论

A New Way of Decomposing Search Domain in a Global Optimization Problem 2nd

A New Way of Decomposing Search Domain in a Global Optimizat...

引用

2nd International Scientific Conference on Intelligent Information Technologies for Industry (IITI)

作者： Sakharov, Maxim Karpenko, Anatoly Bauman MSTU Moscow Russia

ISBN: (纸本)9783319683218

This paper deals with a new method for decomposing search domain in a global optimization problem. Proposed method was designed for parallel population algorithms but also can be used as a diversification tool in sequential algorithms. New decomposition technique was compared with a traditional approach by means of numeric experiments with a use of multi-dimensional benchmark optimization functions and Mind Evolutionary Computation algorithm. Results of the experiments demonstrate the superiority of new technique over a canonical approach which resulted in a higher quality of obtained solutions.

关键词： Global optimization population algorithm Problem decomposition Mind evolutionary computation

来源：评论

学校读者我要写书评

暂无评论

Path relinking for the fixed spectrum frequency assignment problem

引用

EXPERT SYSTEMS WITH APPLICATIONS 2015年第10期42卷 4755-4767页

作者： Lai, Xiangjing Hao, Jin-Kao Univ Angers LERIA F-49045 Angers 01 France

The fixed spectrum frequency assignment problem (FS-FAP) is a highly relevant application in modern wireless systems. This paper presents the first path relinking (PR) approach for solving FS-FAP. We devise four relinking operators to generate intermediate solutions (or paths) and a tabu search procedure for local optimization. We also adopt a diversity-and-quality technique to maintain population diversity. To show the effectiveness of the proposed approach, we present computational results on the set of 42 benchmark instances commonly used in the literature and compare them with the current best results obtained by any other existing methods. By showing improved best results (new upper bounds) for 19 instances, we demonstrate the effectiveness of the proposed PR approach. We investigate the impact of the relinking operators and the population updating strategy. The ideas of the proposed could be applicable to other frequency assignment problems and search problems. (C) 2015 Elsevier Ltd. All rights reserved.

关键词： Path relinking population algorithm Heuristics Frequency assignment

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：