检索结果-内蒙古大学图书馆

AutoLRS: Automatic learning-rate schedule by bayesian optimization on the Fly

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Jin, Yuchen Zhou, Tianyi Zhao, Liangyu Zhu, Yibo Guo, Chuanxiong Canini, Marco Krishnamurthy, Arvind University of Washington United States ByteDance Inc. KAUST

The learning rate (LR) schedule is one of the most important hyper-parameters needing careful tuning in training DNNs. However, it is also one of the least automated parts of machine learning systems and usually costs significant manual effort and computing. Though there are pre-defined LR schedules and optimizers with adaptive LR, they introduce new hyperparameters that need to be tuned separately for different tasks/datasets. In this paper, we consider the question: Can we automatically tune the LR over the course of training without human involvement? We propose an efficient method, AutoLRS, which automatically optimizes the LR for each training stage by modeling training dynamics. AutoLRS aims to find an LR applied to every τ steps that minimizes the resulted validation loss. We solve this black-box optimization on the fly by Bayesian optimization (BO). However, collecting training instances for BO requires a system to evaluate each LR queried by BO's acquisition function for τ steps, which is prohibitively expensive in practice. Instead, we apply each candidate LR for only τ ≤ τ τ steps and train an exponential model to predict the validation loss after τ steps. This mutual-training process between BO and the loss-prediction model allows us to limit the training steps invested in the BO search. We demonstrate the advantages and the generality of AutoLRS through extensive experiments of training DNNs for tasks from diverse domains using different optimizers. The LR schedules auto-generated by AutoLRS lead to a speedup of 1:22×, 1:43×, and 1:5× when training ResNet-50, Transformer, and BERT, respectively, compared to the LR schedules in their original papers, and an average speedup of 1:31× over state-of-the-art heavily-tuned LR schedules. © 2021, CC BY.

关键词： learning algorithms

Contextual Gradient Scaling for Few-Shot learning

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Lee, Sanghyuk Lee, Seunghyun Song, Byung Cheol Department of Electrical and Computer Engineering Inha University

Model-agnostic meta-learning (MAML) is a well-known optimization-based meta-learning algorithm that works well in various computer vision tasks, e.g., few-shot classification. MAML is to learn an initialization so that a model can adapt to a new task in a few steps. However, since the gradient norm of a classifier (head) is much bigger than those of backbone layers, the model focuses on learning the decision boundary of the classifier with similar representations. Furthermore, gradient norms of high-level layers are small than those of the other layers. So, the backbone of MAML usually learns task-generic features, which results in deteriorated adaptation performance in the inner-loop. To resolve or mitigate this problem, we propose contextual gradient scaling (CxGrad), which scales gradient norms of the backbone to facilitate learning task-specific knowledge in the inner-loop. Since the scaling factors are generated from task-conditioned parameters, gradient norms of the backbone can be scaled in a task-wise fashion. Experimental results show that CxGrad effectively encourages the backbone to learn task-specific knowledge in the inner-loop and improves the performance of MAML up to a significant margin in both same- and cross-domain few-shot classification. Copyright © 2021, The Authors. All rights reserved.

关键词： learning algorithms

learning to acquire novel cognitive tasks with evolution, plasticity and meta-meta-learning

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Miconi, Thomas ML Collective

A hallmark of intelligence is the ability to autonomously learn new flexible, cognitive behaviors - that is, behaviors where the appropriate action depends not just on immediate stimuli (as in simple reflexive stimulus-response associations), but on contextual information that must be adequately acquired, stored and processed. While many meta-learning algorithms can design agents that autonomously learn new tasks, cognitive tasks adds another level of learning and memory to typical "learning-to-learn" problems. Here we evolve neural networks, endowed with plastic connections and neuromodulation, over a sizable set of simple cognitive tasks adapted from a computational neuroscience framework. The resulting evolved networks can automatically modify their own connectivity to acquire a novel simple cognitive task, never seen during evolution, from stimuli and rewards alone, through the spontaneous operation of their evolved neural organization and plasticity system. Our results emphasize the importance of carefully considering the multiple learning loops involved in the emergence of intelligent behavior. © 2021, CC BY.

关键词： learning algorithms

Hierarchical learning using deep optimum-path forest

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Afonso, Luis C.S. Pereira, Clayton R. Weber, Silke A.T. Hook, Christian Falcão, Alexandre X. Papa, João P. UFSCar - Federal University of São Carlos Department of Computing São Carlos Brazil UNESP - São Paulo State University School of Sciences Bauru Brazil UNESP - São Paulo State University Medical School Botucatu Brazil Ostbayerische Technische Hochschule Regensburg Germany UNICAMP - University of Campinas Institute of Computing Campinas Brazil

Bag-of-Visual Words (BoVW) and deep learning techniques have been widely used in several domains, which include computer-assisted medical diagnoses. In this work, we are interested in developing tools for the automatic identification of Parkinson’s disease using machine learning and the concept of BoVW. The proposed approach concerns a hierarchical-based learning technique to design visual dictionaries through the Deep Optimum-Path Forest classifier. The proposed method was evaluated in six datasets derived from data collected from individuals when performing handwriting exams. Experimental results showed the potential of the technique, with robust achievements. © 2021, CC BY.

关键词： learning algorithms

A-Optimal Active learning

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Boesen, Tue Haber, Eldad University of British Columbia Vancouver Canada

In this work we discuss the problem of active learning. We present an approach that is based on A-optimal experimental design of ill-posed problems and show how one can optimally label a data set by partially probing it, and use it to train a deep network. We present two approaches that make different assumptions on the data set. The first is based on a Bayesian interpretation of the semi-supervised learning problem with the graph Laplacian that is used for the prior distribution and the second is based on a frequentist approach, that updates the estimation of the bias term based on the recovery of the labels. We demonstrate that this approach can be highly efficient for estimating labels and training a deep *** Codes 68T05 (primary), 68T07, 62K05 © 2021, CC BY-NC-ND.

关键词： learning algorithms

Tesseract: Gradient flips core to secure federated learning against model poisoning attacks

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Sharma, Atul Chen, Wei Zhao, Joshua Qiu, Qiang Chaterji, Somali Bagchi, Saurabh Department of Electrical and Computer Engineering Purdue University

Federated learning—multi-party, distributed learning in a decentralized environment—is vulnerable to model poisoning attacks, even more so than centralized learning approaches. This is because malicious clients can collude and send in carefully tailored model updates to make the global model inaccurate. This motivated the development of Byzantine-resilient federated learning algorithms, such as Krum, Bulyan, FABA, and FoolsGold. However, a recently developed untargeted model poisoning attack showed that all prior defenses can be bypassed. The attack uses the intuition that simply by changing the sign of the gradient updates that the optimizer is computing, for a set of malicious clients, a model can be diverted from the optima to increase the test error rate. In this work, we develop TESSERACT—a defense against this directed deviation attack, a state-of-the-art model poisoning attack. TESSERACT is based on a simple intuition that in a federated learning setting, certain patterns of gradient flips are indicative of an attack. This intuition is remarkably stable across different learning algorithms, models, and datasets. TESSERACT assigns reputation scores to the participating clients based on their behavior during the training phase and then takes a weighted contribution of the clients. We show that TESSERACT provides robustness against even a white-box version of the attack. Copyright © 2021, The Authors. All rights reserved.

关键词： learning algorithms

learning-to-learn non-convex piecewise-Lipschitz functions

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Balcan, Maria-Florina Khodak, Mikhail Sharma, Dravyansh Talwalkar, Ameet

We analyze the meta-learning of the initialization and step-size of learning algorithms for piecewise-Lipschitz functions, a non-convex setting with applications to both machine learning and algorithms. Starting from recent regret bounds for the exponential forecaster on losses with dispersed discontinuities, we generalize them to be initialization-dependent and then use this result to propose a practical meta-learning procedure that learns both the initialization and the step-size of the algorithm from multiple online learning tasks. Asymptotically, we guarantee that the average regret across tasks scales with a natural notion of task-similarity that measures the amount of overlap between near-optimal regions of different tasks. Finally, we instantiate the method and its guarantee in two important settings: robust meta-learning and multi-task data-driven algorithm design. © 2021, CC BY.

关键词： learning algorithms

Effective Full Connection Neural Network Updating Using a Quantized Full Force Algorithm

学校读者我要写书评

暂无评论

SSRN

SSRN 2022年

作者： Heidarian, Mehdi Karimi, Gholamreza Department of Electrical and Computer Engineering Razi University Kermanshah*** Iran

This paper presents a new training algorithm that can update the situation of layers’ network, and therefore, connections,neurons and firing rate of neurons based on FORCE (first-order reduced and controlled error) training algorithm. Unlike traditional networks where only weights are updated, the Quantized Full FORCE Algorithm (QFF) also updates the number of neurons and connections between different layers in the network per iteration in a way that the whole firing rate of each layer is updated via selecting the best neurons and combining strong features. The update method is sequential, so that with each instance passing through the network, the network structure is updated with the Full FORCE algorithm. The algorithm updates the structure of networks with a multiple/single middle layer of the supervised version of feed forward networks such as Multilayer perceptron (MLP) and Restricted Boltzmann machine (RBM) based networks such as DBN, changing them into partially-connected networks. A combination of principal component analysis PCA and Linear Discriminant Analysis (LDA) algorithms has been used to cluster the network input features. The paper focuses on the deep supervised MLP network with backpropagation (BP) and various datasets and its comparison with other MLP based stat of art methodes and hybrid evolutionary algorithms. We achieved 98.15 percent accuracy in 500 epochs for facial expression and 98.6 and 97.7 percent for Wisconsin breast Cancer and Iris Flower in 100 epochs respectively. The training algorithm employed in the study enjoys a lower computational complexity while yielding faster and more accurate convergence, starting with a very low level of errors in comparison with the full connection network and it solves the challenge of getting stuck in local minima and poor convergence of Gradient Decent with BP. © 2022, The Authors. All rights reserved.

关键词： learning algorithms

Influence selection for active learning

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Liu, Zhuoming Ding, Hao Zhong, Huaping Li, Weijia Dai, Jifeng He, Conghui University Southern California Johns Hopkins University SenseTime Research CUHK-SenseTime Joint Lab The Chinese University of Hong Kong

The existing active learning methods select the samples by evaluating the sample's uncertainty or its effect on the diversity of labeled datasets based on different task-specific or model-specific criteria. In this paper, we propose the Influence Selection for Active learning(ISAL) which selects the unlabeled samples that can provide the most positive Influence on model performance. To obtain the Influence of the unlabeled sample in the active learning scenario, we design the Untrained Unlabeled sample Influence Calculation( UUIC) to estimate the unlabeled sample's expected gradient with which we calculate its Influence. To prove the effectiveness of UUIC, we provide both theoretical and experimental analyses. Since the UUIC just depends on the model gradients, which can be obtained easily from any neural network, our active learning algorithm is taskagnostic and model-agnostic. ISAL achieves state-of-theart performance in different active learning settings for different tasks with different datasets. Compared with previous methods, our method decreases the annotation cost at least by 12%, 13% and 16% on CIFAR10, VOC2012 and COCO, respectively. Copyright © 2021, The Authors. All rights reserved.

关键词： learning algorithms