检索结果-内蒙古大学图书馆

Automated learning rate scheduler for large-batch training

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Kim, Chiheon Kim, Saehoon Kim, Jongmin Lee, Donghoon Kim, Sungwoong Kakao Brain Korea Republic of

Large-batch training has been essential in leveraging large-scale datasets and models in deep learning. While it is computationally beneficial to use large batch sizes, it often requires a specially designed learning rate (LR) schedule to achieve a comparable level of performance as in smaller batch training. Especially, when the number of training epochs is constrained, the use of a large LR and a warmup strategy is critical in the final performance of large-batch training due to the reduced number of updating steps. In this work, we propose an automated LR scheduling algorithm which is effective for neural network training with a large batch size under the given epoch budget. In specific, the whole schedule consists of two phases: adaptive warmup and predefined decay, where the LR is increased until the training loss no longer decreases and decreased to zero until the end of training. Here, whether the training loss has reached the minimum value is robustly checked with Gaussian process smoothing in an online manner with a low computational burden. Coupled with adaptive stochastic optimizers such as AdamP and LAMB, the proposed scheduler successfully adjusts the LRs without cumbersome hyperparameter tuning and achieves comparable or better performances than tuned baselines on various image classification benchmarks and architectures with a wide range of batch sizes. Copyright © 2021, The Authors. All rights reserved.

关键词： learning algorithms

FedNL: Making Newton-Type Methods Applicable to Federated learning

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Safaryan, Mher Islamov, Rustem Qian, Xun Richtárik, Peter King Abdullah University of Science and Technology Thuwal Saudi Arabia Thuwal Saudi Arabia Dolgoprudny Russia

Inspired by recent work of Islamov et al (2021), we propose a family of Federated Newton Learn (FedNL) methods, which we believe is a marked step in the direction of making second-order methods applicable to FL. In contrast to the aforementioned work, FedNL employs a different Hessian learning technique which i) enhances privacy as it does not rely on the training data to be revealed to the coordinating server, ii) makes it applicable beyond generalized linear models, and iii) provably works with general contractive compression operators for compressing the local Hessians, such as Top-K or Rank-R, which are vastly superior in practice. Notably, we do not need to rely on error feedback for our methods to work with contractive compressors. Moreover, we develop FedNL-PP, FedNL-CR and FedNL-LS, which are variants of FedNL that support partial participation, and globalization via cubic regularization and line search, respectively, and FedNL-BC, which is a variant that can further benefit from bidirectional compression of gradients and models, i.e., smart uplink gradient and smart downlink model compression. We prove local convergence rates that are independent of the condition number, the number of training data points, and compression variance. Our communication efficient Hessian learning technique provably learns the Hessian at the optimum. Finally, we perform a variety of numerical experiments that show that our FedNL methods have state-of-the-art communication complexity when compared to key baselines. Copyright © 2021, The Authors. All rights reserved.

关键词： learning algorithms

Communication-Efficient Collaborative Best Arm Identification

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Karpov, Nikolai Zhang, Qin Department of Computer Science Indiana University BloomingtonIN47405 United States

We investigate top-m arm identification, a basic problem in bandit theory, in a multi-agent learning model in which agents collaborate to learn an objective function. We are interested in designing collaborative learning algorithms that achieve maximum speedup (compared to single-agent learning algorithms) using minimum communication cost, as communication is frequently the bottleneck in multi-agent learning. We give both algorithmic and impossibility results, and conduct a set of experiments to demonstrate the effectiveness of our algorithms. © 2022, CC BY.

关键词： learning algorithms

Robust Empirical Risk Minimization with Tolerance

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Bhattacharjee, Robi Hopkins, Max Kumar, Akash Yu, Hantao Chaudhuri, Kamalika UCSD United States Colombia

Developing simple, sample-efficient learning algorithms for robust classification is a pressing issue in today’s tech-dominated world, and current theoretical techniques requiring exponential sample complexity and complicated improper learning rules fall far from answering the need. In this work we study the fundamental paradigm of (robust) empirical risk minimization (RERM), a simple process in which the learner outputs any hypothesis minimizing its training error. RERM famously fails to robustly learn VC classes (Montasser et al., 2019a), a bound we show extends even to ‘nice’ settings such as (bounded) halfspaces. As such, we study a recent relaxation of the robust model called tolerant robust learning (Ashtiani et al., 2022) where the output classifier is compared to the best achievable error over slightly larger perturbation sets. We show that under geometric niceness conditions, a natural tolerant variant of RERM is indeed sufficient for γ-tolerant robust learning VC classes over Rd, and requires only Õ ( V C(H)d log γδ D )samples for robustness regions ϵ2 of (maximum) diameter D. © 2022, CC BY.

关键词： learning algorithms

On the cross-lingual transferability of multilingual prototypical models across NLU tasks

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Cattan, Oralie Servan, Christophe Rosset, Sophie QWANT 61 rue de Villiers Neuilly-sur-Seine 92200 France Université Paris-Saclay CNRS LISN Orsay 91405 France

Supervised deep learning-based approaches have been applied to task-oriented dialog and have proven to be effective for limited domain and language applications when a sufficient number of training examples are available. In practice, these approaches suffer from the drawbacks of domain-driven design and under-resourced languages. Domain and language models are supposed to grow and change as the problem space evolves. On one hand, research on transfer learning has demonstrated the cross-lingual ability of multilingual Transformers-based models to learn semantically rich representations. On the other, in addition to the above approaches, meta-learning have enabled the development of task and language learning algorithms capable of far generalization. Through this context, this article proposes to investigate the cross-lingual transferability of using synergistically few-shot learning with prototypical neural networks and multilingual Transformers-based models. Experiments in natural language understanding tasks on MultiATIS++ corpus shows that our approach substantially improves the observed transfer learning performances between the low and the high resource languages. More generally our approach confirms that the meaningful latent space learned in a given language can be can be generalized to unseen and under-resourced ones using *** Codes 68T50 © 2022, CC BY.

关键词： learning algorithms

Formation learning algorithms for mobile agents subject to 2-D dynamically changing topologies

学校读者我要写书评

暂无评论

Formation learning algorithms for mobile agents subject to 2...

American Control Conference (ACC)

作者： Meng, Deyuan Jia, Yingmin Du, Junping Zhang, Jun Li, Wenling Beihang Univ BUAA Div Res 7 Beijing 100191 Peoples R China

ISBN: (纸本)9781479901784

In this paper, we consider a two-dimensional (2-D) formation problem for multi-agent systems subject to switching topologies that dynamically change along both a finite time axis and an infinite iteration axis. We present a distributed iterative learning control (ILC) algorithm via the nearest neighbor rules. By employing the 2-D approach, we develop both the asymptotic and exponentially fast convergence of our formation ILC, which can be guaranteed by conditions in terms of the spectral radius and the matrix norms, respectively.

关键词： Multi-agent systems ILC switching topologies formation learning algorithms

Nevis’22: A Stream of 100 Tasks Sampled from 30 Years of Computer Vision Research

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Bornschein, Jörg Galashov, Alexandre Hemsley, Ross Rannen-Triki, Amal Chen, Yutian Chaudhry, Arslan He, Xu Owen Douillard, Arthur Caccia, Massimo Feng, Qixuan Shen, Jiajun Rebuffi, Sylvestre-Alvise Stacpoole, Kitty de las Casas, Diego Hawkins, Will Lazaridou, Angeliki Teh, Yee Whye Rusu, Andrei A. Pascanu, Razvan Ranzato, Marc Aurelio

A shared goal of several machine learning communities like continual learning, meta-learning and transfer learning, is to design algorithms and models that efficiently and robustly adapt to unseen tasks. An even more ambitious goal is to build models that never stop adapting, and that become increasingly more efficient through time by suitably transferring the accrued knowledge. Beyond the study of the actual learning algorithm and model architecture, there are several hurdles towards our quest to build such models, such as the choice of learning protocol, metric of success and data needed to validate research hypotheses. In this work, we introduce the Never-Ending VIsual-classification Stream (Nevis’22), a benchmark consisting of a stream of over 100 visual classification tasks, sorted chronologically and extracted from papers sampled uniformly from computer vision proceedings spanning the last three decades. The resulting stream reflects what the research community thought was meaningful at any point in time, and it serves as an ideal test bed to assess how well models can adapt to new tasks, and do so better and more efficiently as time goes by. Despite being limited to classification, the resulting stream has a rich diversity of tasks from OCR, to texture analysis, scene recognition, and so forth. The diversity is also reflected in the wide range of dataset sizes, spanning over four orders of magnitude. Overall, Nevis’22 poses an unprecedented challenge for current sequential learning approaches due to the scale and diversity of tasks, yet with a low entry barrier as it is limited to a single modality and well understood supervised learning problems. Moreover, we provide a reference implementation including strong baselines and an evaluation protocol to compare methods in terms of their trade-off between accuracy and compute. We hope that Nevis’22 can be useful to researchers working on continual learning, meta-learning, AutoML and more generally sequential lear

关键词： learning algorithms

Adversarial targeted forgetting in regularization and generative based continual learning models

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Umer, Muhammad Polikar, Robi Department of Electrical & Computer Engineering Rowan University Glassboro United States

Continual (or "incremental") learning approaches are employed when additional knowledge or tasks need to be learned from subsequent batches or from streaming data. However these approaches are typically adversary agnostic, i.e., they do not consider the possibility of a malicious attack. In our prior work, we explored the vulnerabilities of Elastic Weight Consolidation (EWC) to the perceptible misinformation. We now explore the vulnerabilities of other regularization-based as well as generative replay-based continual learning algorithms, and also extend the attack to imperceptible misinformation. We show that an intelligent adversary can take advantage of a continual learning algorithm’s capabilities of retaining existing knowledge over time, and force it to learn and retain deliberately introduced misinformation. To demonstrate this vulnerability, we inject backdoor attack samples into the training data. These attack samples constitute the misinformation, allowing the attacker to capture control of the model at test time. We evaluate the extent of this vulnerability on both rotated and split benchmark variants of the MNIST dataset under two important domain and class incremental learning scenarios. We show that the adversary can create a "false memory" about any task by inserting carefully-designed backdoor samples to the test instances of that task thereby controlling the amount of forgetting of any task of its choosing. Perhaps most importantly, we show this vulnerability to be very acute and damaging: the model memory can be easily compromised with the addition of backdoor samples into as little as 1% of the training data, even when the misinformation is imperceptible to human eye. © 2021, CC BY.

关键词： learning algorithms

Unified group fairness on federated learning

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Zhang, Fengda Kuang, Kun Liu, Yuxuan Wu, Chao Wu, Fei Lu, Jiaxun Shao, Yunfeng Xiao, Jun College of Computer Science and Technology Zhejiang University School of Public Affairs Zhejiang University Huawei Noah’s Ark Lab

Federated learning (FL) has emerged as an important machine learning paradigm where a global model is trained based on the private data from distributed clients. However, most of existing FL algorithms cannot guarantee the performance fairness towards different clients or different groups of samples because of the distribution shift. Recent researches focus on achieving fairness among clients, but they ignore the fairness towards different groups formed by sensitive attribute(s) (e.g., gender and/or race), which is important and practical in real applications. To bridge this gap, we formulate the goal of unified group fairness on FL which is to learn a fair global model with similar performance on different groups. To achieve the unified group fairness for arbitrary sensitive attribute(s), we propose a novel FL algorithm, named Group Distributionally Robust Federated Averaging (G-DRFA), which mitigates the distribution shift across groups with theoretical analysis of convergence rate. Specifically, we treat the performance of the federated global model at each group as an objective and employ the distributionally robust techniques to maximize the performance of the worst-performing group over an uncertainty set by group reweighting. We validate the advantages of the G-DRFA algorithm with various kinds of distribution shift settings in experiments, and the results show that G-DRFA algorithm outperforms the existing fair federated learning algorithms on unified group fairness. Copyright © 2021, The Authors. All rights reserved.

关键词： learning algorithms