检索结果-内蒙古大学图书馆

arXiv 2024年

作者： Huang, Allen Hao Machine Learning and Optimization Laboratory EPFL Switzerland

Activation functions are core components of all deep learning architectures. Currently, the most popular activation functions are smooth ReLU variants like GELU and SiLU. These are self-gated activation functions where the range of the gating function is between zero and one. In this paper, we explore the viability of using arctan as a gating mechanism. A self-gated activation function that uses arctan as its gating function has a monotonically increasing first derivative. To make this activation function competitive, it is necessary to introduce a trainable parameter for every MLP block to expand the range of the gating function beyond zero and one. We find that this technique also improves existing self-gated activation functions. We conduct an empirical evaluation of Expanded ArcTan Linear Unit (xATLU), Expanded GELU (xGELU), and Expanded SiLU (xSiLU) and show that they outperform existing activation functions within a transformer architecture. Additionally, expanded gating ranges show promising results in improving first-order Gated Linear Units (GLU). © 2024, CC BY.

关键词： Chemical activation

来源：评论

学校读者我要写书评

暂无评论

Beyond spectral gap: the role of the topology in decentralized learning

The Journal of Machine Learning Research

引用

The Journal of machine learning Research 2023年第1期24卷 17074-17104页

作者： Thijs Vogels Hadrien Hendrikx Martin Jaggi Machine Learning and Optimization Laboratory EPFL Lausanne Switzerland

In data-parallel optimization of machine learning models, workers collaborate to improve their estimates of the model: more accurate gradients allow them to use larger learning rates and optimize faster. In the decentralized setting, in which workers communicate over a sparse graph, current theory fails to capture important aspects of real-world behavior. First, the 'spectral gap' of the communication graph is not predictive of its empirical performance in (deep) learning. Second, current theory does not explain that collaboration enables larger learning rates than training alone. In fact, it prescribes smaller learning rates, which further decrease as graphs become larger, failing to explain convergence dynamics in infinite graphs. This paper aims to paint an accurate picture of sparsely-connected distributed optimization. We quantify how the graph topology influences convergence in a quadratic toy problem and provide theoretical results for general smooth and (strongly) convex objectives. Our theory matches empirical observations in deep learning, and accurately describes the relative merits of different graph topologies. This paper is an extension of the conference paper by Vogels et al. (2022). Code: ***/epfml/topology-in-decentralized-learning.

关键词： decentralized learning convex optimization stochastic gradient descent gossip algorithms spectral gap

来源：评论

学校读者我要写书评

暂无评论

Beyond spectral gap (extended): The role of the topology in decentralized learning

arXiv

引用

arXiv 2023年

作者： Vogels, Thijs Hendrikx, Hadrien Jaggi, Martin Machine Learning and Optimization Laboratory EPFL Lausanne Switzerland

In data-parallel optimization of machine learning models, workers collaborate to improve their estimates of the model: more accurate gradients allow them to use larger learning rates and optimize faster. In the decentralized setting, in which workers communicate over a sparse graph, current theory fails to capture important aspects of real-world behavior. First, the ‘spectral gap’ of the communication graph is not predictive of its empirical performance in (deep) learning. Second, current theory does not explain that collaboration enables larger learning rates than training alone. In fact, it prescribes smaller learning rates, which further decrease as graphs become larger, failing to explain convergence dynamics in infinite graphs. This paper aims to paint an accurate picture of sparsely-connected distributed optimization. We quantify how the graph topology influences convergence in a quadratic toy problem and provide theoretical results for general smooth and (strongly) convex objectives. Our theory matches empirical observations in deep learning, and accurately describes the relative merits of different graph topologies. This paper is an extension of the conference paper by Vogels et al. (2022). Code: ***/epfml/topology-in-decentralized-learning. Copyright © 2023, The Authors. All rights reserved.

关键词： Convex optimization

来源：评论

学校读者我要写书评

暂无评论

Second-Order optimization with Lazy Hessians

arXiv

引用

arXiv 2022年

作者： Doikov, Nikita Chayti, El Mahdi Jaggi, Martin Machine Learning and Optimization Laboratory EPFL Switzerland

We analyze Newton’s method with lazy Hessian updates for solving general possibly non-convex optimization problems. We propose to reuse a previously seen Hessian for several iterations while computing new gradients at each step of the method. This significantly reduces the overall arithmetic complexity of second-order optimization schemes. By using the cubic regularization technique, we establish fast global convergence of our method to a second-order stationary point, while the Hessian does not need to be updated each iteration. For convex problems, we justify global and local superlinear rates for lazy Newton steps with quadratic regularization, which is easier to compute. The optimal frequency for updating the Hessian is once every d iterations, where d is the dimension of the problem. This provably improves the total arithmetic complexity of second-order algorithms by a factor √d. Copyright © 2022, The Authors. All rights reserved.

关键词： Iterative methods

来源：评论

学校读者我要写书评

暂无评论

Spectral preconditioning for gradient methods on graded non-convex functions 24

Spectral preconditioning for gradient methods on graded non-...

引用

Proceedings of the 41st International Conference on machine learning

作者： Nikita Doikov Sebastian U. Stich Martin Jaggi Machine Learning and Optimization Laboratory (MLO) EPFL Lausanne Switzerland CISPA Helmholtz Center for Information Security Saarbrücken Germany

The performance of optimization methods is often tied to the spectrum of the objective Hessian. Yet, conventional assumptions, such as smoothness, do often not enable us to make finely-grained convergence statements--particularly not for non-convex problems. Striving for a more intricate characterization of complexity, we introduce a unique concept termed graded nonconvexity. This allows to partition the class of nonconvex problems into a nested chain of subclasses. Interestingly, many traditional non-convex objectives, including partially convex problems, matrix factorizations, and neural networks, fall within these subclasses. As a second contribution, we propose gradient methods with spectral preconditioning, which employ inexact top eigenvectors of the Hessian to address the ill-conditioning of the problem, contingent on the grade. Our analysis reveals that these new methods provide provably superior convergence rates compared to basic gradient descent on applicable problem classes, particularly when large gaps exist between the top eigenvalues of the Hessian. Our theory is validated by numerical experiments executed on multiple practical machine learning problems.

关键词：

来源：评论

学校读者我要写书评

暂无评论

MEMORY EFFICIENT MIXED PRECISION OPTIMIZERS

arXiv

引用

arXiv 2023年

作者： Lewandowski, Basile Kosson, Atli Machine Learning Optimization laboratory École Polytechnique Fédérale de Lausanne Switzerland

Traditional optimization methods rely on the use of single-precision floating point arithmetic, which can be costly in terms of memory size and computing power. However, mixed precision optimization techniques leverage the use of both single and half-precision floating point arithmetic to reduce memory requirements while maintaining model accuracy. We provide here an algorithm to further reduce memory usage during the training of a model by getting rid of the floating point copy of the parameters, virtually keeping only half-precision numbers. We also explore the benefits of getting rid of the gradient’s value by executing the optimizer step during the back-propagation. In practice, we achieve up to 25% lower peak memory use and 15% faster training while maintaining the same level of accuracy. © 2023, CC BY.

关键词： Digital arithmetic

来源：评论

学校读者我要写书评

暂无评论

On convergence of incremental gradient for non-convex smooth functions 24

On convergence of incremental gradient for non-convex smooth...

引用

Proceedings of the 41st International Conference on machine learning

作者： Anastasia Koloskova Nikita Doikov Sebastian U. Stich Martin Jaggi Machine Learning and Optimization Laboratory (MLO) EPFL Lausanne Switzerland CISPA Helmholtz Center for Information Security Saarbrücken Germany

In machine learning and neural network optimization, algorithms like incremental gradient, and shuffle SGD are popular due to minimizing the number of cache misses and good practical convergence behavior. However, their optimization properties in theory, especially for nonconvex smooth functions, remain incompletely *** paper delves into the convergence properties of SGD algorithms with arbitrary data ordering, within a broad framework for non-convex smooth functions. Our findings show enhanced convergence guarantees for incremental gradient and single shuffle SGD. Particularly if n is the training set size, we improve n times the optimization term of convergence guarantee to reach accuracy ε from O(n/ε) to O(1/ε).

关键词：

来源：评论

学校读者我要写书评

暂无评论

High-speed identification of suspended carbon nanotubes using Raman spectroscopy and deep learning

引用

Microsystems & Nanoengineering 2022年第1期8卷 259-267页

作者： Jian Zhang Mickael L.Perrin Luis Barba Jan Overbeck Seoho Jung Brock Grassy Aryan Agal Rico Muff Rolf Brönnimann Miroslav Haluska Cosmin Roman Christofer Hierold Martin Jaggi Michel Calame Laboratory for Transport at Nanoscale Interfaces EmpaSwiss Federal Laboratories for Materials Science and TechnologyCH-8600 DübendorfSwitzerland Machine Learning and Optimization Laboratory School of Computer and Communication SciencesEPFLCH-1015 LausanneSwitzerland Department of Physics and Swiss Nanoscience Institute University of BaselCH-4056 BaselSwitzerland Micro-and Nanosystems Department of Mechanical and Process EngineeringETH ZurichCH-8092 ZurichSwitzerland

The identification of nanomaterials with the properties required for energy-efficient electronic systems is usually a tedious human task.A workflow to rapidly localize and characterize nanomaterials at the various stages of their integration into large-scale fabrication processes is essential for quality control and,ultimately,their industrial *** this work,we develop a high-throughput approach to rapidly identify suspended carbon nanotubes(CNTs)by using high-speed Raman imaging and deep learning *** for Raman spectra with extremely low signal-to-noise ratios(SNRs)of 0.9,we achieve a classification accuracy that exceeds 90%,while it reaches 98%for an SNR of *** applying a threshold on the output of the softmax layer of an optimized convolutional neural network(CNN),we further increase the accuracy of the ***,we propose an optimized Raman scanning strategy to minimize the acquisition time while simultaneously identifying the position,amount,and metallicity of CNTs on each *** approach can readily be extended to other types of nanomaterials and has the potential to be integrated into a production line to monitor the quality and properties of nanomaterials during fabrication.

关键词： carbon suspended speed

来源：评论

学校读者我要写书评

暂无评论

PCT-Cap: Point Cloud Transformer for Accurate 3D Capacitance Extraction

PCT-Cap: Point Cloud Transformer for Accurate 3D Capacitance...

引用

Electronics Design Automation (ISEDA), International Symposium of

作者： Ye Cai Yuyao Liang Zhipeng Luo Biwei Xie Xingquan Li College of Computer Science and Software Engineering Shenzhen University Shenzhen China Peng Cheng Laboratory Shenzhen China Center of Machine Learning and Optimization Minnan Normal University Zhangzhou China State Key Lab of Processors Chinese Academy of Sciences Institute of Computing Technology Beijing China

ISBN: (数字)9798350352030

ISBN: (纸本)9798350352047

Accurate parasitic capacitance extraction becomes increasingly essential in advanced technology nodes. 2.5D ex-traction requires more effort to maintain accuracy comparable to the 3D solver. In this work, two experiments reveal that error deviation in the 2.5D method enlarges as wire width shrinks and length increases. A novel Gauss law-based point cloud is proposed to model the 3D capacitance problem. With the proposed data representation, a transformer-based neural network architecture, called PCT-Cap, is designed for 3D pattern matching. PCT-Cap exhibits much better performance than the ResNet-based capacitance models. Extensive experiments on 28nm technology demonstrate that PCT-Cap can accurately predict over 95.41 % of total capacitance with an error margin of less than 5 %. Additionally, PCT-Cap achieves 95.90 % accuracy in predicting coupling capacitance within a margin of error of less than 10%. PCT-Cap achieves at least a 55 × speeding up compared with the commercial extraction tool while consuming negligible memory.

关键词： Point cloud compression Solid modeling Three-dimensional displays Accuracy Capacitance Feature extraction Transformers

来源：评论

学校读者我要写书评

暂无评论

Analysis of the Research Status of Information Technology Education Literature in China and Abroad 24

Analysis of the Research Status of Information Technology Ed...

引用

24th IEEE International Conference on High Performance Computing and Communications, 8th IEEE International Conference on Data Science and Systems, 20th IEEE International Conference on Smart City and 8th IEEE International Conference on Dependability in Sensor, Cloud and Big Data Systems and Application, HPCC/DSS/SmartCity/DependSys 2022

作者： Zhao, Yulin Liu, Kai Li, Junke School of Information Engineering Suqian University Jiangsu Suqian223800 China School of Computer and Information Qiannan Normal University for Nationalities Duyun558000 China Jiangsu Province Engineering Research Center of Smart Poultry Farming and Intelligent Equipment Jiangsu 223800 China Key Laboratory of Machine Learning and Unstructured Data Processing of Qiannan Duyun558000 China Key Laboratory of Complex Systems and Intelligent Optimization of Guizhou Duyun558000 China

ISBN: (纸本)9798350319934

Information technology education contributes to the development of national education. Understanding the research status of information technology education at home and abroad is helpful to the implementation of education informatization. This paper takes the information technology education research literature collected by CNKI and the Web of Science database as the object and uses the bibliometric method to analyze the literature characteristics of information technology education at home and abroad. Specifically, the paper mainly carries out statistical analysis from the time distribution of literature, journal distribution, and hot keywords, and visually describes the literature features of the information technology education discipline, providing references for the research of information technology education discipline at home and abroad. © 2022 IEEE.

关键词： Informatization

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：