检索结果-内蒙古大学图书馆

11th International Conference on learning Representations, ICLR 2023

作者： Zou, Difan Cao, Yuan Li, Yuanzhi Gu, Quanquan Department of Computer Science Institute of Data Science The University of Hong Kong Hong Kong Department of Statistics & Actuarial Science The University of Hong Kong Hong Kong Machine Learning Department Carnegie Mellon University United States Department of Computer Science University of California Los Angeles United States

Adaptive gradient methods such as Adam have gained increasing popularity in deep learning optimization. However, it has been observed in many deep learning applications such as image classification, Adam can converge to a different solution with a worse test error compared to (stochastic) gradient descent, even with a fine-tuned regularization. In this paper, we provide a theoretical explanation for this phenomenon: we show that in the nonconvex setting of learning over-parameterized two-layer convolutional neural networks starting from the same random initialization, for a class of data distributions (inspired from image data), Adam and gradient descent (GD) can converge to different global solutions of the training objective with provably different generalization errors, even with weight decay regularization. In contrast, we show that if the training objective is convex, and the weight decay regularization is employed, any optimization algorithms including Adam and GD will converge to the same solution if the training is successful. This suggests that the generalization gap between Adam and SGD in the presence of weight decay regularization is closely tied to the nonconvex landscape of deep learning optimization, which cannot be covered by the recent neural tangent kernel (NTK) based analysis. © 2023 11th International Conference on learning Representations, ICLR 2023. All rights reserved.

关键词： Gradient methods

来源：评论

学校读者我要写书评

暂无评论

Thermal disorder and phonon softening in the ferroelectric phase transition of lead titanate

引用

Physical Review B 2025年第9期111卷 094113-094113页

作者： Pinchen Xie Yixiao Chen Weinan E Roberto Car Program in Applied and Computational Mathematics Princeton University Princeton New Jersey 08544 USA AI for Science Institute Beijing China and Center for Machine Learning Research and School of Mathematical Sciences Peking University Beijing China

We report a molecular dynamics study of ab initio quality of the ferroelectric phase transition in crystalline PbTiO3. We model anharmonicity accurately in terms of potential energy and polarization surfaces trained on density functional theory data with modern machine learning techniques. Our simulations demonstrate that the transition has a strong order-disorder character, in agreement with diffraction experiments, and provide fresh insight into the approach to equilibrium across the phase transition. We find that the emergence and disappearance of the macroscopic polarization is driven by dipolar switching at the nanometer scale. We also computed the infrared optical absorption spectra in both the ferroelectric and the paraelectric phases, finding good agreement with the experimental Raman frequencies. Often, the almost ideal displacive character of the soft mode detected by Raman scattering in the paraelectric phase has been contrasted with the order-disorder character of the transition suggested by diffraction experiments. We settle this issue by showing that the soft mode coexists with a strong Debye relaxation associated with thermal disordering of the dipoles. The Debye relaxation feature is centered at zero frequency and appears near the transition temperature in both the ferroelectric and the paraelectric phases.

关键词： Ferroelectricity

来源：评论

学校读者我要写书评

暂无评论

Log barriers for safe black-box optimization with application to safe reinforcement learning

The Journal of Machine Learning Research

引用

The Journal of machine learning Research 2024年第1期25卷 8101-8154页

作者： Ilnura Usmanova Yarden As Maryam Kamgarpou Andreas Krause Swiss Data Science Center Paul Scherrer Institute Villigen Switzerland Institute for Machine Learning D-INFK ETH Zürich Zurich Switzerland STI-IGM-Sycamore EPFL Lausanne Switzerland

Optimizing noisy functions online, when evaluating the objective requires experiments on a deployed system, is a crucial task arising in manufacturing, robotics and various other domains. Often, constraints on safe inputs are unknown ahead of time, and we only obtain noisy information, indicating how close we are to violating the constraints. Yet, safety must be guaranteed at all times, not only for the final output of the *** introduce a general approach for seeking a stationary point in high dimensional non-linear stochastic optimization problems in which maintaining safety during learning is crucial. Our approach called LB-SGD, is based on applying stochastic gradient descent (SGD) with a carefully chosen adaptive step size to a logarithmic barrier approximation of the original problem. We provide a complete convergence analysis of non-convex, convex, and strongly-convex smooth constrained problems, with first-order and zeroth-order feedback. Our approach yields efficient updates and scales better with dimensionality compared to existing *** empirically compare the sample complexity and the computational cost of our method with existing safe learning approaches. Beyond synthetic benchmarks, we demonstrate the effectiveness of our approach on minimizing constraint violation in policy search tasks in safe reinforcement learning (RL).

关键词： stochastic optimization safe learning black-box optimization smooth constrained optimization reinforcement learning

来源：评论

学校读者我要写书评

暂无评论

On skip connections and normalisation layers in deep optimisation

arXiv

引用

arXiv 2022年

作者： MacDonald, Lachlan E. Valmadre, Jack Saratchandran, Hemanth Lucey, Simon Mathematical Institute for Data Science Johns Hopkins University United States Australian Institute for Machine Learning University of Adelaide Australia

We introduce a general theoretical framework, designed for the study of gradient optimisation of deep neural networks, that encompasses ubiquitous architecture choices including batch normalisation, weight normalisation and skip connections. Our framework determines the curvature and regularity properties of multilayer loss landscapes in terms of their constituent layers, thereby elucidating the roles played by normalisation layers and skip connections in globalising these properties. We then demonstrate the utility of this framework in two respects. First, we give the only proof of which we are aware that a class of deep neural networks can be trained using gradient descent to global optima even when such optima only exist at infinity, as is the case for the cross-entropy cost. Second, we identify a novel causal mechanism by which skip connections accelerate training, which we verify predictively with ResNets on MNIST, CIFAR10, CIFAR100 and ImageNet. Copyright © 2022, The Authors. All rights reserved.

关键词： Deep neural networks

来源：评论

学校读者我要写书评

暂无评论

Audio Enhancement for Computer Audition—An Iterative Training Paradigm Using Sample Importance

引用

Journal of Computer science & Technology 2024年第4期39卷 895-911页

作者： Manuel Milling Shuo Liu Andreas Triantafyllopoulos Ilhan Aslan Björn W.Schuller Chair of Embedded Intelligence for Health Care and Wellbeing University of AugsburgAugsburg 86159Germany Chair of Health Informatics München rechts der IsarTechnical University of MunichMunich 81675Germany Munich Center for Machine Learning Munich 80333Germany Huawei Technologies MunichMunich 80992Germany Munich Data Science Institute Garching 85748Germany Group on Language Audio and MusicImperial College LondonLondon SW72AZU.K.

Neural network models for audio tasks,such as automatic speech recognition(ASR)and acoustic scene classification(ASC),are susceptible to noise contamination for real-life *** improve audio quality,an enhancement module,which can be developed independently,is explicitly used at the front-end of the target audio *** this paper,we present an end-to-end learning solution to jointly optimise the models for audio enhancement(AE)and the subsequent *** guide the optimisation of the AE module towards a target application,and especially to overcome difficult samples,we make use of the sample-wise performance measure as an indication of sample *** experiments,we consider four representative applications to evaluate our training paradigm,i.e.,ASR,speech command recognition(SCR),speech emotion recognition(SER),and *** applications are associated with speech and nonspeech tasks concerning semantic and non-semantic features,transient and global information,and the experimental results indicate that our proposed approach can considerably boost the noise robustness of the models,especially at low signal-to-noise ratios,for a wide range of computer audition tasks in everyday-life noisy environments.

关键词： audio enhancement computer audition joint optimisation multi-task learning voice suppression

来源：评论

学校读者我要写书评

暂无评论

Deep learning Application for Images Augmentation in Electrical Component Classification System 3rd

Deep Learning Application for Images Augmentation in Electri...

引用

3rd International Conference on Advances in Information and Communication Technology, ICTA 2024

作者： Long, Nguyen The Huong, Nguyen Thu Lien, Pham Thi Gen, Anna Shmeleva Viet, Tran Tuan Department of Informatics Institute of Cybersecurity and Digital Technologies MIREA –Russian Technological University Moscow119454 Russia Laboratory of Artificial Intelligence and Machine Learning Institute of Information Technology and Data Science Irkutsk National Research Technical University Irkutsk664074 Russia University of Information and Communication Technology Thai Nguyen University Thai Nguyen70000 Viet Nam

ISBN: (纸本)9783031809422

The possibility of application a convolutional neural network to assess the augmentation of electrical images is proposed. We studied various conditions for sample preparation, optimizer algorithms, the number of pixels in the samples, the size of the training sample, color schemes, compression quality, and other photometric parameters in view of effect on training the neural network. Due to the proposed preliminary data preparation, the optimum of the architecture and hyperparameters of the neural network we achieved a classification accuracy of at least 98%. This paper outlines measurements of several devices and a process for simulating large amounts of annotated data based on those measurements. A convolutional neural net (CNN) algorithm for classifying signals is described. Performance of the CNN is compared to that of dynamic time warping (DTW) and correlation algorithms, with respect to varying training data set size and noise level. The CNN has the best performance for almost all cases considered. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.

关键词： Convolutional neural networks

来源：评论

学校读者我要写书评

暂无评论

Food-Health-Chain: A Food Supply Chain for Internet of Health Things Using Blockchain

Food-Health-Chain: A Food Supply Chain for Internet of Healt...

引用

International Conference on Network Security and Blockchain Technology, ICNSBT 2023

作者： Das, Puja Haldar, Amrita Singh, Moutushi Pise, Anil Audumbar Guha Roy, Deepsubhra Department of Computer Science HMM College for Women Dakshineswar Kolkata700035 India Department of Computer Science and Business Studies IEM Kolkata700091 India Department of Information Technology IEM Salt Lake Kolkata700091 India Department of Data Science and Machine Learning Computer Science University of the Witwatersrand Johannesburg South Africa Institute of Engineering and Management Kolkata700091 India

ISBN: (纸本)9789819944323

Foodborne disease is still a major public health concern in both advanced and emerging economies. In terms of adopting new technology, the health industry continues to lag behind all other sectors. India has traditionally relied on food farming to make money but which food item is good for whom we have no proper idea. Especially when it comes to healthcare sectors, the Internet of health things (IoHT) and blockchain can play an important role. It can enhance the quality of care for its users and reduce costs by allocating the correct food chain to the targeted users. We are currently employing blockchain in supply chain management to cluster the patients based on their health care type. For example, diabetic patients prefer a particular type of food. Segregating the food type within the supply chain can reduce a lot of effort and cost. Blockchain can provide a solution to the issues that arise in the healthcare-based supply chain system because of its decentralized and immutable nature. The service's execution has been depicted in pictures and accompanied by detailed descriptions. The ultimate result of this study was to demonstrate the immutability, availability, and transparency security of blockchain in the healthcare-based supply chain. © 2024, The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

关键词： Blockchain

来源：评论

学校读者我要写书评

暂无评论

SPEAR: Design and Implementation of an Advanced Virtual Assistant 4

SPEAR: Design and Implementation of an Advanced Virtual Assi...

引用

4th International Conference on Sustainable Expert Systems, ICSES 2024

作者： Jain, Garima Shukla, Amita Bairwa, Nitesh Kumar Chaudhary, Anamika Patel, Ashish Jain, Ankush Noida Institute of Engineering and Technology Department of Computer Science and Buisness Systems Greater Noida India Noida Institute of Engineering and Technology Department of Artificial Intelligence Greater Noida India Noida Institute of Engineering and Technology Department of Data Science Greater Noida India Dronacharya Group of Institutions Department of Artificial Intelligence and Machine Learning Greater Noida India Netaji Subhash University of Technology Department of Computer Science and Enginnering Delhi India

ISBN: (纸本)9798331540364

This research presents the development and evaluation of SPEAR, an advanced voice-activated personal desktop assistant designed to address challenges in existing virtual assistant technology, such as limited language support, low recognition accuracy, and inadequate task automation. Despite recent advancements in Natural Language Processing (NLP) and machine learning, implementation in current systems remains subpar. SPEAR achieves an 88% overall command execution success rate, with a speech recognition accuracy of 92% and an average response time of 1.8 seconds, demonstrating efficiency in real-time interactions. Developed using Python and machine learning libraries, SPEAR supports 15 voice commands across multiple platforms, excelling in note creation (95% success), web searching (94%), and information retrieval (92%). Its natural language understanding score of 0.86 and context retention rate of 75% highlight its conversational capabilities. While the assistant performs well in messaging functionality (85% success) and currently supports three languages, expanding its multilingual capabilities is essential for broader user engagement. Metrics indicate a satisfaction score of 8.2/10 and an average of 12 daily interactions per user, showcasing its practical utility. Additionally, the system's text-to-speech quality (8.5/10) and error recovery rate (82 %) affirm its robustness. These findings contribute valuable insights for future research, focusing on enhancing multilingual support, broadening task categories, and improving context retention, ultimately positioning SPEAR to rival established industry leaders by offering a comprehensive suite of services. © 2024 IEEE.

关键词： Speech recognition

来源：评论

学校读者我要写书评

暂无评论

A MEMORY EFFICIENT RANDOMIZED SUBSPACE OPTIMIZATION METHOD FOR TRAINING LARGE LANGUAGE MODELS

arXiv

引用

arXiv 2025年

作者： Chen, Yiming Zhang, Yuan Liu, Yin Yuan, Kun Wen, Zaiwen Beijing International Center for Mathematical Research Peking University Beijing China Center for Data Science Peking University Beijing China Center for Machine Learning Research Peking University Beijing China

The memory challenges associated with training Large Language Models (LLMs) have become a critical concern, particularly when using the Adam optimizer. To address this issue, numerous memory-efficient techniques have been proposed, with GaLore standing out as a notable example designed to reduce the memory footprint of optimizer states. However, these approaches do not alleviate the memory burden imposed by activations, rendering them unsuitable for scenarios involving long context sequences or large mini-batches. Moreover, their convergence properties are still not well-understood in the literature. In this work, we introduce a Randomized Subspace Optimization framework for pre-training and fine-tuning LLMs. Our approach decomposes the high-dimensional training problem into a series of lower-dimensional subproblems. At each iteration, a random subspace is selected, and the parameters within that subspace are optimized. This structured reduction in dimensionality allows our method to simultaneously reduce memory usage for both activations and optimizer states. We establish comprehensive convergence guarantees and derive rates for various scenarios, accommodating different optimization strategies to solve the subproblems. Extensive experiments validate the superior memory and communication efficiency of our method, achieving performance comparable to GaLore and Adam. Copyright © 2025, The Authors. All rights reserved.

关键词：

来源：评论

学校读者我要写书评

暂无评论

Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling

arXiv

引用

arXiv 2024年

作者： Wang, Mingze Weinan, E. School of Mathematical Sciences Peking University Beijing China Center for Machine Learning Research School of Mathematical Sciences Peking University Beijing China AI for Science Institute Beijing China

We conduct a systematic study of the approximation properties of Transformer for sequence modeling with long, sparse and complicated memory. We investigate the mechanisms through which different components of Transformer, such as the dot-product self-attention, positional encoding and feed-forward layer, affect its expressive power, and we study their combined effects through establishing explicit approximation rates. Our study reveals the roles of critical parameters in the Transformer, such as the number of layers and the number of attention heads. These theoretical insights are validated experimentally and offer natural suggestions for alternative architectures. Copyright © 2024, The Authors. All rights reserved.

关键词： Signal encoding

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：