检索结果-内蒙古大学图书馆

Ensemble of One Model: Creating Model Variations for Transformer with Layer Permutation

学校读者我要写书评

暂无评论

Ensemble of One Model: Creating Model Variations for Transfo...

2021 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2021

作者： Liaw, Andrew Hsu, Jia-Hao Wu, Chung-Hsien National Cheng Kung University Department of Computer Science and Information Engineering Tainan Taiwan

ISBN: (纸本)9789881476890

Ensemble involves combining the outputs of multiple models to increase performance. This technique has enjoyed great success across many fields in machine learning. This study focuses on a novel approach to increase performance of a model without any increase in number of parameters. The proposed approach involves training a model that can have different variations that perform well and different enough for ensemble. The variations are created by changing the order of the layers of a machine learning model. Moreover, this method can be combined with existing ensemble technique to further improve the performance. The task chosen for evaluating the performance is machine translation with Transformer, as Transformer is the current state-of-the-art model for this task as well as many natural language processing tasks. The IWSLT 2014 German to English and French to English datasets see an increase of at least 0.7 BLEU score over single model baseline with the same model size. When combined with multiple model ensemble, minimum increase of 0.3 BLEU is observed with no increase in parameters. © 2021 APSIPA.

关键词： learning algorithms

Constrained Machine learning Through Hyperspherical Representation

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Signorelli, Gaetano Lombardi, Michele University of Bologna Italy

The problem of ensuring constraints satisfaction on the output of machine learning models is critical for many applications, especially in safety-critical domains. Modern approaches rely on penalty-based methods at training time, which do not guarantee to avoid constraints violations;or constraint-specific model architectures (e.g., for monotonocity);or on output projection, which requires to solve an optimization problem that might be computationally demanding. We present the Hypersherical Constrained Representation, a novel method to enforce constraints in the output space for convex and bounded feasibility regions (generalizable to star domains). Our method operates on a different representation system, where Euclidean coordinates are converted into hyperspherical coordinates relative to the constrained region, which can only inherently represent feasible points. Experiments on a synthetic and a real-world dataset show that our method has predictive performance comparable to the other approaches, can guarantee 100% constraint satisfaction, and has a minimal computational cost at inference time. Copyright © 2025, The Authors. All rights reserved.

关键词： learning algorithms

Temperature Estimation in Induction Motors using Machine learning

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Li, Dinan Kakosimos, Panagiotis ABB AB Corporate Research Västeras Sweden

The number of electrified powertrains is ever increasing today towards a more sustainable future;thus, it is essential that unwanted failures are prevented, and a reliable operation is secured. Monitoring the internal temperatures of motors and keeping them under their thresholds is an important first step. Conventional modeling methods require expert knowledge and complicated mathematical approaches. With all the data a modern electric drive collects nowadays during the system operation, it is feasible to apply data-driven approaches for estimating thermal behaviors. In this paper, multiple machine-learning methods are investigated on their capability to approximate the temperatures of the stator winding and bearing in induction motors. The explored algorithms vary from linear to neural networks. For this reason, experimental lab data have been captured from a powertrain under predetermined operating conditions. For each approach, a hyperparameter search is then performed to find the optimal configuration. All the models are evaluated by various metrics, and it has been found that neural networks perform satisfactorily even under transient conditions. © 2025, CC BY-NC-ND.

关键词： learning algorithms

CrossedWires: A dataset of syntactically equivalent but semantically disparate deep learning models

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Zvyagin, Max Brettin, Thomas Ramanathan, Arvind Jha, Sumit K. Argonne National Laboratory LemontIL64309 United States Computer Science Department University of Texas at San Antonio

The training of neural networks using different deep learning frameworks may lead to drastically differing accuracy levels despite the use of the same neural network architecture and identical training hyperparameters such as learning rate and choice of optimization algorithms. Currently, our ability to build standardized deep learning models is limited by the availability of a suite of neural network and corresponding training hyperparameter benchmarks that expose differences between existing deep learning frameworks. In this paper, we present a living dataset of models and hyperparameters, called CrossedWires, that exposes semantic differences between two popular deep learning frameworks: PyTorch and Tensorflow. The CrossedWires dataset currently consists of models trained on CIFAR10 images using three different computer vision architectures: VGG16, ResNet50 and DenseNet121 across a large hyperparameter space. Using hyperparameter optimization, each of the three models was trained on 400 sets of hyperparameters suggested by the HyperSpace search algorithm. The CrossedWires dataset includes PyTorch and Tensforflow models with test accuracies as different as 0.681 on syntactically equivalent models and identical hyperparameter choices. The 340 GB dataset and benchmarks presented here include the performance statistics, training curves, and model weights for all 1200 hyperparameter choices, resulting in 2400 total models. The CrossedWires dataset provides an opportunity to study semantic differences between syntactically equivalent models across popular deep learning frameworks. Further, the insights obtained from this study can enable the development of algorithms and tools that improve reliability and reproducibility of deep learning frameworks. The dataset is freely available at https://***/maxzvyagin/crossedwires through a Python API and direct download link. Copyright © 2021, The Authors. All rights reserved.

关键词： learning algorithms

Improved learning rates for stochastic optimization: Two theoretical viewpoints

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Li, Shaojie Liu, Yong Gaoling School of Artificial Intelligence Renmin University of China Beijing Beijing100080 China

Generalization performance of stochastic optimization stands a central place in machine learning. In this paper, we investigate the excess risk performance and towards improved learning rates for two popular approaches of stochastic optimization: empirical risk minimization (ERM) and stochastic gradient descent (SGD). Although there exists plentiful generalization analysis of ERM and SGD for supervised learning, current theoretical understandings of ERM and SGD are either have stronger assumptions in convex learning, e.g., strong convexity condition, or show slow rates and less studied in nonconvex learning. Motivated by these problems, we aim to provide improved rates under milder assumptions in convex learning and derive faster rates in nonconvex learning. It is notable that our analysis span two popular theoretical viewpoints: stability and uniform convergence. To be specific, in stability regime, we present high probability rates of order O(1/n) w.r.t. the sample size n for ERM and SGD with milder assumptions in convex learning and similar high probability rates of order O(1/n) in nonconvex learning, rather than in expectation. Furthermore, this type of learning rate is improved to faster order O(1/n2) in uniform convergence regime. To the best of our knowledge, for ERM and SGD, the learning rates presented in this paper are all state-of-the-art. Copyright © 2021, The Authors. All rights reserved.

关键词： learning algorithms

Gating is Weighting: Understanding Gated Linear Attention through In-context learning

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Li, Yingcong Tarzanagh, Davoud Ataee Rawat, Ankit Singh Fazel, Maryam Oymak, Samet University of Michigan United States University of Pennsylvania United States Google Research NYC United States University of Washington United States

Linear attention methods offer a compelling alternative to softmax attention due to their efficiency in recurrent decoding. Recent research has focused on enhancing standard linear attention by incorporating gating while retaining its computational benefits. Such Gated Linear Attention (GLA) architectures include competitive models such as Mamba and RWKV. In this work, we investigate the in-context learning capabilities of the GLA model and make the following contributions. We show that a multilayer GLA can implement a general class of Weighted Preconditioned Gradient Descent (WPGD) algorithms with data-dependent weights. These weights are induced by the gating mechanism and the input, enabling the model to control the contribution of individual tokens to prediction. To further understand the mechanics of this weighting, we introduce a novel data model with multitask prompts and characterize the optimization landscape of learning a WPGD algorithm. Under mild conditions, we establish the existence and uniqueness (up to scaling) of a global minimum, corresponding to a unique WPGD solution. Finally, we translate these findings to explore the optimization landscape of GLA and shed light on how gating facilitates context-aware learning and when it is provably better than vanilla linear attention. © 2025, CC BY.

关键词： learning algorithms

Causal representation learning in offline visual reinforcement learning

学校读者我要写书评

暂无评论

Knowledge-Based Systems 2025年 320卷

作者： Zhang, Yaru Chen, Kaizhou Liu, Yunlong Department of Automation Xiamen University Xiamen361005 China

Real-world reinforcement learning (RL) applications contend with high-dimensional visual observations contaminated by confounding factors, which induce spurious correlations and obscure decision-relevant information. Compounding this issue, the inability to interact online necessitates reliance on pre-collected datasets, thereby hampering a deeper understanding of complex environment structures. In this work, by focusing on the causal rather than spurious correlations in the input and explicitly distinguishing between task-related and task-irrelevant elements of the causal variables, we propose a mask-based algorithm for learning task-related minimal causal state representations, namely MMCS. Specifically, MMCS guides the decoupling of minimal causal variables through mask network partitioning and jointly enforcing conditional independence and causal sufficiency, thereby eliminating unnecessary dependencies between variables and uncovering causal dependency structures. More importantly, MMCS is decoupled from downstream policy learning, and can function as a plug-in method compatible with any offline reinforcement learning algorithm. Empirical results on the Visual-D4RL benchmark demonstrate that MMCS significantly improves performance and sample efficiency in downstream policy learning. In addition, its robust performance in various distraction environments highlights the potential of MMCS to improve the generalizability of offline RL, especially under conditions of limited data and visual distractions. Code is available at https://***/DMU-XMU/***. © 2025 Elsevier B.V.

关键词： learning algorithms

PDSL: Privacy-Preserved Decentralized Stochastic learning with Heterogeneous Data Distribution

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Wang, Lina Yuan, Yunsheng Wang, Chunxiao Li, Feng School of Computer Science and Technology Shandong University Qingdao China Jinan China Shandong Provincial Key Laboratory of Computing Power Internet and Service Computing Shandong Fundamental Research Center for Computer Science Jinan China

In the paradigm of decentralized learning, a group of agents collaborates to learn a global model using distributed datasets without a central server. However, due to the heterogeneity of the local data across the different agents, learning a robust global model is rather challenging. Moreover, the collaboration of the agents relies on their gradient information exchange, which poses a risk of privacy leakage. In this paper, to address these issues, we propose PDSL, a novel privacy-preserved decentralized stochastic learning algorithm with heterogeneous data distribution. On one hand, we innovate in utilizing the notion of Shapley values such that each agent can precisely measure the contributions of its heterogeneous neighbors to the global learning goal;on the other hand, we leverage the notion of differential privacy to prevent each agent from suffering privacy leakage when it contributes gradient information to its neighbors. We conduct both solid theoretical analysis and extensive experiments to demonstrate the efficacy of our PDSL algorithm in terms of privacy preservation and convergence. © 2025, CC BY-NC-ND.

关键词： learning algorithms

Adaptive learning algorithm and its convergence analysis with complex-valued error loss network

学校读者我要写书评

暂无评论

Neural Networks 2025年 190卷 107677页

作者： Qian, Guobing Lin, Bingqing Mei, Jiaojiao Qian, Junhui Wang, Shiyuan College of Electronic and Information Engineering Southwest University Chongqing400715 China Cogenda Pte Ltd Jiangsu 215028 China School of Microelectronic and Communication Engineering Chongqing University Chongqing400030 China

In machine learning, the initial task is to construct a model that is capable of predicting the outcomes of new samples with the help of training samples. The loss function plays a key role in this task, as it acts as an important indicator to evaluate the overall model prediction performance. Building upon the work of Chen et al., this study introduces a novel model named as the Complex Error Loss Network (CELN). The CELN is designed to address the scenarios involving complex-valued signals and parameters within the context of supervised learning. Leveraging the contraction mapping theorem, this study investigates the convergence of the corresponding adaptive learning algorithm, underscoring the inherent capability of CELN to consistently approach and potentially reach the global minimum or optimal solution through iterative methods. CELN reduces the error by at least 4.1% compared to benchmark methods while maintaining stability in non-Gaussian noise scenarios. © 2025

关键词： learning algorithms