检索结果-内蒙古大学图书馆

A MEMORY EFFICIENT RANDOMIZED SUBSPACE OPTIMIZATION METHOD FOR TRAINING LARGE LANGUAGE MODELS

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Chen, Yiming Zhang, Yuan Liu, Yin Yuan, Kun Wen, Zaiwen Beijing International Center for Mathematical Research Peking University Beijing China Center for Data Science Peking University Beijing China Center for Machine Learning Research Peking University Beijing China

The memory challenges associated with training Large Language Models (LLMs) have become a critical concern, particularly when using the Adam optimizer. To address this issue, numerous memory-efficient techniques have been proposed, with GaLore standing out as a notable example designed to reduce the memory footprint of optimizer states. However, these approaches do not alleviate the memory burden imposed by activations, rendering them unsuitable for scenarios involving long context sequences or large mini-batches. Moreover, their convergence properties are still not well-understood in the literature. In this work, we introduce a Randomized Subspace Optimization framework for pre-training and fine-tuning LLMs. Our approach decomposes the high-dimensional training problem into a series of lower-dimensional subproblems. At each iteration, a random subspace is selected, and the parameters within that subspace are optimized. This structured reduction in dimensionality allows our method to simultaneously reduce memory usage for both activations and optimizer states. We establish comprehensive convergence guarantees and derive rates for various scenarios, accommodating different optimization strategies to solve the subproblems. Extensive experiments validate the superior memory and communication efficiency of our method, achieving performance comparable to GaLore and Adam. Copyright © 2025, The Authors. All rights reserved.

关键词：

IWILDS'25: The 5th International Workshop on Investigating learning During Web Search 25

学校读者我要写书评

暂无评论

IWILDS'25: The 5th International Workshop on Investigating L...

18th ACM International Conference on Web Search and Data Mining, WSDM 2025

作者： Hoppe, Anett Yu, Ran Liu, Jiqun Bhattacharya, Nilavra TIB - Leibniz Information Centre for Science and Technology Hannover Germany L3S Research Center Leibniz Universität Hannover Hannover Germany University of Bonn Bonn Germany Lamarr Institute for Machine Learning and Artificial Intelligence *** Bonn Germany The University of Oklahoma NormanOK United States ABB AG Corporate Research Center Germany Mannheim Germany

ISBN: (纸本)9798400713293

Web-based learning is evolving rapidly as traditional search engines are complemented by Large Language Models (LLMs) and other AI technologies. This evolution offers new opportunities, such as automated information synthesis and personalized learning experiences. However, this also presents new challenges, including the need for learners to be aware of potential biases and misinformation in AI-generated content, and to maintain focus and depth in their learning journeys. The Search as learning (SAL) field investigates how individuals learn through web interactions, examining the interplay between search behaviors, information synthesis, and learning outcomes. SAL research aims to measure, predict, and support effective learning strategies in modern web environments. IWILDS'25 provides an interdisciplinary platform to explore these dynamics, bringing together researchers and practitioners from information retrieval, education, psychology, and related fields. This full-day workshop will feature keynotes, paper presentations, and discussions on advancing SAL research and practice. © 2025 Copyright held by the owner/author(s).

关键词： Contrastive learning

Adaptive deep probabilistic regression for real-time motor excitability state prediction from human EEG

学校读者我要写书评

暂无评论

Brain Stimulation 2025年第1期18卷 400-401页

作者： Haxel, Lisa Kapoor, Jaivardhan Ahola, Oskari Kahilakoski, Olli-Pekka Kirchhoff, Miriam Roine, Timo Ziemann, Ulf Macke, Jakob H. Machine Learning in Science Excellence Cluster Machine Learning & Tübingen AI Center Germany Hertie Institute for Clinical Brain Research Department Neurology and Stroke Germany Department of Neuroscience and Biomedical Engineering Aalto University School of Science Finland Department Empirical Inference Max Planck Institute for Intelligent Systems Germany

HyperSHAP: Shapley Values and Interactions for Hyperparameter Importance

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Wever, Marcel Muschalik, Maximilian Fumagalli, Fabian Lindauer, Marius L3S Research Center Germany Leibniz University Hannover Germany Munich Center for Machine Learning Germany LMU Munich Germany Bielefeld University Germany

Hyperparameter optimization (HPO) is a crucial step in achieving strong predictive performance. However, the impact of individual hyperparameters on model generalization is highly context-dependent, prohibiting a one-size-fits-all solution and requiring opaque automated machine learning (AutoML) systems to find optimal configurations. The black-box nature of most AutoML systems undermines user trust and discourages adoption. To address this, we propose a game-theoretic explainability framework for HPO that is based on Shapley values and interactions. Our approach provides an additive decomposition of a performance measure across hyperparameters, enabling local and global explanations of hyperparameter importance and interactions. The framework, named HYPERSHAP, offers insights into ablations, the tunability of learning algorithms, and optimizer behavior across different hyperparameter spaces. We evaluate HYPERSHAP on various HPO benchmarks by analyzing the interaction structure of the HPO problem. Our results show that while higher-order interactions exist, most performance improvements can be explained by focusing on lower-order representations. © 2025, CC BY-SA.

关键词： Game theory

Thermal disorder and phonon softening in the ferroelectric phase transition of lead titanate

学校读者我要写书评

暂无评论

Physical Review B 2025年第9期111卷 094113-094113页

作者： Pinchen Xie Yixiao Chen Weinan E Roberto Car Program in Applied and Computational Mathematics Princeton University Princeton New Jersey 08544 USA AI for Science Institute Beijing China and Center for Machine Learning Research and School of Mathematical Sciences Peking University Beijing China

We report a molecular dynamics study of ab initio quality of the ferroelectric phase transition in crystalline PbTiO3. We model anharmonicity accurately in terms of potential energy and polarization surfaces trained on density functional theory data with modern machine learning techniques. Our simulations demonstrate that the transition has a strong order-disorder character, in agreement with diffraction experiments, and provide fresh insight into the approach to equilibrium across the phase transition. We find that the emergence and disappearance of the macroscopic polarization is driven by dipolar switching at the nanometer scale. We also computed the infrared optical absorption spectra in both the ferroelectric and the paraelectric phases, finding good agreement with the experimental Raman frequencies. Often, the almost ideal displacive character of the soft mode detected by Raman scattering in the paraelectric phase has been contrasted with the order-disorder character of the transition suggested by diffraction experiments. We settle this issue by showing that the soft mode coexists with a strong Debye relaxation associated with thermal disordering of the dipoles. The Debye relaxation feature is centered at zero frequency and appears near the transition temperature in both the ferroelectric and the paraelectric phases.

关键词： Ferroelectricity

Breaking Memory Limits: Gradient Wavelet Transform Enhances LLMs Training

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Wen, Ziqing Luo, Ping Wang, Jiahuan Deng, Xiaoge Zou, Jinping Yuan, Kun Sun, Tao Li, Dongsheng College of Computer Science and Technology National University of Defense Technology Hunan Changsha China Center for Machine Learning Research Peking University Bejing China

Large language models (LLMs) have shown impressive performance across a range of natural language processing tasks. However, their vast number of parameters introduces significant memory challenges during training, particularly when using memory-intensive optimizers like Adam. Existing memory-efficient algorithms often rely on techniques such as singular value decomposition projection or weight freezing. While these approaches help alleviate memory constraints, they generally produce suboptimal results compared to full-rank updates. In this paper, we investigate the memory-efficient method beyond low-rank training, proposing a novel solution called Gradient Wavelet Transform (GWT), which applies wavelet transforms to gradients in order to significantly reduce the memory requirements for maintaining optimizer states. We demonstrate that GWT can be seamlessly integrated with memory-intensive optimizers, enabling efficient training without sacrificing performance. Through extensive experiments on both pre-training and fine-tuning tasks, we show that GWT achieves state-of-the-art performance compared with advanced memory-efficient optimizers and full-rank approaches in terms of both memory usage and training performance. © 2025, CC BY.

关键词： Natural language processing systems

Aligning Information Capacity Between Vision and Language via Dense-to-Sparse Feature Distillation for Image-Text Matching

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Liu, Yang Feng, Wentao Liu, Zhuoyao Huang, Shudong Lv, Jiancheng College of Computer Science Sichuan University Chengdu610065 China Engineering Research Center of Machine Learning and Industry Intelligence Ministry of Education Chengdu610065 China

Enabling Visual Semantic Models to effectively handle multi-view description matching has been a longstanding challenge. Existing methods typically learn a set of embeddings to find the optimal match for each view’s text and compute similarity. However, the visual and text embeddings learned through these approaches have limited information capacity and are prone to interference from locally similar negative samples. To address this issue, we argue that the information capacity of embeddings is crucial and propose Dense-to-Sparse Feature Distilled Visual Semantic Embedding (D2S-VSE), which enhances the information capacity of sparse text by leveraging dense text distillation. Specifically, D2S-VSE is a two-stage framework. In the pretraining stage, we align images with dense text to enhance the information capacity of visual semantic embeddings. In the fine-tuning stage, we optimize two tasks simultaneously, distilling dense text embeddings to sparse text embeddings while aligning images and sparse texts, enhancing the information capacity of sparse text embeddings. Our proposed D2S-VSE model is extensively evaluated on the large-scale MS-COCO and Flickr30K datasets, demonstrating its superiority over recent state-of-the-art methods. Copyright © 2025, The Authors. All rights reserved.

关键词： Semantics

Tensor parametric Hamiltonian operator inference

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Vijaywargiya, Arjun McQuarrie, Shane A. Gruber, Anthony Department of Applied and Computational Mathematics and Statistics University of Notre Dame United States Computational Mathematics Center for Computing Research Sandia National Laboratories Scientific Machine Learning Center for Computing Research Sandia National Laboratories

This work presents a tensor-based approach to constructing data-driven reduced-order models corresponding to semi-discrete partial differential equations with canonical Hamiltonian structure. By expressing parameter-varying operators with affine dependence as contractions of a generalized parameter vector against a constant tensor, this method leverages the operator inference framework to capture parametric dependence in the learned reduced-order model via the solution to a convex, least-squares optimization problem. This leads to a concise and straightforward implementation which compactifies previous parametric operator inference approaches and directly extends to learning parametric operators with symmetry constraints—a key feature required for constructing structure-preserving surrogates of Hamiltonian systems. The proposed approach is demonstrated on both a (non-Hamiltonian) heat equation with variable diffusion coefficient as well as a Hamiltonian wave equation with variable wave *** Codes 15A69, 35B30, 65P10 © 2025, CC BY.

关键词： Tensors

TeMP-TraG: Edge-based Temporal Message Passing in Transaction Graphs

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Gounoue, Steve Sao, Ashutosh Gottschalk, Simon University of Bonn Bonn Germany Lamarr Institute for Machine Learning and Artificial Intelligence Bonn Germany L3S Research Center Hannover Germany

Transaction graphs, which represent financial and trade transactions between entities such as bank accounts and companies, can reveal patterns indicative of financial crimes like money laundering and fraud. However, effective detection of such cases requires node and edge classification methods capable of addressing the unique challenges of transaction graphs, including rich edge features, multigraph structures and temporal dynamics. To tackle these challenges, we propose TeMP-TraG, a novel graph neural network mechanism that incorporates temporal dynamics into message passing. TeMP-TraG prioritises more recent transactions when aggregating node messages, enabling better detection of time-sensitive patterns. We demonstrate that TeMP-TraG improves four state-of-the-art graph neural networks by 6.19% on average. Our results highlight TeMP-TraG as an advancement in leveraging transaction graphs to combat financial crime. Copyright © 2025, The Authors. All rights reserved.

关键词： Graph neural networks