检索结果-内蒙古大学图书馆

arXiv 2025年

作者： Chen, Yiming Zhang, Yuan Liu, Yin Yuan, Kun Wen, Zaiwen Beijing International Center for Mathematical Research Peking University Beijing China Center for Data Science Peking University Beijing China Center for Machine Learning Research Peking University Beijing China

The memory challenges associated with training Large Language Models (LLMs) have become a critical concern, particularly when using the Adam optimizer. To address this issue, numerous memory-efficient techniques have been proposed, with GaLore standing out as a notable example designed to reduce the memory footprint of optimizer states. However, these approaches do not alleviate the memory burden imposed by activations, rendering them unsuitable for scenarios involving long context sequences or large mini-batches. Moreover, their convergence properties are still not well-understood in the literature. In this work, we introduce a Randomized Subspace Optimization framework for pre-training and fine-tuning LLMs. Our approach decomposes the high-dimensional training problem into a series of lower-dimensional subproblems. At each iteration, a random subspace is selected, and the parameters within that subspace are optimized. This structured reduction in dimensionality allows our method to simultaneously reduce memory usage for both activations and optimizer states. We establish comprehensive convergence guarantees and derive rates for various scenarios, accommodating different optimization strategies to solve the subproblems. Extensive experiments validate the superior memory and communication efficiency of our method, achieving performance comparable to GaLore and Adam. Copyright © 2025, The Authors. All rights reserved.

关键词：

来源：评论

学校读者我要写书评

暂无评论

EPR-Net: constructing a non-equilibrium potential landscape via a variational force projection formulation

引用

National Science Review 2024年第7期11卷 135-147页

作者： Yue Zhao Wei Zhang Tiejun Li Center for Data Science Peking University Zuse Institute Berlin Department of Mathematics and Computer Science Freie Universit?t Berlin Laboratory of Mathematics and Applied Mathematics(LMAM)and School of Mathematical Sciences Peking University Center for Machine Learning Research Peking University

We present EPR-Net, a novel and effective deep learning approach that tackles a crucial challenge in biophysics: constructing potential landscapes for high-dimensional non-equilibrium steady-state ***-Net leverages a nice mathematical fact that the desired negative potential gradient is simply the orthogonal projection of the driving force of the underlying dynamics in a weighted inner-product ***, our loss function has an intimate connection with the steady entropy production rate(EPR),enabling simultaneous landscape construction and EPR estimation. We introduce an enhanced learning strategy for systems with small noise, and extend our framework to include dimensionality reduction and the state-dependent diffusion coefficient case in a unified fashion. Comparative evaluations on benchmark problems demonstrate the superior accuracy, effectiveness and robustness of EPR-Net compared to existing methods. We apply our approach to challenging biophysical problems, such as an eight-dimensional(8D)limit cycle and a 52D multi-stability problem, which provide accurate solutions and interesting insights on constructed landscapes. With its versatility and power, EPR-Net offers a promising solution for diverse landscape construction problems in biophysics.

关键词： high-dimensional potential landscape non-equilibrium system entropy production rate dimensionality reduction deep learning

来源：评论

学校读者我要写书评

暂无评论

Method for Finding an Investment Strategy in the Case of a Sparse Covariance Matrix

Method for Finding an Investment Strategy in the Case of a S...

引用

International Conference on Management of Large-Scale System Development (MLSD)

作者： Victor Gorelik Tatiana Zolotova Department of Simulation Systems and Operations Research Federal Research Center “Computer Science and Control” of the Russian Academy of Sciences Moscow Russia Department of Data Analysis and Machine Learning Financial University under the Government of the Russian Federation Moscow Russia

An optimality principle is proposed for making investment decisions based on efficiency and risk assessments with a sparse covariance matrix. The method is implemented as a program with a graphical interface and demon... 详细信息

ISBN: (数字)9798350375718

ISBN: (纸本)9798350375725

关键词： Large-scale systems Sparse matrices Risk management Covariance matrices Stock markets Portfolios Investment

来源：评论

学校读者我要写书评

暂无评论

How Transformers Get Rich: Approximation and Dynamics analysis

arXiv

引用

arXiv 2024年

作者： Wang, Mingze Yu, Ruoxi Weinan, E. Wu, Lei School of Mathematical Sciences Peking University China School of Mathematical Sciences Center for Machine Learning Research AI for Science Institute Peking University China Center for Data Science Peking University China School of Mathematical Sciences Center for Machine Learning Research Peking University China

Transformers have demonstrated exceptional in-context learning capabilities, yet the theoretical understanding of the underlying mechanisms remains limited. A recent work (Elhage et al., 2021) identified a "rich" in-context mechanism known as induction head, contrasting with "lazy" n-gram models that overlook long-range dependencies. In this work, we provide both approximation and dynamics analyses of how transformers implement induction heads. In the approximation analysis, we formalize both standard and generalized induction head mechanisms, and examine how transformers can efficiently implement them, with an emphasis on the distinct role of each transformer submodule. For the dynamics analysis, we study the training dynamics on a synthetic mixed target, composed of a 4-gram and an in-context 2-gram component. This controlled setting allows us to precisely characterize the entire training process and uncover an abrupt transition from lazy (4-gram) to rich (induction head) mechanisms as training progresses. Copyright © 2024, The Authors. All rights reserved.

关键词： Distribution transformers

来源：评论

学校读者我要写书评

暂无评论

Multi-Scale Clinical-Guided Binocular Fusion Framework for Predicting New-Onset Hypertension Over a Four-Year Period

Multi-Scale Clinical-Guided Binocular Fusion Framework for P...

引用

IEEE International Symposium on Biomedical Imaging

作者： Haoshen Li Zifan Chen Jie Zhao Heyun Chen Hexin Dong Mingze Yuan Bin Dong Li Zhang Center for Data Science Peking University China National Engineering Laboratory for Big Data Analysis and Applications Peking University China Peking University Changsha Institute for Computing and Digital Economy China Beijing International Center for Mathematical Research Peking University China Center for Machine Learning Research Peking University China

ISBN: (数字)9798350313338

ISBN: (纸本)9798350313345

Hypertension is a major global health concern, linked to various cardiovascular diseases and associated with distinct ocular manifestations. While recent advances in artificial intelligence have enabled accurate diagnosis of current hypertension through fundus images, predicting the future onset of hypertension remains an uncharted domain. In this study, we introduce the multi-scale clinical-guided binocular fusion framework (MCBO), designed to predict the likelihood of developing hypertension within the next four years. MCBO uniquely integrates left and right fundus images and clinical data, utilizing a shared-weight multi-stage Transformer-based encoder. Our multi-scale clinical-guided module (MCM) ensures image feature extraction is clinically contextualized based on clinical information, and our binocular fusion module (BFM) fuses binocular information. Comparative performance against seven baseline models establishes MCBO’s supremacy, with improvements of 6.7% in Area Under Curve (AUC), 6.9% in Accuracy (ACC), 5.1% in Sensitivity (SEN) and 5.5% in Specificity (SPE). This approach offers a promising avenue for proactive hypertension management, underscoring the potential of integrating Deep learning with clinical data for enhanced healthcare outcomes. Our code is available at https://***/HaoshenLi/MCBO.

关键词： Hypertension Deep learning Accuracy Sensitivity Codes Fuses Feature extraction

来源：评论

学校读者我要写书评

暂无评论

Revealing excited states of rotational Bose-Einstein condensates

引用

The Innovation 2024年第1期5卷 41-48页

作者： Jianyuan Yin Zhen Huang Yongyong Cai Qiang Du Lei Zhang School of Mathematical Sciences Laboratory of Mathematics and Applied MathematicsPeking UniversityBeijing 100871China Department of Mathematics National University of SingaporeSingapore 119076Singapore Department of Mathematics University of CaliforniaBerkeleyBerkeleyCA 94720USA School of Mathematical Sciences Beijing Normal UniversityBeijing 100875China Department of Applied Physics and Applied Mathematics and Data Science Institute Columbia UniversityNew YorkNY 10027USA Beijing International Center for Mathematical Research Center for Quantitative BiologyCenter for Machine Learning ResearchPeking UniversityBeijing 100871China

Rotational Bose-Einstein condensates can exhibit quantized vortices as topological *** this study,the ground and excited states of the rotational Bose-Einstein condensates are systematically studied by calculating the stationary points of the Gross-Pitaevskii energy *** excited states and their connections at different rotational frequencies are revealed in solution landscapes constructed with the constrained high-index saddle dynamics *** excitation mechanisms are identified:vortex addition,rearrangement,merging,and *** demonstrate changes in the ground state with increasing rotational frequencies and decipher the evolution of the stability of ground states.

关键词： method rotational excited

来源：评论

学校读者我要写书评

暂无评论

ENHANCING ZEROTH-ORDER FINE-TUNING FOR LANGUAGE MODELS WITH LOW-RANK STRUCTURES

arXiv

引用

arXiv 2024年

作者： Chen, Yiming Zhang, Yuan Cao, Liyuan Yuan, Kun Wen, Zaiwen Beijing International Center for Mathematical Research Peking University Beijing China Center for Data Science Peking University Beijing China Center for Machine Learning Research Peking University Beijing China

Parameter-efficient fine-tuning (PEFT) significantly reduces memory costs when adapting large language models (LLMs) for downstream applications. However, traditional first-order (FO) fine-tuning algorithms incur substantial memory overhead due to the need to store activation values for back-propagation during gradient computation, particularly in long-context fine-tuning tasks. Zeroth-order (ZO) algorithms offer a promising alternative by approximating gradients using finite differences of function values, thus eliminating the need for activation storage. Nevertheless, existing ZO methods struggle to capture the low-rank gradient structure common in LLM fine-tuning, leading to suboptimal performance. This paper proposes a low-rank ZO gradient estimator and introduces a novel low-rank ZO algorithm (LOZO) that effectively captures this structure in LLMs. We provide convergence guarantees for LOZO by framing it as a subspace optimization method. Additionally, its low-rank nature enables LOZO to integrate with momentum techniques while incurring negligible extra memory costs. Extensive experiments across various model sizes and downstream tasks demonstrate that LOZO and its momentum-based variant outperform existing ZO methods and closely approach the performance of FO algorithms. Copyright © 2024, The Authors. All rights reserved.

关键词：

来源：评论

学校读者我要写书评

暂无评论

Thai Conversational Chatbot Classification Using BiLSTM and data Augmentation 1st

Thai Conversational Chatbot Classification Using BiLSTM and ...

引用

1st International Conference on data Science and Artificial Intelligence, DSAI 2023

作者： Lhasiw, Nunthawat Tanantong, Tanatorn Sanglerdsinlapachai, Nuttapong Thammasat Research Unit in Data Innovation and Artificial Intelligence Department of Computer Science Faculty of Science and Technology Thammasat University Pathum Thani Thailand Strategic Analytics Networks with Machine Learning and AI Research Team National Electronics and Computer Technology Center Pathum Thani Thailand

ISBN: (纸本)9789819979684

Chatbot platforms, e.g., Facebook and Line, have revolutionized human interaction in the digital age. In order to develop an automatic chatbot classification, there are several challenges especially for Thai chat messages. Conversational messages are usually short and ambiguous. Therefore, it is difficult to find a dataset for constructing an effective classification model. To address the limited size of the dataset, data augmentation techniques can be possibly applied. data augmentation involves generating synthetic messages by applying various transformations to existing data samples while preserving their original meaning. In this study, the size and diversity of the dataset is increased by two methods, i.e., text augmentation using word2vec from Thai2Fit and English-Thai machine translation models proposed by VISTEC. Based on the augmented messages, a Deep learning technique, BiLSTM, is used to construct a chatbot classification model. The experimental obtained results demonstrate that data augmentation can help to increase the classification performance. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd 2023.

关键词： Classification (of information)

来源：评论

学校读者我要写书评

暂无评论

Deliberative XAI: How Explanations Impact Understanding and Decision-Making of AI Novices in Collective and Individual Settings

arXiv

引用

arXiv 2024年

作者： Schmude, Timothée Koesten, Laura Möller, Torsten Tschiatschek, Sebastian University of Vienna Faculty of Computer Science Research Network Data Science Doctoral School Computer Science Austria University of Vienna Faculty of Computer Science Research Group Visualization and Data Analysis Austria University of Vienna Faculty of Computer Science Research Group Visualization and Data Analysis Research Network Data Science Austria University of Vienna Faculty of Computer Science Research Network Data Science Research Group Data Mining and Machine Learning Austria

XAI research often focuses on settings where people learn about and assess algorithmic systems individually. However, as more public AI systems are deployed, it becomes essential for XAI to facilitate collective understanding and deliberation. We conducted a task-based interview study involving 8 focus groups and 12 individual interviews to explore how explanations can support AI novices in understanding and forming opinions about AI systems. Participants received a collection of explanations organized into four information categories to solve tasks and decide about a system’s deployment. These explanations improved or calibrated participants’ self-reported understanding and decision confidence and facilitated group discussions. Participants valued both technical and contextual information and the self-directed and modular explanation structure. Our contributions include an explanation approach that facilitates both individual and collaborative interaction and explanation design recommendations, including active and controllable exploration, different levels of information detail and breadth, and adaptations to the needs of decision subjects. Copyright © 2024, The Authors. All rights reserved.

关键词：

来源：评论

学校读者我要写书评

暂无评论

NC-ALG: Graph-Based Active learning under Noisy Crowd 40

NC-ALG: Graph-Based Active Learning under Noisy Crowd

引用

40th IEEE International Conference on data Engineering, ICDE 2024

作者： Zhang, Wentao Wang, Yexin You, Zhenbang Li, Yang Cao, Gang Yang, Zhi Cui, Bin Center for Machine Learning Research Peking University China Key Lab of High Confidence Software Technologies Peking University China Institute of Advanced Algorithms Research Shanghai China Institute of Computational Social Science Peking University Qingdao China National Engineering Labratory for Big Data Analytics and Applications China TEG Tencent Inc. Department of Data Platform China Beijing Academy of Artificial Intelligence China

ISBN: (纸本)9798350317152

Graph Neural Networks (GNNs) have achieved great success in various data mining tasks but they heavily rely on a large number of annotated nodes, requiring considerable human efforts. Despite the effectiveness of existing GNN-based Active learning (AL) methods, they assume that the annotated labels are always correct, which is contradictory to the error-prone labeling process in a practical crowdsourcing environment. Besides, due to this impractical assumption, existing works only focus on optimizing the node selection in AL but neglect optimizing the labeling process. Therefore, we present NC-ALG, the first GNN-based AL framework that optimizes both the node selection and node labeling process under a noisy crowd. For node selection, NC-ALG introduces a new measurement to model influence reliability and an effective influence maximization objective to select nodes. For node labeling, NC-ALG significantly reduces the labeling cost by considering the model-predicted labels and the labels of mirror nodes. To the best of our knowledge, this is the first attempt to consider GNN-based AL under the practical noisy crowd. Empirical studies on public datasets demonstrate that NC-ALG significantly outperforms existing methods in terms labeling efficiency. Notably, it only takes NC-ALG one-third of the labeling budget that the competitive baseline GRAIN needs to achieve an accuracy of 70.7 % on PubMed. © 2024 IEEE.

关键词： Crowdsourcing

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：