检索结果-内蒙古大学图书馆

Training large-scale language models with limited GPU memory:a survey

Frontiers of Information Technology & Electronic Engineering 2025年第3期26卷 309-331页

作者： Yu TANG Linbo QIAO Lujia YIN Peng LIANG Ao SHEN Zhilin YANG Lizhi ZHANG Dongsheng LI National Key Laboratory of Parallel and Distributed Computing College of ComputerNational University of Defense TechnologyChangsha 410073China

Large-scale models have gained significant attention in a wide range of fields,such as computer vision and natural language processing,due to their effectiveness across various ***,a notable hurdle in training these large-scale models is the limited memory capacity of graphics processing units(GPUs).In this paper,we present a comprehensive survey focused on training large-scale models with limited GPU *** exploration commences by scrutinizing the factors that contribute to the consumption of GPU memory during the training process,namely model parameters,model states,and model *** this analysis,we present an in-depth overview of the relevant research work that addresses these aspects ***,the paper concludes by presenting an outlook on the future of memory optimization in training large-scale language models,emphasizing the necessity for continued research and innovation in this *** survey serves as a valuable resource for researchers and practitioners keen on comprehending the challenges and advancements in training large-scale language models with limited GPU memory.

关键词： Training techniques Memory optimization Model parameters Model states Model activations

来源：评论

学校读者我要写书评

暂无评论

Leaders and Collaborators: Addressing Sparse Reward Challenges in Multi-Agent Reinforcement Learning

引用

IEEE TRANSACTIONS ON EMERGING TOPICS IN COMPUTATIONAL INTELLIGENCE 2025年第2期9卷 1976-1989页

作者： Sun, Shaoqi Liu, Hui Xu, Kele Ding, Bo Natl Univ Def Technol Natl Key Lab Parallel & Distributed Proc Changsha 410003 Peoples R China

Cooperative multi-agent reinforcement learning (MARL) has emerged as an effective tool for addressing complex control tasks. However, sparse team rewards present significant challenges for MARL, leading to low exploration efficiency, slow learning speed, and homogenized behaviors among agents. To address these issues, we propose a novel Leader-Collaborator (LC) MARL framework inspired by human social collaboration. The LC framework introduces parallel online knowledge distillation for policy networks (KDPN). KDPN extracts knowledge from two policy networks with different training objectives: one aims to maximize individual rewards, while the other aims to maximize team rewards. The extracted knowledge is utilized to construct team leaders and collaborators. By effectively balancing individual and team rewards, our approach enhances exploration efficiency and promotes behavioral diversity among agents. This addresses the issue of low learning efficiency caused by the lack of objectives early in the agent's learning process and facilitates the development of more effective and differentiated team interaction policies. Additionally, we present the Self-Repairing Strategy (SRS) and Self-Augmenting Strategy (SAS) to facilitate team policies learning while preserving the initial team goal. We evaluate the effectiveness of the LC framework by conducting extensive experiments on the Multi-Agent Particle Environment (MPE), the Google Research Football (GRF), and StarCraft Multi-Agent Challenge (SMAC) with varying levels of difficulty. Our experimental results demonstrate that LC significantly improves the efficiency of the agent's exploration, achieves state-of-the-art performance, and accelerates the learning of the optimal policy. Specifically, in the SMAC scenarios, our method increases the winning rate by 21.9%, increases the average cumulative reward by 12%, and reduces the training time by 57% to achieve optimal performance.

关键词： Computational modeling Training Reinforcement learning Sports Knowledge engineering Computational intelligence Teamwork Sun Standards Predictive models Multi-agent reinforcement learning sparse rewards online knowledge distillation exploration efficiency

来源：评论

学校读者我要写书评

暂无评论

A parallel Radial Basis Probabilistic Neural Network for scalable data mining in distributed memory machines

A Parallel Radial Basis Probabilistic Neural Network for sca...

引用

IEEE 24th International Conference on Tools with Artificial Intelligence (ICTAI)

作者： Kokkinos, Yiannis Margaritis, Konstantinos Univ Macedonia Parallel & Distributed Proc Lab Dept Appl Informat Thessaloniki 54006 Greece

ISBN: (纸本)9780769549156;9781479902279

This work presents scalable algorithms for basic construction of parallel Radial Basis Probabilistic Neural Networks. The final goal is to build a neural network that can efficiently be implemented in distributed memory machines. Thus a fast simple parallel training scheme for RBPNNs is studied, that is based almost solely on Gaussian summations which can by their part be efficiently mapped on parallel as well as on pipeline distributed machines. The suggested training scheme is tested for accuracy and performance and can guarantee simplicity, parallelization and linear speed ups in common parallel implementations, namely neuron parallel and pipelining studied here.

关键词： parallel processing Radial Basis Probabilistic Neural networks Large Scale

来源：评论

学校读者我要写书评

暂无评论

Memory-efficient tensor parallelism for long-sequence Transformer training

引用

FRONTIERS OF INFORMATION TECHNOLOGY & ELECTRONIC ENGINEERING 2025年 1-18页

作者： Liang, Peng Qiao, Linbo Shi, Yanqi Zheng, Hao Tang, Yu Li, Dongsheng Natl Univ Def Technol Coll Comp Sci & Technol Natl Key Lab Parallel & Distributed Comp Changsha 410073 Peoples R China

Transformer-based models like large language models (LLMs) have attracted significant attention in recent years due to their superior performance. A long sequence of input tokens is essential for industrial LLMs to provide better user services. However, memory consumption increases quadratically with the increase of sequence length, posing challenges for scaling up long-sequence training. Current parallelism methods produce duplicated tensors during execution, leaving space for improving memory efficiency. Additionally, tensor parallelism (TP) cannot achieve effective overlap between computation and communication. To solve these weaknesses, we propose a general parallelism method called memory-efficient tensor parallelism (METP), designed for the computation of two consecutive matrix multiplications and a possible function between them (O = f(AB)C), which is the kernel computation component in Transformer training. METP distributes subtasks of computing O to multiple devices and uses send/recv instead of collective communication to exchange submatrices for finishing the computation, avoiding producing duplicated tensors. We also apply the double buffering technique to achieve better overlap between computation and communication. We present the theoretical condition of full overlap to help instruct the long-sequence training of Transformers. Suppose the parallel degree is p;through theoretical analysis, we prove that METP provides O(1/p(3)) memory overhead when not using FlashAttention to compute attention and could save at least 41.7% memory compared to TP when using FlashAttention to compute multi-head self-attention. Our experimental results demonstrate that METP can increase the sequence length by 2.38-2.99 times compared to other methods when using eight A100 graphics processing units (GPUs).

关键词： distributed learning Large language model (LLM) Long sequence Machine learning system Memory efficiency Tensor parallelism TP183

来源：评论

学校读者我要写书评

暂无评论

DPFS: A distributed parallel File System

DPFS: A Distributed Parallel File System

引用

30th International Conference on parallel Processing (ICPP 01)

作者： Shen, XH Choudhary, A Northwestern Univ Dept Elect & Comp Engn Ctr Parallel & Distributed Comp Evanston IL 60208 USA

ISBN: (纸本)0769512577;0769512585

One of challenges brought by large-scale scientific applications is how to avoid remote storage access by collectively using enough local storage resources to hold huge amount of data generated by the simulation while providing high performance I/O. DPFS, a distributed parallel File System, is designed and implemented to address this problem. DPFS collects locally distributed unused storage resources as a supplement to the internal storage of parallel computing systems to satisfy the storage capacity requirement of large-scale applications. In addition, like parallel file systems, DPFS provides striping mechanisms that divides a file into small pieces and distributes them across multiple storage devices for parallel data access. The unique feature of DPFS is that it provides three file levels with each file level corresponding to a file striping method. In addition to the traditional linear striping method, DPFS also provides a novel Multidimensional striping method that can solve performance problems of linear striping for many popular access patterns. Other issues such as load-balanceing and user interface are also addressed in DPFS.

关键词： Computational modeling Computer simulation distributed computing distributed power generation Ear File systems Large-scale systems Multidimensional systems parallel processing Spatial databases

来源：评论

学校读者我要写书评

暂无评论

A prediction-based parallel replication algorithm in distributed storage system

引用

4th International Conference on Grid and Cooperative Computing - GCC 2005

作者： Wang, Yijie Zhang, Xiaoming National Laboratory for Parallel and Distributed Processing Institute of Computer National University of Defense Technology Changsha 410073 China

ISBN: (纸本)3540305106

Data replication can be used to reduce bandwidth consumption and access latency in the distributed system where users require remote access to large data objects. In this paper, according to the intrinsic characteristic of distributed storage system, the prediction-based parallel replication algorithm PPR is proposed. In the PPR, according to the characteristic of spatial data, the data that will be accessed is predicted, then the data is prefetched;during replication, according to the network state, several replicas of a data object are selected, which are of the least access cost;the different parts of the data object are transferred from these replicas, and they are used to make a new replica. The results of performance evaluation show that the PPR can utilize the network bandwidth efficiently, provide high data replication efficiency and substantially better access efficiency, and can avoid the interference between different replications efficiently. © Springer-Verlag Berlin Heidelberg 2005.

关键词： distributed computer systems

来源：评论

学校读者我要写书评

暂无评论

An intelligent mesh-smoothing method with graph neural networks

引用

Frontiers of Information Technology & Electronic Engineering 2025年第3期26卷 367-384页

作者： Zhichao WANG Xinhai CHEN Junjun YAN Jie LIU Science and Technology on Parallel and Distributed Processing Laboratory National University of Defense TechnologyChangsha 410073China Laboratory of Digitizing Software for Frontier Equipment National University of Defense TechnologyChangsha 410073China

In computational fluid dynamics(CFD),mesh-smoothing methods are widely used to refine the mesh quality for achieving high-precision numerical ***,optimization-based smoothing is used for high-quality mesh smoothing,but it incurs significant computational *** works have improved its smoothing efficiency by adopting supervised learning to learn smoothing methods from high-quality ***,they pose difficulties in smoothing the mesh nodes with varying degrees and require data augmentation to address the node input sequence ***,the required labeled high-quality meshes further limit the applicability of the proposed *** this paper,we present graph-based smoothing mesh net(GMSNet),a lightweight neural network model for intelligent mesh *** adopts graph neural networks(GNNs)to extract features of the node’s neighbors and outputs the optimal node *** smoothing,we also introduce a fault-tolerance mechanism to prevent GMSNet from generating negative volume *** a lightweight model,GMSNet can effectively smooth mesh nodes with varying degrees and remain unaffected by the order of input data.A novel loss function,MetricLoss,is developed to eliminate the need for high-quality meshes,which provides stable and rapid convergence during *** compare GMSNet with commonly used mesh-smoothing methods on two-dimensional(2D)triangle *** results show that GMSNet achieves outstanding mesh-smoothing performances with 5%of the model parameters compared to the previous model,but offers a speedup of 13.56 times over the optimization-based smoothing.

关键词： Unstructured mesh Mesh smoothing Graph neural network Optimization-based smoothing

来源：评论

学校读者我要写书评

暂无评论

NSC-YOLOv8: A Small Target Detection Method for UAV-Acquired Images Based on Self-Adaptive Embedding

引用

ELECTRONICS 2025年第8期14卷 1548-1548页

作者： Chen, Dongmin Chen, Danyang Zhong, Cheng Zhan, Feng Guangxi Univ Sch Comp Elect & Informat Nanning 530004 Peoples R China Key Lab Parallel Distributed & Intelligent Comp Gu Nanning 530004 Peoples R China

Existing drone image processing algorithms for small target detection in Unmanned Aerial Vehicle (UAV) aerial images struggle with challenges like missed detection of small objects, information loss from downsampling, loss of low-dimensional features, and information drop of contextual features. In order to alleviate the four problems just mentioned, we propose a self-adaptive small target detection method, NSC-YOLOv8, based on the YOLOv8 model. First, we introduce a small target detection head that enhances the model's ability to fuse shallow and deep features, effectively handling low-pixel targets. Second, a Non-lossy Downsampling Block (NDB) is introduced into the backbone, which optimizes the detection accuracy of small targets in large scenes through dimensional transformation. In addition, we introduce a Self-Adaptive Embedding Block (SAEB) based on low-dimensional information, which enhances the comprehensive performance of the model by expanding the local sensing field to enhance the focus on important contextual information. Finally, we design a Content-Aware Resampling Block (CARB), which is able to enhance the model's ability to recognize small targets by resampling low-dimensional features. Experiments on the VisDrone2019-DET dataset show that NSC-YOLOv8s improves target detection accuracy over YOLOv8s, with an 11.7% increase in mAP@0.5. Additionally, removing a large detection head and adjusting the bottom-up layers reduces NSC-YOLOv8s' parameters by 1.2 M compared to YOLOv8s. Therefore, NSC-YOLOv8 shows better performance in small target detection for UAV imagery.

关键词： small target detection YOLOv8 UAV aerial images

来源：评论

学校读者我要写书评

暂无评论

Few-Shot Object Detection for Remote Sensing Images via Pseudo-Sample Generation and Feature Enhancement

引用

APPLIED SCIENCES-BASEL 2025年第8期15卷 4477-4477页

作者： Huang, Zhaoguo Chen, Danyang Zhong, Cheng Guangxi Univ Sch Comp Elect & Informat Nanning 530004 Peoples R China Key Lab Parallel Distributed & lntelligent Comp Gu Nanning 530004 Peoples R China

Few-shot object detection (FSOD) based on fine-tuning is essential for analyzing optical remote sensing images. However, existing methods mainly focus on natural images and overlook the scale variations in remote sensing images, leading to feature confusion among foreground instances of different classes. Additionally, since only a subset of instances are labeled in FSOD training data, the model might mistakenly treat unlabeled instances as background, leading to confusion between foreground features and background features, particularly those of novel classes. The preceding phenomenon indicates that severe feature confusion in remote sensing FSOD hampers the ability of the model to accurately classify and localize instances. To address these issues, this paper proposes a two-stage FSOD framework based on transfer learning via pseudo-sample generation and feature enhancement (PSGFE), including pseudo-sample generation module (PSGM) and feature enhancement module (FEM). The former reduces the feature confusion between foreground and background by generating pseudo-samples for unannotated background areas. The latter dynamically captures and enhances multi-scale features on the region of interest (ROI), and extracts unique core information for each class to eliminate the feature confusion among foreground instances of different classes. Our method has been validated on the optical remote sensing datasets DIOR and RSOD. It demonstrates superior performance compared to existing methods.

关键词： few-shot object detection remote sensing images pseudo-sample generation feature enhancement

来源：评论

学校读者我要写书评

暂无评论

Optimizing Fine-Tuning in Quantized Language Models:An In-Depth Analysis of Key Variables

引用

Computers, Materials & Continua 2025年第1期82卷 307-325页

作者： Ao Shen Zhiquan Lai Dongsheng Li Xiaoyu Hu National Key Laboratory of Parallel and Distributed Computing National University of Defense TechnologyChangsha410073China Strategic Assessments and Consultation Institute Academy of Military ScienceBeijing100091China

Large-scale Language Models(LLMs)have achieved significant breakthroughs in Natural Language Processing(NLP),driven by the pre-training and fine-tuning *** this approach allows models to specialize in specific tasks with reduced training costs,the substantial memory requirements during fine-tuning present a barrier to broader ***-Efficient Fine-Tuning(PEFT)techniques,such as Low-Rank Adaptation(LoRA),and parameter quantization methods have emerged as solutions to address these challenges by optimizing memory usage and computational *** these,QLoRA,which combines PEFT and quantization,has demonstrated notable success in reducing memory footprints during fine-tuning,prompting the development of various QLoRA *** these advancements,the quantitative impact of key variables on the fine-tuning performance of quantized LLMs remains *** study presents a comprehensive analysis of these key variables,focusing on their influence across different layer types and depths within LLM *** investigation uncovers several critical findings:(1)Larger layers,such as MLP layers,can maintain performance despite reductions in adapter rank,while smaller layers,like self-attention layers,aremore sensitive to such changes;(2)The effectiveness of balancing factors depends more on specific values rather than layer type or depth;(3)In quantization-aware fine-tuning,larger layers can effectively utilize smaller adapters,whereas smaller layers struggle to do *** insights suggest that layer type is a more significant determinant of fine-tuning success than layer depth when optimizing quantized ***,for the same discount of trainable parameters,reducing the trainable parameters in a larger layer is more effective in preserving fine-tuning accuracy than in a smaller *** study provides valuable guidance for more efficient fine-tuning strategies and opens avenues for further research into optimizing LLM

关键词： Large-scale Language Model Parameter-Efficient Fine-Tuning parameter quantization key variable trainable parameters experimental analysis

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：