检索结果-内蒙古大学图书馆

Memory-efficient tensor parallelism for long-sequence Transformer training

FRONTIERS OF INFORMATION TECHNOLOGY & ELECTRONIC ENGINEERING 2025年 1-18页

作者： Liang, Peng Qiao, Linbo Shi, Yanqi Zheng, Hao Tang, Yu Li, Dongsheng Natl Univ Def Technol Coll Comp Sci & Technol Natl Key Lab Parallel & Distributed Comp Changsha 410073 Peoples R China

Transformer-based models like large language models (LLMs) have attracted significant attention in recent years due to their superior performance. A long sequence of input tokens is essential for industrial LLMs to provide better user services. However, memory consumption increases quadratically with the increase of sequence length, posing challenges for scaling up long-sequence training. Current parallelism methods produce duplicated tensors during execution, leaving space for improving memory efficiency. Additionally, tensor parallelism (TP) cannot achieve effective overlap between computation and communication. To solve these weaknesses, we propose a general parallelism method called memory-efficient tensor parallelism (METP), designed for the computation of two consecutive matrix multiplications and a possible function between them (O = f(AB)C), which is the kernel computation component in Transformer training. METP distributes subtasks of computing O to multiple devices and uses send/recv instead of collective communication to exchange submatrices for finishing the computation, avoiding producing duplicated tensors. We also apply the double buffering technique to achieve better overlap between computation and communication. We present the theoretical condition of full overlap to help instruct the long-sequence training of Transformers. Suppose the parallel degree is p;through theoretical analysis, we prove that METP provides O(1/p(3)) memory overhead when not using FlashAttention to compute attention and could save at least 41.7% memory compared to TP when using FlashAttention to compute multi-head self-attention. Our experimental results demonstrate that METP can increase the sequence length by 2.38-2.99 times compared to other methods when using eight A100 graphics processing units (GPUs).

关键词： distributed learning Large language model (LLM) Long sequence Machine learning system Memory efficiency Tensor parallelism TP183

来源：评论

学校读者我要写书评

暂无评论

Leaders and Collaborators: Addressing Sparse Reward Challenges in Multi-Agent Reinforcement Learning

引用

IEEE TRANSACTIONS ON EMERGING TOPICS IN compUTATIONAL INTELLIGENCE 2025年第2期9卷 1976-1989页

作者： Sun, Shaoqi Liu, Hui Xu, Kele Ding, Bo Natl Univ Def Technol Natl Key Lab Parallel & Distributed Proc Changsha 410003 Peoples R China

Cooperative multi-agent reinforcement learning (MARL) has emerged as an effective tool for addressing complex control tasks. However, sparse team rewards present significant challenges for MARL, leading to low exploration efficiency, slow learning speed, and homogenized behaviors among agents. To address these issues, we propose a novel Leader-Collaborator (LC) MARL framework inspired by human social collaboration. The LC framework introduces parallel online knowledge distillation for policy networks (KDPN). KDPN extracts knowledge from two policy networks with different training objectives: one aims to maximize individual rewards, while the other aims to maximize team rewards. The extracted knowledge is utilized to construct team leaders and collaborators. By effectively balancing individual and team rewards, our approach enhances exploration efficiency and promotes behavioral diversity among agents. This addresses the issue of low learning efficiency caused by the lack of objectives early in the agent's learning process and facilitates the development of more effective and differentiated team interaction policies. Additionally, we present the Self-Repairing Strategy (SRS) and Self-Augmenting Strategy (SAS) to facilitate team policies learning while preserving the initial team goal. We evaluate the effectiveness of the LC framework by conducting extensive experiments on the Multi-Agent Particle Environment (MPE), the Google Research Football (GRF), and StarCraft Multi-Agent Challenge (SMAC) with varying levels of difficulty. Our experimental results demonstrate that LC significantly improves the efficiency of the agent's exploration, achieves state-of-the-art performance, and accelerates the learning of the optimal policy. Specifically, in the SMAC scenarios, our method increases the winning rate by 21.9%, increases the average cumulative reward by 12%, and reduces the training time by 57% to achieve optimal performance.

关键词： computational modeling Training Reinforcement learning Sports Knowledge engineering computational intelligence Teamwork Sun Standards Predictive models Multi-agent reinforcement learning sparse rewards online knowledge distillation exploration efficiency

来源：评论

学校读者我要写书评

暂无评论

Efficient deep neural network training via decreasing precision with layer capacity

引用

FRONTIERS OF compUTER SCIENCE 2025年第10期19卷 1-17页

作者： Shen, Ao Lai, Zhiquan Sun, Tao Li, Shengwei Ge, Keshi Liu, Weijie Li, Dongsheng Natl Univ Def Technol Natl Key Lab Parallel & Distributed Comp Changsha 410073 Peoples R China Natl Univ Def Technol Coll Comp Changsha 410073 Peoples R China

Low-precision training has emerged as a practical approach, saving the cost of time, memory, and energy during deep neural networks (DNNs) training. Typically, the use of lower precision introduces quantization errors that need to be minimized to maintain model performance, often neglecting to consider the potential benefits of reducing training precision. This paper rethinks low-precision training, highlighting the potential benefits of lowering precision: (1) low precision can serve as a form of regularization in DNN training by constraining excessive variance in the model;(2) layer-wise low precision can be seen as an alternative dimension of sparsity, orthogonal to pruning, contributing to improved generalization in DNNs. Based on these analyses, we propose a simple yet powerful technique-DPC (Decreasing Precision with layer Capacity), which directly assigns different bit-widths to model layers, without the need for an exhaustive analysis of the training process or any delicate low-precision criteria. Thorough extensive experiments on five datasets and fourteen models across various applications consistently demonstrate the effectiveness of the proposed DPC technique in saving computational cost (-16.21%--44.37%) while achieving comparable or even superior accuracy (up to +0.68%, +0.21% on average). Furthermore, we offer feature embedding visualizations and conduct further analysis with experiments to investigate the underlying mechanisms behind DPC's effectiveness, enhancing our understanding of low-precision training. Our source code will be released upon paper acceptance.

关键词： low precision efficient training generalization regularization bit-width assignment

来源：评论

学校读者我要写书评

暂无评论

ER-SFM: Efficient and Robust Cluster-Based Structure from Motion 7th

ER-SFM: Efficient and Robust Cluster-Based Structure from Mo...

引用

7th Chinese Conference on Pattern Recognition and computer Vision

作者： Ye, Zongxin Li, Wenyu Liu, Sidun Qiao, Peng Dou, Yong Natl Univ Def Technol Coll Comp Natl Key Lab Parallel & Distributed Comp Changsha Peoples R China

ISBN: (纸本)9789819785070;9789819785087

Structure from Motion (SfM) is a fundamental computer vision technique that recovers scene structure and camera motion from multi-view images. When facing large-scale scenarios, cluster-based methods are commonly employed to improve reconstruction efficiency. However, these methods currently face challenges regarding their limited robustness, redundant computation, and drift. To address these issues, we propose a unified pipeline called ER-SfM, which enhances the three key aspects of cluster-based SfM: image clustering, local reconstruction, and merging. In terms of image clustering, we propose a three-stage image clustering method to ensure adequate and reliable connections between clusters. In the local reconstruction stage, we expedite the reconstruction process by eliminating duplicate point cloud computation. In the final merging stage, we introduce a global merging algorithm without scale ambiguity to address the drift problem. Extensive experimental results demonstrate the superior performance of our method in terms of both robustness and efficiency compared to state-of-the-art methods.

关键词： parallel structure from motion 3D Reconstruction Image clustering Global averaging

来源：评论

学校读者我要写书评

暂无评论

End-To-End High-Quality Transformer Object Detection Model Applied to Human Head Detection 7th

End-To-End High-Quality Transformer Object Detection Model A...

引用

7th Chinese Conference on Pattern Recognition and computer Vision

作者： Zhou, Zhen Li, Rongchun Qiao, Peng Jiang, Jingfei Natl Univ Def Technol Sch Comp Natl Key Lab Parallel & Distributed Comp Changsha 410073 Peoples R China

ISBN: (纸本)9789819788576;9789819788583

Head detection is a challenging and widely applied object detection task. Although previous CNN-based head detectors have made good progress, the inherent locality of CNN restricts the extraction of global contextual information, which leads to low precision and recall rates in head detection. In this article, we propose an end-to-end high-quality head detector based on Transformer, which effectively models the contextual relationships between heads, other objects and the background. To extract and generate discriminative feature maps suitable for detecting small head targets, we incorporate specific CNN-based auxiliary detector heads for joint training. The GIoU-aware classification loss function is improved to generate bounding boxes with high localization quality and high classification confidence, and a feature fusion module is introduced to enhance the feature representation capabilities of the model. We conduct experiments on COCO 2017 dataset and Brainwash head dataset, and the results demonstrate that our method outperforms in both COCO generalized object detection and Brainwash head detection tasks compared to previous CNN-based detectors as well as other current mainstream Transformer-based object detection models.

关键词： Head detection Vision transformer Auxiliary detector heads GIoU-aware Feature fusion

来源：评论

学校读者我要写书评

暂无评论

3DMeshNet: A three-dimensional differential neural network for structured mesh generation

引用

GRAPHICAL MODELS 2025年 139卷

作者： Peng, Jiaming Chen, Xinhai Liu, Jie Natl Univ Def Technol Lab Digitizing Software Frontier Equipment Changsha 410073 Peoples R China Natl Univ Def Technol Natl Key Lab Parallel & Distributed Comp Changsha 410073 Peoples R China

Mesh generation is a crucial step in numerical simulations, significantly impacting simulation accuracy and efficiency. However, generating meshes remains time-consuming and requires expensive computational resources. In this paper, we propose a novel method, 3DMeshNet, for three-dimensional structured mesh generation. The method embeds the meshing-related differential equations into the loss function of neural networks, formulating the meshing task as an unsupervised optimization problem. It takes geometric points as input to learn the potential mapping between parametric and computational spaces. After suitable offline training, 3DMeshNet can efficiently output a three-dimensional structured mesh with a user-defined number of quadrilateral/hexahedral cells through the feed-forward neural prediction. To enhance training stability and accelerate convergence, we integrate loss function reweighting through weight adjustments and gradient projection alongside applying finite difference methods to streamline derivative computations in the loss. Experiments on different cases show that 3DMeshNet is robust and fast. It outperforms neural network- based methods and yields superior meshes compared to traditional mesh partitioning methods. 3DMeshNet significantly reduces training times by up to 85% compared to other neural network-based approaches and lowers meshing overhead by 4 to 8 times relative to traditional meshing methods.

关键词： Mesh generation Three-dimensional structured mesh Differential equations Neural network

来源：评论

学校读者我要写书评

暂无评论

Few-Shot Object Detection for Remote Sensing Images via Pseudo-Sample Generation and Feature Enhancement

引用

APPLIED SCIENCES-BASEL 2025年第8期15卷 4477-4477页

作者： Huang, Zhaoguo Chen, Danyang Zhong, Cheng Guangxi Univ Sch Comp Elect & Informat Nanning 530004 Peoples R China Key Lab Parallel Distributed & lntelligent Comp Gu Nanning 530004 Peoples R China

Few-shot object detection (FSOD) based on fine-tuning is essential for analyzing optical remote sensing images. However, existing methods mainly focus on natural images and overlook the scale variations in remote sensing images, leading to feature confusion among foreground instances of different classes. Additionally, since only a subset of instances are labeled in FSOD training data, the model might mistakenly treat unlabeled instances as background, leading to confusion between foreground features and background features, particularly those of novel classes. The preceding phenomenon indicates that severe feature confusion in remote sensing FSOD hampers the ability of the model to accurately classify and localize instances. To address these issues, this paper proposes a two-stage FSOD framework based on transfer learning via pseudo-sample generation and feature enhancement (PSGFE), including pseudo-sample generation module (PSGM) and feature enhancement module (FEM). The former reduces the feature confusion between foreground and background by generating pseudo-samples for unannotated background areas. The latter dynamically captures and enhances multi-scale features on the region of interest (ROI), and extracts unique core information for each class to eliminate the feature confusion among foreground instances of different classes. Our method has been validated on the optical remote sensing datasets DIOR and RSOD. It demonstrates superior performance compared to existing methods.

关键词： few-shot object detection remote sensing images pseudo-sample generation feature enhancement

来源：评论

学校读者我要写书评

暂无评论

Large Pretrained Foundation Model for key Performance Indicator Multivariate Time Series Anomaly Detection

IEEE OPEN JOURNAL OF THE COMPUTER SOCIETY

引用

IEEE OPEN JOURNAL OF THE compUTER SOCIETY 2025年第1期6卷 176-187页

作者： Wang, Xu Xu, Qisheng Xu, Kele Yu, Ting Ding, Bo Feng, Dawei Dou, Yong Natl Univ Def Technol Coll Comp Sci & Technol Changsha 410073 Peoples R China Natl Univ Def Technol Natl Key Lab Parallel & Distributed Proc Changsha 410073 Peoples R China

In the realm of key Performance Indicator (KPI) anomaly detection, deep learning has emerged as a pivotal technology. Yet, the development of effective deep learning models is hindered by several challenges: scarce and complex labeled data, noise interference from data handling, the necessity to capture temporal dependencies in time series KPI data, and the complexity of multivariate data analysis. Despite recent progress in large models that show potential for handling complex, multidimensional tasks, the lack of extensive, high-quality datasets presents a significant barrier for directly training these models in KPI anomaly detection. This scarcity limits the models' ability to learn and generalize effectively within this specific domain. To overcome this, we propose an innovative approach to adapt fully pretrained large models from other domains to KPI anomaly detection, thereby mitigating data constraints and enhancing detection precision. Our approach involves adapting large models to anomaly detection tasks using patch operations and fine-tuning techniques, which significantly enhances the model's temporal dependency capture capabilities. Furthermore, to address the multivariate challenge, we introduce a novel feature extraction method based on channel independence to optimize information processing across multidimensional features. Additionally, we leverage frequency domain information to design a feature enhancement method, further boosting the model's detection accuracy. By integrating these innovative techniques, we have developed a large-scale KPI anomaly detection model named ViTSD. Empirical evidence from experiments on five benchmark datasets and two additional datasets demonstrates ViTSD's superior performance, outperforming existing models across various evaluation metrics.

关键词： Feature extraction Anomaly detection Time series analysis key performance indicator Adaptation models Data models Analytical models Time-frequency analysis Transformers computational modeling large foundation model patch operation key Performance Indicator (KPI) anomaly detection

来源：评论

学校读者我要写书评

暂无评论

A physics-informed generative adversarial network for advancing solutions in ocean acoustics

引用

PHYSICS OF FLUIDS 2025年第3期37卷

作者： Xia, Rui Guo, Xiao-Wei Zhang, Huajian Li, Genglin Xiao, Jing Xiao, Qisong Song, Min Li, Chao Liu, Jie Natl Univ Def Technol Lab Digitizing Software Frontier Equipment Changsha 410073 Peoples R China Natl Univ Def Technol Coll Comp Sci & Technol Changsha 410073 Peoples R China Natl Univ Def Technol Natl Key Lab Parallel & Distributed Comp Changsha 410073 Peoples R China

Advancements in artificial intelligence, notably the groundbreaking efforts in deep learning exemplified by physics-informed neural networks, have opened up innovative pathways for addressing intricate ocean acoustic problems. However, conventional physics-informed neural networks are limited in solving high-frequency forward and inverse problems. This paper introduces a novel physics-informed generative adversarial network integrating a forward-solving network (generator) and an inverse parameter-estimating network (discriminator). The generator network incorporates convolutional neural networks with hard-constrained boundary conditions and optimized loss functions to effectively predict the solution governed by the time-domain wave equation. For inverse problems, a discriminator is introduced for parameter estimation to complete the generative adversarial network. Furthermore, customized optimization strategies and an adaptive weighting loss function are devised to boost the training performance further. The test results of both forward and reverse cases show the advantage of our model over existing methods in terms of accuracy. The result indicates its vast potential for applications in ocean acoustics engineering.

关键词： Discriminators

来源：评论

学校读者我要写书评

暂无评论

A heterogeneous parallel algorithm for the Cartesian discrete ordinates for multizone heterogeneous system

引用

JOURNAL OF SUPERcompUTING 2025年第4期81卷 1-33页

作者： Li, Runhua Wang, Qinglin Liu, Jie Natl Univ Def Technol Coll Comp Sci & Technol Changsha 410073 Hunan Peoples R China Natl Univ Def Technol Lab Digitizing Software Frontier Equipment Changsha 410073 Hunan Peoples R China Natl Univ Def Technol Natl Key Lab Parallel & Distributed Comp Changsha 410073 Hunan Peoples R China

Advancements in computing technology have revolutionized the efficiency and cost-effectiveness of realistic reactor core simulations. The discrete ordinates (SN\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S_N$$\end{document}) method is a widely adopted approach for numerically solving the Boltzmann transport equation (BTE), which describes neutron distribution in nuclear reactors. Recently, the MT-3000, a novel multizone heterogeneous architecture designed for high-performance computing, has been developed, offering a peak double-precision performance of 11.6 TFLOPS at 1.2 GHz. In this study, we propose an efficient four-level heterogeneous SN\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S_N$$\end{document} parallel algorithm for structured hexahedral grids on the MT-3000 system. The algorithm incorporates a two-level KBA strategy with an optimized communication scheme among the MT-3000's acceleration cores and employs a software caching technique to reduce memory access latency. Numerical experiments reveal that our algorithm achieves 1.68 TFLOPS on a single MT-3000 chip, representing 14.5% of its peak performance. The heterogeneous parallel algorithm demonstrates strong scaling efficiency, consistently exceeding 50% across 4 to 256 MT-3000 systems. Furthermore, we develop a performance model that closely aligns with the experimental results.

关键词： Discrete ordinates KBA Neutron transport High-performance computing MT-3000

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：