检索结果-内蒙古大学图书馆

Optimizing Fine-Tuning in Quantized Language Models:An In-Depth Analysis of Key Variables

Computers, Materials & Continua 2025年第1期82卷 307-325页

作者： Ao Shen Zhiquan Lai Dongsheng Li Xiaoyu Hu National Key Laboratory of Parallel and Distributed Computing National University of Defense TechnologyChangsha410073China Strategic Assessments and Consultation Institute Academy of Military ScienceBeijing100091China

Large-scale Language Models(LLMs)have achieved significant breakthroughs in Natural Language processing(NLP),driven by the pre-training and fine-tuning *** this approach allows models to specialize in specific tasks with reduced training costs,the substantial memory requirements during fine-tuning present a barrier to broader ***-Efficient Fine-Tuning(PEFT)techniques,such as Low-Rank Adaptation(LoRA),and parameter quantization methods have emerged as solutions to address these challenges by optimizing memory usage and computational *** these,QLoRA,which combines PEFT and quantization,has demonstrated notable success in reducing memory footprints during fine-tuning,prompting the development of various QLoRA *** these advancements,the quantitative impact of key variables on the fine-tuning performance of quantized LLMs remains *** study presents a comprehensive analysis of these key variables,focusing on their influence across different layer types and depths within LLM *** investigation uncovers several critical findings:(1)Larger layers,such as MLP layers,can maintain performance despite reductions in adapter rank,while smaller layers,like self-attention layers,aremore sensitive to such changes;(2)The effectiveness of balancing factors depends more on specific values rather than layer type or depth;(3)In quantization-aware fine-tuning,larger layers can effectively utilize smaller adapters,whereas smaller layers struggle to do *** insights suggest that layer type is a more significant determinant of fine-tuning success than layer depth when optimizing quantized ***,for the same discount of trainable parameters,reducing the trainable parameters in a larger layer is more effective in preserving fine-tuning accuracy than in a smaller *** study provides valuable guidance for more efficient fine-tuning strategies and opens avenues for further research into optimizing LLM

关键词： Large-scale Language Model Parameter-Efficient Fine-Tuning parameter quantization key variable trainable parameters experimental analysis

来源：评论

学校读者我要写书评

暂无评论

DaCP: Accelerating Synchronization-Free SpTRSV via GPU-Friendly Data Communication and parallelism Strategies 20th

DaCP: Accelerating Synchronization-Free SpTRSV via GPU-Frie...

引用

20th IFIP WG 10.3 International Conference on Network and parallel Computing, NPC 2024

作者： Guo, Mingfeng Deng, Liang Dai, Zhe Li, Ruitian Lin, Gaofeng Liu, Jie Computational Aerodynamics Institute China Aerodynamics Research and Development Center Mianyang China Science and Technology on Parallel and Distributed Processing Laboratory National University of Defense Technology Changsha China

ISBN: (纸本)9789819628292

Sparse triangular solve (SpTRSV) is a vital component in various scientific applications, and numerous GPU-based SpTRSV algorithms have been proposed. Synchronization-free SpTRSV is currently the mainstream algorithm on GPU due to its short preprocessing time and outstanding performance. However, we observed that this algorithm still has two performance bottlenecks. Firstly, the thread-level parallel mode can introduce to thread divergence issues within GPU warps during the writing phase. Secondly, the thread-level and warp-level fusion mode may struggles to fully exploit GPU resources due to suboptimal mapping relationships between rows and threads. To address these issues, this paper proposes DaCPSpTRSV, a new synchronization-free algorithm with GPU-friendly data communication and parallelism strategies. Specifically, we first develop a fast-forward thread-level approach, incorporating an efficient global memory access pattern and a light-weight dependency control mechanism, to optimize data communication and alleviate thread divergence. A fine-grained fusion strategy is then proposed to maximize GPU parallelism by adaptively selecting the suitable thread-level or warp-level modes. Moreover, the commonly-used compressed sparse row (CSR) format is employed in our DaCPSpTRSV, enhancing the versatility of our algorithm. We evaluate our approach using 245 matrices from the SuiteSparse Matrix Collection on two NVIDIA GPUs, demonstrating speedup ratios of up to 4.77×, 4.94×, 1.67×, and 1.62× compared to cuSPARSE, Sync-Free, CapelliniSpTRSV, and YuenyeungSpTRSV, respectively. The project is open-sourced at https://***/gmfff12334/DaCP. © IFIP International Federation for Information processing 2025.

关键词： Graphics processing unit

来源：评论

学校读者我要写书评

暂无评论

A Data-Centric Approach for Efficient and Scalable CFD Implementation on Multi-GPUs Clusters 24th

A Data-Centric Approach for Efficient and Scalable CFD Imple...

引用

24th International Conference on parallel and distributed Computing, Applications and Technologies, PDCAT 2023

作者： Li, Ruitian Deng, Liang Dai, Zhe Zhang, Jian Liu, Jie Liu, Gang China Aerodynamic Research and Development Center Computational Aerodynamic Institute Mianyang China Science and Technology on Parallel and Distributed Processing Laboratory National University of Defense Technology Changsha China

ISBN: (纸本)9789819982103

Scalability is a crucial factor determining the performance of massive heterogeneous parallel CFD applications on the multi-GPUs platforms, particularly after the single-GPU implementations have achieved optimal performance through numerous optimizations. A novel Data-Centric hybrid MPI-CUDA CFD model is proposed in this paper to enable efficient scalability of CFD applications on large-scale heterogeneous platforms. Based on the Data-Centric approach, Minimum-cost MPI transfer strategy and the code refactoring technique are realized for a better balance between data transfer and floating-point computation performance, which could significantly improve the scalability and reduce the time-to-solution. Subsequently, those approaches are integrated into the industrial unstructured CFD software, FlowStar, to evaluate their effectiveness. Numerical results demonstrate that Minimum-cost MPI strategy achieves more than 2.0 times performance improvement compared to the traditional Model-Centric implementation, and the code refactoring technique boosts performance by 40% to 50% over the minimum-cost MPI version. Moreover, the Data-Centric implementation on 64 A100 GPUs platform show a speedup ratio of over 120 when compared to the original MPI implementation with 64 ranks. © 2024, The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

关键词： Scalability

来源：评论

学校读者我要写书评

暂无评论

Increasing Momentum-Like Factors:A Method for Reducing Training Errors on Multiple GPUs

引用

Tsinghua science and technology 2022年第1期27卷 114-126页

作者： Yu Tang Zhigang Kan Lujia Yin Zhiquan Lai Zhaoning Zhang Linbo Qiao Dongsheng Li Science and Technology on Paralled and Distributed Processing Laboratory and College of Computer Science and TechnologyNational University of Defense TechnologyChangsha 473000China

In distributed training,increasing batch size can improve parallelism,but it can also bring many difficulties to the training process and cause training *** this work,we investigate the occurrence of training errors in theory and train ResNet-50 on CIFAR-10 by using Stochastic Gradient Descent(SGD) and Adaptive moment estimation(Adam) while keeping the total batch size in the parameter server constant and lowering the batch size on each Graphics processing Unit(GPU).A new method that considers momentum to eliminate training errors in distributed training is *** define a Momentum-like Factor(MF) to represent the influence of former gradients on parameter updates in each ***,we modify the MF values and conduct experiments to explore how different MF values influence the training performance based on SGD,Adam,and Nesterov accelerated *** results reveal that increasing MFs is a reliable method for reducing training errors in distributed *** analysis of convergent conditions in distributed training with consideration of a large batch size and multiple GPUs is presented in this paper.

关键词： multiple Graphics processing Units(GPUs) batch size training error distributed training momentum-like factors

来源：评论

学校读者我要写书评

暂无评论

Factorization Machine-based Unsupervised Model Selection Method

Factorization Machine-based Unsupervised Model Selection Met...

引用

2022 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2022

作者： Zhang, Ruyi Wang, Yijie Xu, Hongzuo Zhou, Haifang National University of Defense Technology Science and Technology on Parallel and Distributed Processing Laboratory China

ISBN: (数字)9781665452588

ISBN: (纸本)9781665452588

Machine learning is broadly used in many intelligent cybernetic systems. With the burgeoning of the communities of AI, the number of machine learning-based models is rapidly increasing, but picking a suitable and optimal (or relatively good) model from overwhelming options has become a conundrum when deploying a new system. Therefore, we are motivated by an intriguing question: Can we automatically select a proper model for new data? However, unsupervised model selection poses two main challenges: (i) Evaluation and comparison of candidate models on the new data are infeasible due to the lack of labels;and (ii) It is non-trivial to build relationships between model performance and data characteristics when the interaction between these characteristics should be considered. In light of these limitations, this paper proposes a factorization machine-based unsupervised model selection method. Following mainstream model selection protocols, we also leverage model performance on prior known datasets. Differently, we learn higher-order complex relationships between model performance and dataset characteristics. Specifically, our method transfers the historical performance into a second-order function of meta-features and embedding weights by harnessing the power of factorization machine. This function can be subsequently used to select a proper model when given a new dataset. Extensive experiments show that our method obtains more superior model selection performance than five state-of-the-art approaches, and our method executes faster than its competitors by approximate three magnitudes. © 2022 IEEE.

关键词： Factorization

来源：评论

学校读者我要写书评

暂无评论

EdgeAnchor: A Rapid and Balanced File Storage Strategy at the Network Edge 29

EdgeAnchor: A Rapid and Balanced File Storage Strategy at th...

引用

29th IEEE International Conference on parallel and distributed Systems, ICPADS 2023

作者： Liu, Han Xie, Xingrui Zhang, Zhuopu Cheng, Geyao Luo, Lailong Guo, Deke National University of Defense Technology Science and Technology on Information Systems Engineering Laboratory China National University of Defense Technology National Laboratory for Parallel and Distributed Processing China

ISBN: (纸本)9798350330717

Storing files at the network edge has become a new paradigm of storage systems, which is promising to mitigate network congestion and reduce file retrieval latency. However, the traditional file storage scheme cannot effectively meet the requirements of rapid indexing and load balance when applied directly to the edge. Moreover, due to the dynamic nature of the edge environment where edge servers can join or leave at will, it is necessary for the storage scheme to adjust with minimal disruption. In this paper, we propose EdgeAnchor, a novel edge storage strategy that is composed of the two-layer hash mappings. The first layer, file-to-bucket mapping, adopts the pseudo-deletion algorithm to deal with the variations in file size, while the second layer utilizes the multiple bucket-to-server mapping to adapt to the heterogeneity in the servers' storage capacities. Furthermore, EdgeAnchor constructs a list of deleted or added working sets for each bucket and creates a dictionary for the mappings between buckets and edge servers. In the manner, EdgeAnchor ensures a rapid file index and balances server load at the dynamic network edge. We also attach the mathematical analyses to EdgeAnchor, which theoretically proves its logarithmic complexity of hash operations and memory accesses. The experiments conducted on real-world datasets demonstrate that EdgeAnchor achieves the file index throughput twice as high as that of Consistent Hashing, under the constraints of load balance. Additionally, it ensures a low and stable data migration volume, when adding or removing edge servers consecutively. © 2023 IEEE.

关键词： dynamic network edge file storage load balance rapid indexing

来源：评论

学校读者我要写书评

暂无评论

CD-Sched: An Automated Scheduling Framework for Accelerating Neural Network Training on Shared Memory CPU-DSP Platforms

CD-Sched: An Automated Scheduling Framework for Accelerating...

引用

2023 International Conference on Power, Communication, Computing and Networking Technologies, PCCNT 2023

作者： Xiao, Yuanyuan Lai, Zhiquan Li, Dongsheng National Key Laboratory of Parallel and Distributed Processing Computer College National University of Defense Technology Changsha China

ISBN: (纸本)9781450399951

DSP holds significant potential for important applications in Deep Neural Networks. However, there is currently a lack of research focused on shared-memory CPU-DSP heterogeneous chips. This paper proposes CD-Sched, an automated scheduling framework that aims to address this gap. By predicting the latency of operators on both CPU and DSP, CD-Sched automatically schedules the computation of operators to the appropriate computing device. This scheduling optimization accelerates the computation of individual operators and ultimately improves the overall training time of neural networks. In end-to-end training tasks, CD-Sched can significantly reduce the overall training time, with an average reduction of approximately 10.77%. © 2023 ACM.

关键词： Deep neural networks

来源：评论

学校读者我要写书评

暂无评论

SMVAR: A Novel RNN Accelerator Based on Non-blocking Data Distribution Structure 25

SMVAR: A Novel RNN Accelerator Based on Non-blocking Data Di...

引用

25th IEEE International Conferences on High Performance Computing and Communications, 9th International Conference on Data science and Systems, 21st IEEE International Conference on Smart City and 9th IEEE International Conference on Dependability in Sensor, Cloud and Big Data Systems and Applications, HPCC/DSS/SmartCity/DependSys 2023

作者： Xu, Jinwei Jiang, Jingfei Xu, Shiyao Gao, Lei National University of Defense Technology Science and Technology on Parallel and Distributed Laboratory Hunan Changsha China

ISBN: (纸本)9798350330014

Recurrent neural networks (RNNs) have become common models in the field of artificial intelligence to process temporal sequence task, such as speech recognition, text analysis, natural language processing, etc. To speedup RNNs inference, previous research proposed model sparse pruning techniques. However, the pruning rate of existing sparse pruning algorithms will be affected by pruning granularity and hardware friendliness. In order to approximate nonstructured pruning algorithm, this paper proposes Large Region Balanced Sparse (LRBS) pruning method, which does not limit sub-matrix shape and effectively improves pruning rate. Furthermore, we propose Sparse Matrix Vector Multiplication Accelerator for RNNs (SMVAR), which adopt non-blocking data distribution structure to solve the problem of efficient execution of large region irreg-ular matrix multiplication. To further improve the accelerator performance, SMVAR fine-grained adjusts the pipeline between macro-operations to reduce the idle of compute components. In addition, according to the coarse-grained block characteristics of LRBS algorithm, we develop the coarse-grained parallelism of accelerator with multiply compute units(CUs) structure. Experiments show that the pruning rate of our proposed LRBS is 1.25x-2.5x higher than that of the existing pruning algorithms. Compared with the existing work, the execution efficiency is improved by more than 2.02x-35.9x in the same application scenario. © 2023 IEEE.

关键词： Field programmable gate arrays (FPGA)

来源：评论

学校读者我要写书评

暂无评论

YFLM: An Improved Levenberg-Marquardt Algorithm for Global Bundle Adjustment 41st

YFLM: An Improved Levenberg-Marquardt Algorithm for Global ...

引用

41st Computer Graphics International Conference, CGI 2024

作者： Peng, Jiaxin Li, Tao Jiang, Qin Liu, Jie Wang, Ruibo Laboratory of Software Engineering for Complex Systems School of Computer Science National University of Defense Technology Hunan Changsha410073 China Parallel and Distributed Processing Laboratory School of Computer Science National University of Defense Technology Hunan Changsha410073 China

ISBN: (纸本)9783031820205

The conventional Levenberg-Marquardt (LM) algorithm is a state-of-the-art trust-region optimization method for solving bundle adjustment problems in the Structure-from-Motion community, which not only takes advantage of the fast convergence of the Gauss-Newton method, but also the stability of the gradient descent method when approaching optimal solutions. However, the damping ratio of LM is simply provided by trial-and-error, which causes slow convergence rate for large-scale problems. This paper proposes the Yamashita-Fukushima LM (YFLM) algorithm to reduce the time complexity for global bundle adjustment, where the damping factor is determined by Yamashita and Fukushima’s method. YFLM dynamically calculates a more reasonable and optimal damping ratio according to the newest reprojection error. The experimental results show that the YFLM algorithm outperforms the conventional LM algorithm for most public bundle adjustment datasets. Besides this, the convergence of the YFLM algorithm is also evaluated with different σ∈(0,2]. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.

关键词： Damping

来源：评论

学校读者我要写书评

暂无评论

Optimizing Yinyang K-Means Algorithm on ARMv8 Many-Core CPUs 22nd

Optimizing Yinyang K-Means Algorithm on ARMv8 Many-Core CPU...

引用

22nd International Conference on Algorithms and Architectures for parallel processing, ICA3PP 2022

作者： Zhou, Tianyang Wang, Qinglin Yin, Shangfei Hao, Ruochen Liu, Jie Science and Technology on Parallel and Distributed Processing Laboratory National University of Defense Technology Changsha410073 China School of Computer Science National University of Defense Technology Changsha410073 China

ISBN: (纸本)9783031226762

K-Means algorithm is one of the most common clustering algorithms widely applied in various data analysis applications. Yinyang K-Means algorithm is a popular enhanced K-Means algorithm that avoids most unnecessary calculations using triangle inequality. However, Yinyang K-Means algorithm is time-consuming when the problem size is large. Due to the influence of performance and energy-efficiency, ARM CPUs have appeared in high performance computing. Therefore, it is very interesting to accelerate Yinyang K-Means algorithm on ARM CPUs. In this paper, we propose an efficient parallel implementation of Yinyang K-Means algorithm on ARMv8 many-core CPUs by means of vectorization, NUMA affinity memory optimization and data layout optimization. The experiment on two ARMv8 many-core CPUs has shown that our implementation can achieve up to 5.6 times faster than the open-source multi-threaded one of Yinyang K-Means algorithm. To the best of our knowledge, this is the first work that studies the optimization of Yinyang K-Means algorithms on ARMv8 CPUs. © 2023, Springer Nature Switzerland AG.

关键词： K-means clustering

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：