检索结果-内蒙古大学图书馆

Optimizing Fine-Tuning in Quantized Language Models:An In-Depth Analysis of Key Variables

Computers, Materials & Continua 2025年第1期82卷 307-325页

作者： Ao Shen Zhiquan Lai Dongsheng Li Xiaoyu Hu National Key Laboratory of Parallel and Distributed Computing National University of Defense TechnologyChangsha410073China Strategic Assessments and Consultation Institute Academy of Military ScienceBeijing100091China

Large-scale Language Models(LLMs)have achieved significant breakthroughs in Natural Language Processing(NLP),driven by the pre-training and fine-tuning *** this approach allows models to specialize in specific tasks with reduced training costs,the substantial memory requirements during fine-tuning present a barrier to broader ***-Efficient Fine-Tuning(PEFT)techniques,such as Low-Rank Adaptation(LoRA),and parameter quantization methods have emerged as solutions to address these challenges by optimizing memory usage and computational *** these,QLoRA,which combines PEFT and quantization,has demonstrated notable success in reducing memory footprints during fine-tuning,prompting the development of various QLoRA *** these advancements,the quantitative impact of key variables on the fine-tuning performance of quantized LLMs remains *** study presents a comprehensive analysis of these key variables,focusing on their influence across different layer types and depths within LLM *** investigation uncovers several critical findings:(1)Larger layers,such as MLP layers,can maintain performance despite reductions in adapter rank,while smaller layers,like self-attention layers,aremore sensitive to such changes;(2)The effectiveness of balancing factors depends more on specific values rather than layer type or depth;(3)In quantization-aware fine-tuning,larger layers can effectively utilize smaller adapters,whereas smaller layers struggle to do *** insights suggest that layer type is a more significant determinant of fine-tuning success than layer depth when optimizing quantized ***,for the same discount of trainable parameters,reducing the trainable parameters in a larger layer is more effective in preserving fine-tuning accuracy than in a smaller *** study provides valuable guidance for more efficient fine-tuning strategies and opens avenues for further research into optimizing LLM

关键词： Large-scale Language Model Parameter-Efficient Fine-Tuning parameter quantization key variable trainable parameters experimental analysis

来源：评论

学校读者我要写书评

暂无评论

U-shaped Dual Attention Transformer: An Efficient Transformer Based on Channel and Spatial Attention 4

U-shaped Dual Attention Transformer: An Efficient Transforme...

引用

4th International Conference on Artificial Intelligence, Robotics, and Communication, ICAIRC 2024

作者： Zhai, Zhaoyuan Qiao, Peng Li, Rongchun Zhou, Zhen National University of Defense Technology National Key Laboratory of Parallel and Distributed Computing Changsha China

ISBN: (纸本)9798331531225

Transformer-based methods have demonstrated remarkable performance on image super-resolution tasks. Due to high computational complexity, researchers have been working to achieve a balance between computation costs and performance. Restormer has achieved commendable balance by utilizing global channel attention. However, the performance is limited by insufficient local pixel reconstruction. In this paper, we propose a U-shaped Dual Attention Transformer (UDAT) with local-global receptive field, addressing the limitation of Restormer in local pixel reconstruction. We propose a dense window channel attention to enhance the local feature representation, more efficient in computational complexity. Experiments demonstrate that our UDAT achieve superior performance compared with Restormer on benchmark datasets, surpassing Restormer by 0.57 dB on the Urban100 dataset. On-par with SwinIR, our method reduces computational complexity by 3.2 times and improves inference speed by 2 times, achieving a better balance between computational costs and performance. © 2024 IEEE.

关键词： Pixels

来源：评论

学校读者我要写书评

暂无评论

A Quantitative Analysis of OpenMP Task Runtime Systems 14th

A Quantitative Analysis of OpenMP Task Runtime Systems

引用

14th International Symposium on Benchmarking, Measuring, and Optimization, Bench 2022

作者： Hunold, Sascha Kraßnitzer, Klaus Research Group for Parallel Computing Faculty of Informatics TU Wien Vienna Austria

ISBN: (纸本)9783031311796

Although OpenMP is heavily used to parallelize for-loops, it also supports task-parallel programming, which is important for parallelizing irregular applications. In this work, we focus on the performance of OpenMP runtime systems for task-based applications. In particular, we investigate the performance of different OpenMP runtime systems when scheduling a large set independent tasks of different granularity. To that end, we propose a new OpenMP benchmark, which features profiling and tracing options that help developers to reason about the observed performance differences. We compare the execution times measured for a variety of compilers, such as gcc, icc, clang, aocc, and pgcc, for both homogeneous and heterogeneous workloads. Our study shows that there are significant performance differences between the different OpenMP implementations. We also show that the performance attainable with a compiler strongly depends on the machine architecture, the number of threads, the thread-pinning strategy, and the task granularity. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.

关键词： Benchmarking

来源：评论

学校读者我要写书评

暂无评论

A Multi-Objective Artificial Physics Optimization Algorithm Based on Two-Phase Search 8

A Multi-Objective Artificial Physics Optimization Algorithm ...

引用

8th International Symposium on Computer Science and Intelligent Control, ISCSIC 2024

作者： Zhang, Huihua Xie, Liping Taiyuan University of Science and Technology Shanxi Key Laboratory of Big Data Analysis and Parallel Computing Taiyuan China

ISBN: (纸本)9798350380286

Aiming at the problems of insufficient utilization of information about elite particles in archive and instability of particle motion in the population in the multi-objective artificial physics optimization algorithm (MOAPO) in solving multiobjective optimization problems, A multi-objective artificial physics optimization algorithm based on two-phase search (TPMOAPO) is proposed. To begin with, the algorithm improves the calculation of the mass of particles, so that the strength and weakness of the particles can be accurately transformed into the corresponding masses while improving the efficiency of particle mass calculation. Next, a two-phase search strategy is proposed, which makes the algorithm have strong exploration ability in the first phase, and the second phase gradually enhances the exploitation capability with iterations, which solves the problem of instability motion of particles in the search process. Finally, the simulated binary crossover (SBX) and polynomial-based mutation (PM) operators are adopted in the archive to further enhance the search capability of the algorithm. For verifying the performance of TP-MOAPO, 21 benchmark functions were selected to compare with the classical multi-objective particle swarm optimization algorithms: MOPSO, dMOPSO, SMPSO, MMOPSO, and NMPSO, and the experimental results show the superiority of TP-MOAPO in these functions. © 2024 IEEE.

关键词： Optimization algorithms

来源：评论

学校读者我要写书评

暂无评论

Funnel: An Efficient Sparse Attention Accelerator with Multi-Dataflow Fusion 22

Funnel: An Efficient Sparse Attention Accelerator with Multi...

引用

22nd IEEE International Symposium on parallel and Distributed Processing with Applications, ISPA 2024

作者： Ma, Shenghong Xu, Jinwei Jiang, Jingfei Wang, Yaohua Li, Dongsheng National University of Defense Technology National Key Laboratory of Parallel and Distributed Computing College of Computer Changsha China

ISBN: (纸本)9798331509712

The self-attention mechanism is the core component of Transformer, which provides a powerful ability to understand the sequence context. However, the self-attention mechanism also suffers from a large amount of redundant computation. Model sparsification can effectively reduce computational load, but the irregularity of non-zeros introduced by sparsification significantly decreases hardware efficiency. This paper proposes Funnel, an accelerator that dynamically predicts sparse attention patterns and efficiently processes unstructured sparse data. Firstly, we adopt a fast quantization method based on lookup table to minimize the cost of sparse patterns prediction. Secondly, we propose Funnel computing Unit (FCU), a hardware architecture that efficiently handles sparse attention through multi-dataflow fusion. Sampled Dense-Dense Matrix Multiplication (SDDMM) and Sparse-Dense Matrix Multiplication (SpMM) are core components of sparse attention mechanism. FCU unifies the computation ways of matrix inner product and row-wise product to support SDDMM and SpMM at the same time, which greatly reduces the storage and movement overhead of intermediate results. Lastly, we devise a lightweight buffer and data tiling strategy tailored to the proposed accelerator, aimed at enhancing data reuse. Experiments demonstrate that our accelerator achieves 0.10-0.25 sparsity with small accuracy loss. When computing the self-attention layer, it attains hardware efficiency ranging from 60% to 85%. Compared to CPU and GPU, it achieves 5.60x and 8.20x speedup. Compared to the state-of-the-art attention accelerators A3, SpAtten, FTRANS, and Sanger, it achieves 7.37x, 4.52x, 9.58x, and 3.08x speedup. © 2024 IEEE.

关键词： FPGA Funnel computing unit Sparse attention Transformer

来源：评论

学校读者我要写书评

暂无评论

Large-Scale Multi-Objective Optimization Algorithm Based on Weighted Overlapping Grouping of Decision Variables

引用

Computer Modeling in Engineering & Sciences 2024年第7期140卷 363-383页

作者： Liang Chen Jingbo Zhang Linjie Wu Xingjuan Cai Yubin Xu Shanxi Key Laboratory of Big Data Analysis and Parallel Computing Taiyuan University of Science and TechnologyTaiyuan030024China School of State Key Laboratory of Novel Software Technology Nanjing UniversityNanjing210008China

The large-scale multi-objective optimization algorithm(LSMOA),based on the grouping of decision variables,is an advanced method for handling high-dimensional decision ***,in practical problems,the interaction among decision variables is intricate,leading to large group sizes and suboptimal optimization effects;hence a large-scale multi-objective optimization algorithm based on weighted overlapping grouping of decision variables(MOEAWOD)is proposed in this ***,the decision variables are perturbed and categorized into convergence and diversity variables;subsequently,the convergence variables are subdivided into groups based on the interactions among different decision *** the size of a group surpasses the set threshold,that group undergoes a process of weighting and overlapping ***,the interaction strength is evaluated based on the interaction frequency and number of objectives among various decision *** decision variable with the highest interaction in the group is identified and disregarded,and the remaining variables are then reclassified into ***,the decision variable with the strongest interaction is added to each *** minimizes the interactivity between different groups and maximizes the interactivity of decision variables within groups,which contributed to the optimized direction of convergence and diversity exploration with different *** was subjected to testing on 18 benchmark large-scale optimization problems,and the experimental results demonstrate the effectiveness of our *** with the other algorithms,our method is still at an advantage.

关键词： Decision variable grouping large-scale multi-objective optimization algorithms weighted overlapping grouping direction-guided evolution

来源：评论

学校读者我要写书评

暂无评论

Mbapp: Efficient Memory-Balanced Pipeline parallelism for Large Model Fine-Tuning on Commodity GPU Servers 24

Mbapp: Efficient Memory-Balanced Pipeline Parallelism for La...

引用

5th International Conference on Computer Information and Big Data Applications, CIBDA 2024

作者： Liu, Yujie Lai, Zhiquan Li, Dongsheng National Key Laboratory of Parallel and Distributed Computing College of Computer National University of Defense Technology Changsha410000 China

ISBN: (纸本)9798400718106

Large-scale models have demonstrated outstanding performance across various downstream tasks. Pipeline parallelism is essential for fine-tuning large models on commodity GPU servers, as it plays a crucial role in making their exceptional performance more accessible and widespread. Existing approaches have encountered challenges in attaining effective memory-balanced pipeline parallelism. This paper presents a novel memory load-balanced pipeline parallel solution that aims to distribute memory usage evenly across stages on commodity GPU servers by leveraging NVLink bridges. The solution presents a novel approach for transferring data from GPUs to CPUs through the PCI-e link between adjacent GPUs interconnected by the NVLink bridge. Moreover, our methodology orchestrates data transfer operations to reduce offloading latency during the fine-tuning of large models. The evaluation demonstrates that our approach enhances switching efficiency by 1.46x to 1.76x and improves throughput by 2.04x to 3.59x compared to the PyTorch offloading technique. © 2024 ACM.

关键词： Deep neural networks

来源：评论

学校读者我要写书评

暂无评论

Electromagnetic Physics Models for parallel computing Architectures

引用

Journal of Physics: Conference Series 2016年第1期762卷

作者： G Amadio A Ananya J Apostolakis A Aurora M Bandieramonte A Bhattacharyya C Bianchini R Brun P Canal F Carminati L Duhem D Elvira A Gheata M Gheata I Goulas R Iope S Y Jun G Lima A Mohanty T Nikitina M Novak W Pokorski A Ribon R Seghal O Shadura S Vallecorsa S Wenzel Y Zhang Parallel Computing Center at São Paulo State University (UNESP) São Paulo Brazil CERN EP Department Geneva Switzerland Bhabha Atomic Research Centre (BARC) Mumbai India Mackenzie Presbyterian University São Paulo Brazil Fermilab MS234 P.O. Box 500 Batavia IL 60510 USA Intel Corporation Santa Clara CA 95052 USA Institute of Space Sciences Bucharest-Magurele Romania

The recent emergence of hardware architectures characterized by many-core or accelerated processors has opened new opportunities for concurrent programming models taking advantage of both SIMD and SIMT architectures. GeantV, a next generation detector simulation, has been designed to exploit both the vector capability of mainstream CPUs and multi-threading capabilities of coprocessors including NVidia GPUs and Intel Xeon Phi. The characteristics of these architectures are very different in terms of the vectorization depth and type of parallelization needed to achieve optimal performance. In this paper we describe implementation of electromagnetic physics models developed for parallel computing architectures as a part of the GeantV project. Results of preliminary performance evaluation and physics validation are presented as well.

关键词：

来源：评论

学校读者我要写书评

暂无评论

DP2: A Highly parallel Range Join for Genome Analysis on Distributed computing Platform

DP2: A Highly Parallel Range Join for Genome Analysis on Dis...

引用

2019 International Conference on High Performance computing and Simulation, HPCS 2019

作者： Sinha, Aman Lai, Bo-Cheng National Chiao Tung University Parallel Computing System Lab Hsinchu City Taiwan

ISBN: (纸本)9781728144849

Rapid growth of the sheer amount of genome data and intense computation become great challenges for downstream genome analytics. Efficient parallel processing and distributed computing are the two effective schemes to address the analysis of big data. Range join is a widely used, effective, yet time-consuming operation that finds the overlap between two different sets of genome features. The current widely adopted BEDTools [6] pipeline adopts single-node binary tree approach, while the distributed GenAp scheme fails to exploit the massive parallel computation on modern throughput processors, such as GPU (Graphic Processing Unit). This paper proposes a novel Distributed parallel P-ary search (DP2) that applies novel P-ary analysis to enable high parallelism at algorithmic level, and extensively utilize multiple GPUs at system and architecture level. Efficient computation allocation is implemented to leverage the distributed computing on clusters. The proposed framework can be well integrated with current BEDTools [6] pipeline, and achieves an average of 25x speedup for the actual range-join operation when compared with Binary tree approach of GenAp and a 13x end-to-end (total execution time) speedup in comparison to ADAM. © 2019 IEEE.

关键词： Pipelines

来源：评论

学校读者我要写书评

暂无评论

HiHiSpMV: Sparse Matrix Vector Multiplication with Hierarchical Row Reductions on FPGAs with High Bandwidth Memory 32

HiHiSpMV: Sparse Matrix Vector Multiplication with Hierarchi...

引用

32nd IEEE Annual International Symposium on Field-Programmable Custom computing Machines, FCCM 2024

作者： Tareen, Abdul Rehman Meyer, Marius Plessl, Christian Kenter, Tobias Paderborn University Paderborn Center for Parallel Computing Department of Computer Science Warburger Str. 100 Paderborn33098 Germany

ISBN: (纸本)9798350372434

The multiplication of a sparse matrix with a dense vector is a vital operation in linear algebra, with applications in numerous contexts. After earlier research on FPGA acceleration of this operation had shown the potential to achieve a high bandwidth efficiency, this workload has received renewed attention with the introduction of high bandwidth memory on FPGA platforms. However, previous designs fell short of scaling to the full bandwidth potential of current FPGA platforms with high bandwidth memory. In this work, we present HiHiSpMV with a novel design approach around hierarchical accumulation, which allows us to overcome several limitations of the related work. Our design is completely implemented with high-level synthesis and compiled with Vitis, and instantiates 16 independent compute units, each processing up to 16 matrix elements per clock cycle. The reduction of these elements is performed without any latency or resource overhead and the subsequent accumulation uses the well-established shift-register design pattern. Due to the independent nature of compute units, our design can connect to all of the 32 high bandwidth memory pseudo-channels in 512-bit interface mode on an Alveo U280 FPGA board. In our tests, we reach up to 86% of the theoretical available bandwidth of this FPGA platform, enabling a computational throughput of up to 98 GFLOPS. This is about 1.5 × faster than the peak throughput of the best related work in that regard, Serpens. On average, the throughput of HiHiSpMV is even 2.7× higher than Serpens. © 2024 IEEE.

关键词： Shift registers

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：