Transformer-based models suffer from large num-ber of parameters and high inference latency, whose deployment are not green due to the potential environmental damage caused by high inference energy consumption. In add...
Transformer-based models suffer from large num-ber of parameters and high inference latency, whose deployment are not green due to the potential environmental damage caused by high inference energy consumption. In addition, it is difficult to deploy such models on devices, especially on resource constrained devices such as FPGA. Various model pruning methods are proposed to shrink the model size and resource consumption, so as to fit the models on hardware. However, such methods often introduce floating point of operations (FLOPs) as an agent of hardware performance, which is not accurate. Furthermore, structural pruning methods are always in a single head-wise or layer-wise pattern, which fails to compress the models to the extreme. To resolve the above issues, we propose a green BERT deployment method on FPGA via hardware-aware and hybrid pruning, named g-BERT. Specifically, two hardware-aware metrics are introduced by High Level Synthesis (HLS) to evaluate the latency and power consumption of inference on FPGA, which can be optimized directly while pruning. Moreover, we simultaneously consider pruning of heads and full encoder layers. To efficiently find the optimal structure, g-BERT applies differentiable neural architecture search (NAS) with a special 0–1 loss function. Compared with the BERT-base, g-BERT achieves $2.1\times$ speedup, $1.9\times$ power consumption reduction and $1.8\times$ model size reduction with comparable accuracy, on par with the state-of-the-art methods.
Temporal Coarse-Grained Reconfigurable Architecture (CGRA) is a typical category of CGRA that supports single-cycle context switching and time-multiplexing hardware resources to perform both spatial and temporal compu...
Temporal Coarse-Grained Reconfigurable Architecture (CGRA) is a typical category of CGRA that supports single-cycle context switching and time-multiplexing hardware resources to perform both spatial and temporal computations. Compared with the spatial CGRA, it can be used in area and power budget-constrained scenarios, with the sacrifice of the throughput. Therefore, achieving minimum Initialization Interval (II) for higher throughput is the main objective in many works for temporal CGRA mapping.
Existing accelerators for transformer networks with field-programmable gate array (FPGA) either focus only on attention computation or suffer from fixed data streams without flexibility. Moreover, compression and appr...
Existing accelerators for transformer networks with field-programmable gate array (FPGA) either focus only on attention computation or suffer from fixed data streams without flexibility. Moreover, compression and approximation methods of transformer networks have the potential for further optimization. In this article, we propose a low-latency FPGA-based overlay processor, named LTrans-OPU for general accelerations of transformer networks. Specifically, we design a domain-specific overlay architecture, including a computation unit for matrix multiplication of arbitrary dimensions. An instruction set customized for our overlay architecture is also introduced, dynamically controlling data flows by generated instructions. In addition, we introduce a hybrid pruning method common to various transformer networks, along with an efficient non-linear function approximation method. Experimental results show that our design is rather competitive and has low latency. LTrans-OPU achieves 11.10-32.20× speedup compared with CPU and 2.44-6.18 × latency reduction compared with GPU. We also observe 2.36-12.43 × lower latency compared with customized FPGA/asic accelerators, and can be 3.10× faster than NPE.
Conjugate gradient (CG) is widely used in training sparse neural networks. However, CG, involving a large amount of sparse matrix and vector operations, cannot be efficiently implemented on resource-limited edge devic...
Conjugate gradient (CG) is widely used in training sparse neural networks. However, CG, involving a large amount of sparse matrix and vector operations, cannot be efficiently implemented on resource-limited edge devices. In this paper, a high-performance and energy-efficient CG accelerator implemented on edge Field Programmable Gate Array is proposed for fast onsite neural networks training. According to the profiling, we propose a unified matrix multiplier that is compatible with the sparse and dense matrix. We also design a novel T-engine to handle transpose operation with the compressed sparse format. Experimental results show that our proposal outperforms the state-of-the-art FPGA work with a resource reduction of up to 41.3%. In addition, we achieve on average $10.2\times$ and $2.0\times$ speedup, while $10.1\times$ and $3.5\times$ better energy efficiency than implementations on CPU and GPU, respectively.
Transformer models have been widely adopted in the field of Natural Language Processing (NLP) and Computer Vision (CV). However, the excellent performance of Transformers comes at the cost of heavy memory footprints a...
Transformer models have been widely adopted in the field of Natural Language Processing (NLP) and Computer Vision (CV). However, the excellent performance of Transformers comes at the cost of heavy memory footprints and gigantic computing complexity. To deploy Transformers on resource constrained platforms, e.g., FPGA, diverse weight pruning strategies have been proposed. However, pattern pruning, as an alternative pruning method, is not well explored in the context of Transformers. In this paper, we propose PP-Transformer, a framework specifically designed to efficiently deploy Transformer models on FPGA using pattern pruning. At the algorithm level, we leverage pattern pruning, a coarse-grained structured pruning strategy, to reduce parameter storage. Meanwhile, we have developed a dedicated hardware architecture, featuring a custom computing engine tailored to support pattern pruning algorithm. Experimental results demonstrate that our algorithm achieves up to $2.26\times$ reduction in parameter storage with acceptable accuracy degradation. Additionally, our hardware implementation exhibits $839.72\times$ and $5.72\times$ speedup in comparison to CPU and GPU implementations.
Coarse-grained reconfigurable architecture (CGRA), composed of word-level processing elements (PEs) and interconnects, has emerged as a promising architecture due to its high performance, energy efficiency, and flexib...
Coarse-grained reconfigurable architecture (CGRA), composed of word-level processing elements (PEs) and interconnects, has emerged as a promising architecture due to its high performance, energy efficiency, and flexibility. Although multiple CGRA frameworks have been proposed, a complete heterogeneous CGRA exploration framework with tunable interconnect flexibility and fast design space exploration (DSE) is still lacking. In this paper, we propose an open-source template-based CGRA exploration framework that integrates the modeling of heterogeneous PEs and interconnects, RTL generation, DFG mapping, automatic simulation and verification, and fast DSE based on a CGRA framework TRAM. Moreover, we present a novel resource-efficient shared reconfigurable delay unit (RDU) for data synchronization, which can save the CGRA area by 7%, compared with the separated RDU. Further, the explored optimal heterogeneous architecture can reduce the area and power by 44.7% and 42.9% respectively, and improve the PE utilization by 20.4%, compared with the 8 × 8 baseline architecture in TRAM.
When an application is accelerated with Coarse-Grained Reconfigurable Architecture (CGRA), it is compiled into Data Flow Graph (DFG). In conventional CGRA frameworks, only one DFG is accelerated in each epoch. Consequ...
When an application is accelerated with Coarse-Grained Reconfigurable Architecture (CGRA), it is compiled into Data Flow Graph (DFG). In conventional CGRA frameworks, only one DFG is accelerated in each epoch. Consequently, single-context CGRAs can’t fully utilize hardware resources when executing multi-kernel applications. In this paper, we propose a dynamic partial reconfigurable CGRA framework for multi-kernel applications. The modeled CGRA can flexibly partition hardware resources and support parallelism of multiple DFGs by implementing dynamic partial reconfiguration (DPR). A multi-kernel scheduler based on integer linear programming (ILP) makes a timetable for the execution state of the application, and an incremental mapper compiles DFGs according to the timetable. Compared with the baseline, TRAM, our framework achieves an average throughput increase of 67.30% and utilization increase of 32.46% for a single task with multi-kernels while an average execution time reduction of 55.71% and an average utilization increase of 70.43% for applications with multiple tasks.
Machine learning has been used extensively in the bioactivity value (BAV) prediction of G Protein-Coupled Receptors (GPCR) targeting ligands. However, the performance of over 140 types of GPCR endogenous ligands, also...
Machine learning has been used extensively in the bioactivity value (BAV) prediction of G Protein-Coupled Receptors (GPCR) targeting ligands. However, the performance of over 140 types of GPCR endogenous ligands, also called orphan GPCRs (oGPCRs), is still unsatisfactory due to the limited sample size. Also, current works are far from meeting the demand for fast inference time and energy efficiency. We propose the Multi-Source Transfer-Graph Attention Network (MSTL-GAT), as well as its FPGA-based accelerator. Firstly, we make use of the three ideal data sources for transfer learning, oGPCRs, experimentally validated GPCRs, and invalidated GPCRs similar to the former one. Secondly, we transform GPCRs from the SIMLEs format to graphics as the input of GAT to improve prediction accuracy. Moreover, we propose an FPGA-based accelerator tailored for the inference phase of MSTL-GAT. Finally, our experimental results show that MSTL-GAT remarkably improves the prediction of GPCRs ligand activity value compared with previous studies. On average, the two evaluation indexes we adopt, R2 and RMSE, improve by 34.76% and 13.16%, respectively. The proposed FPGA accelerator achieves 2.7× and 4.7× speedup, 29.7×, and 3.6× energy efficiency compared with works on GPU implementation and the state-of-the-art FPGA accelerator, respectively.
In today’s tech-driven society, the emphasis on data privacy and security has skyrocketed. With technological progress, the emergence of new encryption algorithms and advanced attack technologies compel the need for ...
In today’s tech-driven society, the emphasis on data privacy and security has skyrocketed. With technological progress, the emergence of new encryption algorithms and advanced attack technologies compel the need for algorithm upgrades. With rising hardware costs and demand for flexible cryptographic platforms, single-algorithm accelerators are insufficient, making versatile accelerators supporting multiple encryption algorithms essential. Besides flexibility, energy and area efficiency are increasingly important for various encrypted application platforms like embedded devices. Currently, few cryptographic processing accelerators like Anole [1] prioritize energy efficiency and flexibility. However, their primary focus is on symmetric key algorithms and Hash algorithms, without including Fully Homomorphic Encryption over the Torus (TFHE) [2]. Solutions like MATCHA [3] focus on TFHE but compromise compatibility with other algorithms. Additionally, a user-friendly, end-to-end toolchain is lacking in existing solutions. To address these challenges, we propose $\mathrm{E}^{2}$-ACE, based on the TRAM [4], supporting symmetric key algorithms, Hash algorithms, and TFHE.
The approximation of non-linear operation can simplify the logic design and save the system resources during the neural network inference on Field-Programmable Gate Array (FPGA). Prior work can approximate the non-lin...
The approximation of non-linear operation can simplify the logic design and save the system resources during the neural network inference on Field-Programmable Gate Array (FPGA). Prior work can approximate the non-linear operations with piecewise linear (PWL) function, but such approximation neglects considering the hardware overhead simultaneously. This paper proposes a novel approximation framework called Auto-LUT, which leverages a neural network to automatically approximate the non-linear operations. The framework formulates the approximation error and hardware overhead as a multi-objective optimization problem and employs an automated search mechanism to find the minimum number of segments and data bit width. To improve the approximation accuracy, we propose a bias clipping operation during the training of approximation networks, which enforces the model to approximate within the range of interest. Moreover, a hardware-friendly quantization scheme is further introduced to simulate the hardware behavior, thereby reducing the hardware overhead. Finally, a customized hardware architecture based on FPGA is utilized to deploy the quantized result. The experimental results show that Auto-LUT costs 56.32% less LUTs and 32.31% less flip-flops (FF) while reducing 4.32% approximation error compared to the state-of-the-art method.
暂无评论