Low-bit integer training emerges as a promising approach to mitigate the heavy burden during network training by quantizing the weights, activations, and gradients. However, existing methods cannot well achieve mixed-...
详细信息
Low-bit integer training emerges as a promising approach to mitigate the heavy burden during network training by quantizing the weights, activations, and gradients. However, existing methods cannot well achieve mixed-precision quantization for low-bit training and are commonly limited to INT8 precision. In this paper, we propose a novel low-bit integer training framework that, for the first time, achieves adaptive mixed-precision allocation (AMPA) for weights, activations, and gradients, and pushes the boundaries to a precision level below INT8. We develop a novel magnitude-based sensitivity measurement with regard to the quantization losses of weight, activation, and gradient quantization and the average gradient magnitudes, which is demonstrated as an upper bound of quantization influence in theory. We further design a layer-wise precision update strategy under observations on the quantization losses and their effects on model performance in low-bit training. Extensive experiments on different backbones and datasets show that, compared to INT8 quantization, the proposed method can achieve more than 38% BitOPs reduction with a tolerable loss below 2% in image classification, image segmentation, and language modeling. Copyright 2024 by the author(s)
The transition to a carbon-neutral power system requires replacing conventional generation with distributed renewable energy (RE) sources. This poses challenges related to reduced system inertia and large and long-dis...
详细信息
The so-called Aggregated Energy Systems (AES) represent an opportunity to satisfy the energy demand more efficiently, and economically with respect to traditional energy systems. For instance, this may be achieved by ...
详细信息
This paper proposes an algorithm for deploying dynamic defense strategies on multiple devices in the industrial Internet based on zero trust, aiming at coping with complex and ever-changing security challenges. Malici...
详细信息
The probabilistic safety verification problem of stochastic hybrid systems is very important. In this paper, for a given stochastic hybrid system, an algorithm for generating probabilistic barrier certificates is prop...
详细信息
This paper explores a class of vehicle routing problems that considers the load-dependent distance objective function instead of the total distance. The problem, originating from Thailand's e-commerce delivery bus...
详细信息
This study focuses on the motion planning of a 6-DOF robot arm in complex task scenarios, and deeply explores the key issues such as kinematic modeling, joint Angle path optimization, energy consumption minimization, ...
详细信息
This article examines the management of user traffic to the network access point and within the network, from the user's access point to the destination server containing the required information. This study is co...
详细信息
This paper addresses the challenges associated with modeling storage systems and characteristics of hydrogen in the planning of island microgrids powered solely by renewable generation. We propose a novel mixed-Intege...
详细信息
Faults on distribution networks due to abnormal weather events can lead to disruption and can cause high socio-economic losses. In line with the rising frequency of such events, the paper proposes an algorithm for the...
详细信息
暂无评论