检索结果-内蒙古大学图书馆

deep neural network compression by Tucker decomposition with nonlinear response

KNOWLEDGE-BASED SYSTEMS 2022年 241卷

作者： Liu, Ye Ng, Michael K. South China Univ Technol Sch Future Technol Guangzhou Guangdong Peoples R China Univ Hong Kong Dept Math Pokfulam Hong Kong Peoples R China

deep neural networks have shown impressive performance in many areas, including computer vision and natural language processing. Millions of parameters in deep neural network limit its deployment in low-end devices due to intensive memory and expensive computational cost. In the literature, several network compression techniques based on tensor decompositions have been proposed to compress deep neural networks. Existing techniques are designed in each network unit by approximating linear response or kernel tensor using various tensor decomposition methods. What is more, research has shown that there exists significant redundancy between different filters and feature channels of kernel tensor in each convolution layer. In this paper, we propose a new algorithm to compress deep neural network by considering both nonlinear response and the multilinear low-rank constraint in the kernel tensor. To overcome the resulted difficulty of nonconvex optimization, we propose a convex relaxation scheme such that it can be solved by alternating direction method of multipliers (ADMM) directly. Thus, the Tucker-2 rank and the feature matrix of Tucker decomposition can be determined simultaneously. The effectiveness of the proposed method is evaluated on CIFAR-10 and large-scale ILSVRC12 datasets for CNNs including ResNet-18, AlexNet and GoogleNet. According to our numerical computation, the proposed method is able to obtain highly reduction in model size with a small loss in accuracy. The compression performance of the proposed method is better than existing methods. (C) 2022 Elsevier B.V. All rights reserved.

关键词： deep neural network compression Convolution kernel Low rank approximation Tucker decomposition Optimization

来源：评论

学校读者我要写书评

暂无评论

Evaluation of deep neural network compression Methods for Edge Devices Using Weighted Score-Based Ranking Scheme

引用

SENSORS 2021年第22期21卷 7529页

作者： Ademola, Olutosin Ajibola Leier, Mairo Petlenkov, Eduard Tallinn Univ Technol Dept Comp Syst Embedded AI Res Lab Ehitajate Tee 5 EE-19086 Tallinn Estonia Tallinn Univ Technol Ctr Intelligent Syst Dept Comp Syst Ehitajate Tee 5 EE-19086 Tallinn Estonia

The demand for object detection capability in edge computing systems has surged. As such, the need for lightweight Convolutional neural network (CNN)-based object detection models has become a focal point. Current models are large in memory and deployment in edge devices is demanding. This shows that the models need to be optimized for the hardware without performance degradation. There exist several model compression methods;however, determining the most efficient method is of major concern. Our goal was to rank the performance of these methods using our application as a case study. We aimed to develop a real-time vehicle tracking system for cargo ships. To address this, we developed a weighted score-based ranking scheme that utilizes the model performance metrics. We demonstrated the effectiveness of this method by applying it on the baseline, compressed, and micro-CNN models trained on our dataset. The result showed that quantization is the most efficient compression method for the application, having the highest rank, with an average weighted score of 9.00, followed by binarization, having an average weighted score of 8.07. Our proposed method is extendable and can be used as a framework for the selection of suitable model compression methods for edge devices in different applications.

关键词： deep neural network compression compression method evaluation weighted score-based ranking embedded deep learning edge computing

来源：评论

学校读者我要写书评

暂无评论

Compressing neural networks on Limited Computing Resources

引用

IEEE ACCESS 2025年 13卷 80063-80075页

作者： Lee, Seunghyun Lee, Dongjun Hyun, Minju Kim, Heeje Song, Byung Cheol Inha Univ Dept Elect & Comp Engn Incheon 22212 South Korea

network compression is a crucial technique for applying deep learning models to edge or mobile devices. However, the cost of achieving higher benchmark performance through compression is continuously increasing, making network compression a significant burden-especially for small industries focused on developing compact models. Specifically, existing network compression techniques often require extensive computational resources, rendering them impractical for edge devices and small-scale applications. To democratize network compression, we propose a general-purpose framework that combines novel filter pruning and knowledge distillation techniques. First, unlike conventional filter pruning methods based on static heuristics and costly neural architecture search (NAS)-based approaches, our method leverages meta-learning for rapid and fine examination of the importance of each gate. This enables rapid and stable sub-network discovery, significantly improving the pruning process. Second, to minimize the computational cost of knowledge distillation, we introduce a synthetic teacher assistant that leverages precomputed fixed knowledge-referring to the stored feature maps/logits of the teacher network. By leveraging fixed knowledge, we mitigate the cost incurred by the teacher network and facilitate the transmission of fixed knowledge to the student via synthetic teacher assistants, thereby preventing distribution collapse. Our proposed framework dramatically reduces the compression overhead while maintaining high accuracy, achieving a 55.2% reduction in FLOPs of ResNet-50 trained on ImageNet while preserving 76.2% top-1 accuracy with only 199 GPU hours-significantly lower than previous state-of-the-art methods. Overall, our framework democratizes deep learning compression by offering a cost-effective and computationally feasible solution, enabling broader adoption in low-resource environments.

关键词： Costs Knowledge engineering Logic gates Filtering algorithms Graphics processing units Computational modeling Accuracy Image coding deep learning Workstations deep neural network compression filter pruning knowledge transfer efficient neural networks lightweight deep learning models

来源：评论

学校读者我要写书评

暂无评论

QLP: deep Q-Learning for Pruning deep neural networks

引用

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 2022年第10期32卷 6488-6501页

作者： Camci, Efe Gupta, Manas Wu, Min Lin, Jie ASTAR Inst Infocomm Res I2R Singapore 138632 Singapore

We present a novel, deep Q-learning based method, QLP, for pruning deep neural networks (DNNs). Given a DNN, our method intelligently determines favorable layer-wise sparsity ratios, which are then implemented via unstructured, magnitude-based, weight pruning. In contrast to previous reinforcement learning (RL) based pruning methods, our method is not forced to prune a DNN within a single, sequential pass from the first layer to the last. It visits each layer multiple times and prunes them little by little at each visit, achieving superior granular pruning. Moreover, our method is not restricted to a subset of actions within the feasible action space. It has the flexibility to execute a whole range of sparsity ratios (0% - 100%) for each layer. This enables aggressive pruning without compromising accuracy. Furthermore, our method does not require a complex state definition;it features a simple, generic definition that is composed of only the index and the density of the layers, which leads to less computational demand while observing the state at each interaction. Lastly, our method utilizes a carefully designed curriculum that enables learning targeted policies for each sparsity regime, which helps to deliver better accuracy, especially at high sparsity levels. We conduct batched performance tests at compelling sparsity levels (up to 98%), present extensive ablation studies to justify our RL-related design choices, and compare our method with the state-of-the-art, including RL-based and other pruning methods. Our method sets the new state-of-the-art results in most of the experiments with ResNet-32 and ResNet-56 over CIFAR-10 dataset as well as ResNet-50 and MobileNet-v1 over ILSVRC2012 (ImageNet) dataset.

关键词： Training neural networks Indexes Computer architecture deep learning Biological neural networks Task analysis deep neural network compression pruning deep reinforcement learning

来源：评论

学校读者我要写书评

暂无评论

Multiaccess Edge Integrated networking for Internet of Vehicles: A Blockchain-Based deep Compressed Cooperative Learning Approach

引用

IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS 2022年第11期23卷 21593-21607页

作者： Zhang, Dajun Shi, Wei St-Hilaire, Marc Yang, Ruizhe Carleton Univ Dept Syst & Comp Engn Ottawa ON K1S 5B6 Canada Carleton Univ Sch Informat Technol Ottawa ON K1S 5B6 Canada Beijing Univ Technol Beijing Lab Adv Informat Networks Beijing 100021 Peoples R China

Recently, Internet of Vehicles (IoV) and Machine Learning (ML) have attracted more and more attention. Considering inefficient real-time training and high requirements on computing capabilities of centralized data collection, performing Distributed Machine Learning (DML) in IoV has become an important research branch. However, the heterogeneity, mobility, and distrust among IoV nodes affect how to execute DML effectively, securely, and in a salable manner. In this paper, a blockchain-based Cooperative Learning framework combined with a deep compression method (CLDC) is proposed. First, we improve the local training efficiency of lightweight IoV nodes by using deep compression method. Meanwhile, we have introduced a blockchain system in CLDC, the significance of which is that we have completed the transformation from centralized architecture to distributed framework through the blockchain, and shared local training results in a verifiable manner. The framework uses non-tamperable features of the blockchain to ensure the security of local training results. Moreover, we propose a Learning-based Redundant Byzantine Fault Tolerance (L-RBFT) protocol, in which the primary node needs to confirm the loss percentage of learning in the transaction before forwarding the RBFT messages. The significance of L-RBFT is to ensure that IoV nodes obtain the best training results through the consensus of blockchain nodes. We use it to solve the computing and communication resource allocation problem in IoV to clarify the operating mechanism of the proposed framework. The experimental results prove that this scheme performs better when compared with the traditional centralized deep reinforcement learning method.

关键词： Blockchains Computer architecture Training Servers deep learning neural networks Peer-to-peer computing Internet of Vehicles blockchain cooperative Q-learning deep neural network compression

来源：评论

学校读者我要写书评

暂无评论

Learning Pruned Structure and Weights Simultaneously from Scratch: an Attention based Approach

Learning Pruned Structure and Weights Simultaneously from Sc...

引用

2023 IEEE International Conference on Big Data, BigData 2023

作者： He, Qisheng Shi, Weisong Dong, Ming Wayne State University Department of Computer Science Detroit United States

ISBN: (纸本)9798350324457

As a deep learning model typically contains millions of trainable weights, there has been a growing demand for a more efficient network structure with reduced storage space and improved run-time efficiency. Pruning is one of the most popular network compression techniques. In this paper, we propose a novel unstructured pruning pipeline, Attention-based Simultaneous sparse structure and Weight Learning (ASWL). In ASWL, an efficient algorithm is proposed to calculate the pruning ratios layer-wisely from attentions, and both weights for the dense network and the sparse network are tracked so that the pruned structure is simultaneously learned from randomly initialized weights. Our experiments on MNIST, Cifar10, and ImageNet show that ASWL achieves superior pruning results in terms of accuracy, pruning ratio and operating efficiency when compared with state-of-the-art network pruning methods. © 2023 IEEE.

关键词： deep Learning deep neural network compression network Pruning

来源：评论

学校读者我要写书评

暂无评论

Transferring Lottery Tickets in Computer Vision Models: A Dynamic Pruning Approach

Transferring Lottery Tickets in Computer Vision Models: A Dy...

引用

2023 IEEE International Conference on Big Data, BigData 2023

作者： He, Qisheng Dong, Ming Wayne State University Department of Computer Science Detroit United States

ISBN: (纸本)9798350324457

deep neural networks can achieve state-of-the-art results on small size datasets by transferring the backbone from a network pre-trained on large datasets. Recent work has shown that pruned networks can also be used as pre-trained models in transfer learning. In this paper, we proposed a novel framework, Transferring Lottery Ticket (TLT), to adapt both masks and weights of a pre-trained and pruned network dynamically during the knowledge transfer to downstream tasks. We show that the lottery tickets of downstream tasks are dramatically different from each other and from the one obtained from the pre-trained network. Thus, both masks and weights need to be learned to better adapt a pre-trained model to the target domain. Our extensive experiments on multiple computer vision tasks, such as image classification and segmentation, show that the transferred networks with adapted masks outperform the ones with original masks at various pruning ratios. © 2023 IEEE.

关键词： deep Learning deep neural network compression Lottery Tickets network Pruning Transfer Learning

来源：评论

学校读者我要写书评

暂无评论

Energy-Efficient deep neural network Optimization via Pooling-Based Input Masking

Energy-Efficient Deep Neural Network Optimization via Poolin...

引用

IEEE International Conference on Fuzzy Systems (FUZZ-IEEE) / IEEE World Congress on Computational Intelligence (IEEE WCCI) / International Joint Conference on neural networks (IJCNN) / IEEE Congress on Evolutionary Computation (IEEE CEC)

作者： Ren, Jiankang Lv, Huawei Bi, Ran Liu, Qian Ni, Zheng Tan, Guozhen Dalian Univ Technol Sch Comp Sci & Technol Dalian Peoples R China

ISBN: (数字)9781728186719

ISBN: (纸本)9781728186719

deep neural networks (DNNs) are increasingly deployed in battery-powered and resource-constrained devices. However, the most accurate DNNs usually require millions of parameters and operations, making them computation-heavy and energy-expensive, so it is an important topic to develop energy efficient DNN models. In this paper, we present an efficient DNN training framework under energy constraint to improve the energy efficiency of DNN inference. The key idea of this research is inspired by the observation that the input data of DNNs is usually inherently sparse and such sparsity can be exploited by sparse tensor DNN accelerators to eliminate ineffectual data access and compute. Therefore, we can enhance the inference accuracy within the energy budget by strategically controlling the sparsity of the input data. We build an energy consumption model for the sparse tensor DNN accelerator to quantify the inference energy consumption from the perspective of data access and data processing. In particular, we define a metric (named sporadic degree) to characterise the influence of the number of sporadic values in the sparse input on the energy consumption of data access for the sparse tensor DNN accelerator. Based on the proposed quantitative energy consumption model, we present an efficient pooling-based input mask training algorithm to optimize the energy efficiency of DNN inference by enhancing the input sparsity and reducing the number of sporadic values in the masked input. Experiments show that compared with the state-of-the-art methods, our proposed method can achieve higher inference accuracy with lower energy consumption and storage requirement owing to higher sparsity and lower sporadic degree of the masked input.

关键词： deep neural network compression input sparsity energy constraint pooling input mask

来源：评论

学校读者我要写书评

暂无评论

NMS-KSD: Efficient Knowledge Distillation for Dense Object Detection via Non-Maximum Suppression and Feature Storage

引用

IEEE ACCESS 2025年 13卷 80723-80736页

作者： Son, Suho Song, Byung Cheol Inha Univ Dept Elect & Comp Engn Incheon 22212 South Korea

Recently, many studies have proposed knowledge distillation (KD) frameworks for object detection. However, these frameworks did not take into account the inefficiencies caused by the teacher detector. The inefficiency refers to the computational cost incurred during the process of passing input data to the teacher model to acquire its knowledge. To solve this inefficiency in image classification, Fast Knowledge Distillation (FKD) was proposed, which stores the teacher model's knowledge in advance and then uses it in the distillation process. However, directly applying FKD's knowledge storage mechanism to dense object detectors causes a storage space problem. To address this issue, we propose NMS-KSD, a novel knowledge storage distillation method designed for dense object detection tasks. The core of NMS-KSD is the integration of Non-Maximum Suppression (NMS) and channel max pooling to effectively select and store key features from the teacher model's intermediate feature maps. By storing and reusing these key features, NMS-KSD addresses the inefficiencies of traditional KD frameworks, significantly reducing training time while maintaining high performance. We validate the effectiveness and efficiency of our method across various dense object detectors through extensive experiments on the COCO, PASCAL VOC, and Cityscapes datasets.

关键词： Object detection Detectors Feature extraction Computational modeling Training Predictive models Computational efficiency Proposals Image classification Data augmentation Dense object detection deep neural network compression knowledge distillation non-maximum suppression

来源：评论

学校读者我要写书评

暂无评论

A Fast View Synthesis Implementation Method for Light Field Applications

引用

ACM TRANSACTIONS ON MULTIMEDIA COMPUTING COMMUNICATIONS AND APPLICATIONS 2021年第4期17卷 1–20页

作者： Gao, Wei Zhou, Linjie Tao, Lvfang Peking Univ Sch Elect & Comp Engn Shenzhen Grad Sch Shenzhen 518055 Peoples R China Peng Cheng Lab Artificial Intelligence Res Ctr Shenzhen 518055 Peoples R China

View synthesis (VS) for light field images is a very time-consuming task due to the great quantity of involved pixels and intensive computations, which may prevent it from the practical three-dimensional real-time systems. In this article, we propose an acceleration approach for deep learning-based light field view synthesis, which can significantly reduce calculations by using compact-resolution (CR) representation and super-resolution (SR) techniques, as well as light-weight neural networks. The proposed architecture has three cascaded neural networks, including a CR network to generate the compact representation for original input views, a VS network to synthesize new views from down-scaled compact views, and a SR network to reconstruct high-quality views with full resolution. All these networks are jointly trained with the integrated losses of CR, VS, and SR networks. Moreover, due to the redundancy of deep neural networks, we use the efficient light-weight strategy to prune filters for simplification and inference acceleration. Experimental results demonstrate that the proposed method can greatly reduce the processing time and become much more computationally efficient with competitive image quality.

关键词： deep learning-based view synthesis real-time acceleration compact representation filter pruning deep neural network compression light field systems

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：