检索结果-内蒙古大学图书馆

An energy-efficient federated learning method with gradient compression and parameter optimization

INTERNATIONAL JOURNAL OF MACHINE LEARNING AND CYBERNETICS 2025年 1-18页

作者： Huang, Huijie Chen, Zhenping Lu, You Zuo, Yi Feng, Jing Suzhou Univ Sci & Technol Suzhou 215009 Jiangsu Peoples R China Jiangsu Ind Intelligent & Low Carbon Technol Engn Suzhou 215000 Peoples R China Suzhou Key Lab Intelligent Low Carton Technol Appl Suzhou 215009 Jiangsu Peoples R China

To address the communication bottleneck in federated learning, in this paper, an energy-efficient gradient compression method called Qs\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q_s$$\end{document}Top-k-SGD, combines Top-k sparsification and stochastic quantization at s levels. A joint optimization model on FL communication cost and convergence speed is constructed to achieve the optimized compression parameters. First, the Top-k algorithm and stochastic s-level quantization are integrated into the federated learning by using sparsification to retain important gradient values and further quantizing these gradient values. A federated learning architecture based on Qs\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q_s$$\end{document}Top-k-SGD is proposed, and the response-based knowledge distillation technology is introduced to address the Non-i.i.d. data issue. Next, the convergence analysis of the proposed Qs\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q_s$$\end{document}Top-k-SGD-based federated learning algorithm is conducted, and the upper bound of the convergence speed of the proposed federated learning algorithm is quantitatively analyzed. Furthermore, by analyzing the amount of data that needs to be shared per round of iteration on devices, the communication cost of the proposed federated learning is estimated, and a joint optimization model for convergence speed and communication cost is established. With the introd

关键词： Energy-efficient federated learning Communication cost Convergence speed gradient compression Quantization Sparsification

来源：评论

学校读者我要写书评

暂无评论

Federated Learning for IoV Adaptive Vehicle Clusters: A Dynamic gradient compression Strategy

引用

IEEE INTERNET OF THINGS JOURNAL 2025年第11期12卷 17526-17541页

作者： Xing, Ling Zhao, Pengcheng Gao, Jianping Wu, Honghai Ma, Huahong Zhang, Xiaohui Henan Univ Sci & Technol Coll Informat Engn Luoyang 471003 Peoples R China Henan Univ Sci & Technol Sch Software Luoyang 471003 Peoples R China Henan Univ Sci & Technol Coll Vehicle & Traff Engn Luoyang 471003 Peoples R China

Federated learning (FL) has been extensively utilized in distributed learning scenarios for the Internet of Vehicles (IoV). However, two key challenges exist: 1) gradients are frequently transmitted between vehicles during FL training, which reduces the communication timeliness between the traffic participants and 2) the fixed gradient compression method cannot sufficiently adapt to the dynamic IoV network topology. Therefore, we design an adaptive vehicle clustering method constructed according to multiple attributes, such as computational resources, communication distance, and latency. Accordingly, we propose a dynamic gradient compression strategy that filters similar local training models between vehicles and uses the Wasserstein distance to compute a sparsity threshold. This threshold acts as a dynamic compression factor that compresses gradient model parameters, reducing redundant parameter transmission. Furthermore, we conduct experiments using two datasets to evaluate the proposed strategy's effectiveness. The compression ratio improved by 132- and 178-fold compared to the baselines, and the aggregated accuracy increased by an average of 10.13%. Additionally, experiments incorporating communication noise revealed that the aggregation model of the signal noise ratio is -29 dB.

关键词： Training Vehicle dynamics Adaptation models Computational modeling Accuracy Federated learning Convergence Quantization (signal) Network topology Internet of Vehicles Adaptive vehicle cluster federated learning (FL) gradient compression Internet of vehicles (IoV)

来源：评论

学校读者我要写书评

暂无评论

Stochastic gradient compression for Federated Learning over Wireless Network

引用

China Communications 2024年第4期21卷 230-247页

作者： Lin Xiaohan Liu Yuan Chen Fangjiong Huang Yang Ge Xiaohu School of Electronic and Information Engineering South China University of TechnologyGuangzhou 510641China Key Laboratory of Dynamic Cognitive System of Electromagnetic Spectrum Space Ministry of Industry and Information TechnologyNanjing University of Aeronautics and AstronauticsNanjing 210016China School of Electronic Information and Communications Huazhong University of Science and TechnologyWuhan 430074China

As a mature distributed machine learning paradigm,federated learning enables wireless edge devices to collaboratively train a shared AI-model by stochastic gradient descent(SGD).However,devices need to upload high-dimensional stochastic gradients to edge server in training,which cause severe communication *** address this problem,we compress the communication by sparsifying and quantizing the stochastic gradients of edge *** first derive a closed form of the communication compression in terms of sparsification and quantization ***,the convergence rate of this communicationcompressed system is analyzed and several insights are ***,we formulate and deal with the quantization resource allocation problem for the goal of minimizing the convergence upper bound,under the constraint of multiple-access channel *** show that the proposed scheme outperforms the benchmarks.

关键词： federated learning gradient compression quantization resource allocation stochastic gradient descent(SGD)

来源：评论

学校读者我要写书评

暂无评论

GFL-ALDPA: a gradient compression federated learning framework based on adaptive local differential privacy budget allocation

引用

MULTIMEDIA TOOLS AND APPLICATIONS 2024年第9期83卷 26349-26368页

作者： Yang, Jiawei Chen, Shuhong Wang, Guojun Wang, Zijia Jie, Zhiyong Arif, Muhammad Guangzhou Univ Sch Comp Sci & Cyber Engn Guangzhou 510006 Guangdong Peoples R China Super Univ Dept Comp Sci Lahore 54000 Pakistan

Federated learning(FL) is a popular distributed machine learning framework which can protect users' private data from being exposed to adversaries. However, related work shows that sensitive private information can still be compromised by analyzing parameters uploaded by clients. Applying differential privacy to federated learning has been a popular privacy-preserving way to achieve strict privacy guarantees in recent years. To reduce the impact of noise, this paper proposes to apply local differential privacy(LDP) to federated learning. We propose a gradient compression federated learning framework based on adaptive local differential privacy budget allocation(GFL-ALDPA). We propose a novel adaptive privacy budget allocation scheme based on communication rounds to reduce the loss of privacy budget and the amount of model noise. It can maximize the limited privacy budget and improve the model accuracy by assigning different privacy budgets to different communication rounds during training. Furthermore, we also propose a gradient compression mechanism based on dimension reduction, which can reduce the communication cost, overall noise size, and loss of the total privacy budget of the model simultaneously to ensure accuracy under a specific privacy-preserving guarantee. Finally, this paper presents the experimental evaluation on the MINIST dataset. Theoretical analysis and experiments demonstrate that our framework can achieve a better trade-off between privacy preservation, communication efficiency, and model accuracy.

关键词： Federated learning Differential privacy Privacy-preserving gradient compression Privacy budget allocation

来源：评论

学校读者我要写书评

暂无评论

Beyond Throughput and compression Ratios: Towards High End-to-end Utility of gradient compression 3

Beyond Throughput and Compression Ratios: Towards High End-t...

引用

23rd Workshop on Hot Topics in Networks

作者： Han, Wenchen Vargaftik, Shay Mitzenmacher, Michael Karp, Brad Ben Basat, Ran UCL London England VMware Res Palo Alto CA USA Harvard Univ Cambridge MA 02138 USA

ISBN: (纸本)9798400712722

gradient aggregation has long been identified as a major bottleneck in today's large-scale distributed machine learning training systems. One promising solution to mitigate such bottlenecks is gradient compression, directly reducing communicated gradient data volume. However, in practice, many gradient compression schemes do not achieve acceleration of the training process while also preserving accuracy. In this work, we identify common issues in previous gradient compression systems and evaluation methodologies. These include excessive computational overheads;incompatibility with all-reduce;and insufficient evaluation methods, such as not using an end-to-end metric or using a 32-bit baseline instead of the stronger 16-bit baseline. We revisit common compression approaches (sparsification, quantization, and low-rank decomposition) and demonstrate how considering the above issues can lead to minor but strategic design changes, resulting in notably better performance. Our goal is to raise awareness of the need for design and evaluation standards that naturally translate to the end-to-end utility of gradient compression.

关键词： gradient compression Collective communication All-reduce

来源：评论

学校读者我要写书评

暂无评论

FedSC: Compatible gradient compression for Communication-Efficient Federated Learning 23rd

FedSC: Compatible Gradient Compression for Communication-Eff...

引用

23rd International Conference on Algorithms and Architectures for Parallel Processing (ICA3PP)

作者： Yu, Xinlei Gao, Zhipeng Zhao, Chen Mo, Zijia Beijing Univ Posts & Telecommun State Key Lab Networking & Switching Technol Beijing Peoples R China

ISBN: (纸本)9789819708338;9789819708345

Federated Learning (FL) communication costs hinge on communication frequency, device count, and per-communication-round costs. Ideally, minimizing these within the device cluster tolerance significantly curtails data traffic. Among various methods to reduce per-communication-round costs, gradient compression stands out. gradient compression concentrates on the changes in the model rather than the model parameters. This not only prevents synchronization issues caused by varying compression levels across devices, but also incurs only minor precision loss. This makes it especially apt for distributed scenarios like FL. Existing gradient compression methods are tailored to capitalize on elements nearing zero in high-frequency settings, aiming for sparse representation and efficient encoding. Yet, in low-frequency scenarios where elements deviate from zero, this strong emphasis on sparsity results in significant compression errors. To resolve this, we propose Federated Statistical compression (FedSC) shifts the focus from sparsity. Designed specifically for low-frequency settings, it hones in on the inherent statistical characteristics and relationships among the gradient elements. It extracts statistical information from each gradient, replacing the original gradient sent to the central server. We further introduce hierarchical estimation to improve accuracy, grouping elements in a gradient by layer label. Experimentally, FedSC outperforms most existing gradient compression algorithms for FL, converging with fewer communication bits while maintaining high model accuracy.

关键词： Federated learning communication-efficient gradient compression

来源：评论

学校读者我要写书评

暂无评论

A Joint Approach to Local Updating and gradient compression for Efficient Asynchronous Federated Learning 30th

A Joint Approach to Local Updating and Gradient Compression ...

引用

30th European Conference on Parallel and Distributed Processing (Euro-Par)

作者： Song, Jiajun Luo, Jiajun Lu, Rongwei Xie, Shuzhao Chen, Bin Wang, Zhi Tsinghua Univ SIGS Shenzhen Peoples R China Tsinghua Univ TBSI Shenzhen Peoples R China Southern Univ Sci & Technol Dept Comp Sci & Engn Shenzhen Peoples R China Harbin Inst Technol Sch Comp Sci & Technol Shenzhen Peoples R China

ISBN: (纸本)9783031695827;9783031695834

Asynchronous Federated Learning (AFL) confronts inherent challenges arising from the heterogeneity of devices (e.g., their computation capacities) and low-bandwidth environments, both potentially causing stale model updates (e.g., local gradients) for global aggregation. Traditional approaches mitigating the staleness of updates typically focus on either adjusting the local updating or gradient compression, but not both. Recognizing this gap, we introduce a novel approach that synergizes local updating with gradient compression. Our research begins by examining the interplay between local updating frequency and gradient compression rate, and their collective impact on convergence speed. The theoretical upper bound shows that the local updating frequency and gradient compression rate of each device are jointly determined by its computing power, communication capabilities and other factors. Building on this foundation, we propose an AFL framework called FedLuck that adaptively optimizes both local update frequency and gradient compression rates. Experiments on image classification and speech recognization show that FedLuck reduces communication consumption by 56% and training time by 55% on average, achieving competitive performance in heterogeneous and low-bandwidth scenarios compared to the baselines.

关键词： Asynchronous Federated Learning gradient compression Local Update Staleness Joint Optimization

来源：评论

学校读者我要写书评

暂无评论

引用

IEEE/ACM 32nd International Symposium on Quality of Service (IWQoS)

作者： Liu, Jingling Huang, Jiawei Li, Yijun Li, Zhaoyi Lyu, Wenjun Jiang, Wenchao Wang, Jianxin Cent South Univ Changsha Peoples R China Rutgers State Univ New Brunswick NJ USA Singapore Univ Technol & Design Singapore Singapore

ISBN: (纸本)9798350350128

The collective communication has become the bottleneck of large-scale distributed deep learning due to the huge volume of gradients aggregated during the training process. Despite much recent progress in reducing traffic volume by compressing the stochastic gradients inside each training worker, how to share the inter-worker data redundancy to alleviate communication overhead has remained elusive. In this paper, we reveal that most gradients have a great similarity with close value among training workers. From this hypothesis, we propose a Similarity-guided gradient compression framework named SGC which skips aggregating the similar gradients among each worker which utilizes local one rather than average value to save communication expenses. Each worker utilizes local SGC firstly quantifies the similarity of gradients among workers, and then elaborately adjusts the aggregation frequency of similar gradients without hurting DNN model accuracy. Meanwhile, we theoretically analyze the convergency accuracy of SGC. The comprehensive evaluation demonstrates that SGC outperforms the state-of-the-art schemes by up to 47% in convergence time.

关键词： distributed deep learning gradient compression similar gradient

来源：评论

学校读者我要写书评

暂无评论

When ML Training Cuts Through Congestion: Just-in-Time gradient compression via Packet Trimming 3

When ML Training Cuts Through Congestion: Just-in-Time Gradi...

引用

23rd Workshop on Hot Topics in Networks

作者： Chen, Xiaoqi Vargaftik, Shay Ben Basat, Ran Purdue Univ W Lafayette IN 47907 USA VMware Res Palo Alto CA USA UCL London England VMware Res Grp Palo Alto CA USA

ISBN: (纸本)9798400712722

Distributed training of ML models generates significant network traffic when exchanging gradients and is sensitive to packet drops and retransmission caused by congestion when other traffic is sharing the network. While training processes can tolerate gradient compression to reduce traffic volume, it does not eliminate queue buildup in shallow-buffer switches, causing high latency and extended training time. Our observation is that the emerging capability of handling temporal congestion through packet trimming provides the perfect opportunity to enable just-in-time gradient compression. Accordingly, we propose a novel approach where gradient packets are specifically constructed to be compressible via packet trimming, thus avoiding retransmission delays. Preliminary evaluations indicate that our trimmable encoding based on the Randomized Hadamard Transform has low computational overhead and that it achieves good training quality and shorter time to accuracy at high trim rates of up to 50%.

关键词： Packet Trimming Distributed Training AllReduce gradient compression Collective Communications

来源：评论

学校读者我要写书评

暂无评论

Enhancing gradient compression for Distributed Deep Learning 8

Enhancing Gradient Compression for Distributed Deep Learning

引用

8th Asia-Pacific Workshop on Networking (APNET)

作者： Bai, Zhe Yu, Enda Dong, Dezun Lu, Pingjing Natl Univ Def Technol Coll Comp Changsha Peoples R China

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：