检索结果-内蒙古大学图书馆

Accelerating Massively distributed Deep Learning Through Efficient Pseudo-Synchronous Update Method

INTERNATIONAL JOURNAL OF PARALLEL PROGRAMMING 2024年第3期52卷 125-146页

作者： Wen, Yingpeng Qiu, Zhilin Zhang, Dongyu Huang, Dan Xiao, Nong Lin, Liang Sun Yat Sen Univ Sch Comp Sci & Engn Guangzhou Guangdong Peoples R China

In recent years, deep learning models have been successfully applied to large-scale data analysis, including image classification, video caption, natural language processing, etc. Large-scale data analyses take advantage of parallel computing to accelerate the speed of model training, in which data parallelism has become the dominant method for deep learning model training due to its high throughput rate. Synchronous stochastic gradient descent optimization becomes a well-recognized optimization method to ensure model convergence, but the overhead of gradients synchronization increases linearly as the number of workers increases, causing a huge waste of time. Although some efficiency-first asynchronous methods have been proposed, these methods cannot guarantee their convergence in large-scale distributed training. To solve this problem, we propose an efficient pseudo-synchronous approach that updates the network with the previous gradient, performing the synchronization of a new gradient to overlap computation and synchronization. This idea will obviously affect the normal convergence of the model, so we propose a novel adaptive exponential smoothing predicted gradient algorithm for model optimization, which can adaptively adjust the confidence coefficient of the history gradient to ensure the normal convergence of the training process. Experiments prove that our method can speed up the training process and achieve a comparable accuracy rate with standard synchronous SGD. Besides, our method has more efficient weak scalability compared to the traditional synchronous SGD and those in previous related work. We apply our methods to image recognition and video caption applications at most 12288 cores with strong scalability on Tianhe II. Evaluations show that, when configured appropriately, our method attains near-linear scalability using 128 nodes. We get 93.4% weak scaling efficiency on 64 nodes, 90.5% on 128 nodes.

关键词： Deep learning (DL) Stochastic gradient descent (SGD) Optimization distributed computing Tianhe-2 supercomputer Data parallel

来源：评论

学校读者我要写书评

暂无评论

distributed Evolution Strategies With Multi-Level Learning for Large-Scale Black-Box Optimization

引用

IEEE TRANSACTIONS ON PARALLEL AND distributed SYSTEMS 2024年第11期35卷 2087-2101页

作者： Duan, Qiqi Shao, Chang Zhou, Guochen Zhang, Minghan Zhao, Qi Shi, Yuhui Harbin Inst Technol Harbin 150001 Peoples R China Southern Univ Sci & Technol Shenzhen 518055 Peoples R China Univ Technol Sydney Australian Artificial Intelligence Inst Sydney NSW 2007 Australia Southern Univ Sci & Technol Dept Comp Sci & Engn Shenzhen 518055 Peoples R China Univ Warwick Coventry CV4 7AL England

In the post-Moore era, main performance gains of black-box optimizers are increasingly depending on parallelism, especially for large-scale optimization (LSO). Here we propose to parallelize the well-established covariance matrix adaptation evolution strategy (CMA-ES) and in particular its one latest LSO variant called limited-memory CMA-ES (LM-CMA). To achieve efficiency while approximating its powerful invariance property, we present a multilevel learning-based meta-framework for distributed LM-CMA. Owing to its hierarchically organized structure, Meta-ES is well-suited to implement our distributed meta-framework, wherein the outer-ES controls strategy parameters while all parallel inner-ESs run the serial LM-CMA with different settings. For the distribution mean update of the outer-ES, both the elitist and multi-recombination strategy are used in parallel to avoid stagnation and regression, respectively. To exploit spatiotemporal information, the global step-size adaptation combines Meta-ES with the parallel cumulative step-size adaptation. After each isolation time, our meta-framework employs both the structure and parameter learning strategy to combine aligned evolution paths for CMA reconstruction. Experiments on a set of large-scale benchmarking functions with memory-intensive evaluations, arguably reflecting many data-driven optimization problems, validate the benefits (e.g., effectiveness w.r.t. solution quality, and adaptability w.r.t. second-order learning) and costs of our meta-framework.

关键词： Parallel processing Optimization Computational modeling Closed box distributed computing Complexity theory Runtime Black-box optimization (BBO) distributed optimization evolution strategies (ESs) large-scale optimization (LSO) parallelism

来源：评论

学校读者我要写书评

暂无评论

Joint Resource Allocation for RIS-Assisted Heterogeneous Networks With Centralized and distributed Frameworks

引用

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS I-REGULAR PAPERS 2024年第5期71卷 2132-2145页

作者： Cai, Yujin Yu, Wenwu Nie, Xiaokai Cheng, Qiang Cui, Tiejun Southeast Univ Sch Cyber Sci Engn Nanjing 210096 Peoples R China Purple Mt Labs Nanjing 211111 Peoples R China Southeast Univ Sch Math Nanjing 210096 Jiangsu Peoples R China Southeast Univ Sch Automat Nanjing 210096 Peoples R China Southeast Univ Shenzhen Res Inst Shenzhen 518057 Peoples R China Southeast Univ State Key Lab Millimeter Waves Nanjing 210096 Peoples R China

Reconfigurable intelligent surface (RIS) is a radical and cost-efficient technology to improve energy efficiency and mitigate interference in the heterogeneous network. In this paper, the resource allocation problem of sub-channels, transmit power and RIS coefficients is investigated in the RIS-aided heterogeneous network. To solve the formulated mixed integer nonlinear programming problem, a two-step centralized resource allocation algorithm and a two-step distributed resource allocation algorithm are proposed based on the alternating optimization method. In the centralized algorithm, the sub-channel allocation, transmit power and RIS coefficients are optimized by the macro base station solely, where the non-convex power optimization problem is transformed into a convex one based on the convex approximation method. In the distributed algorithm, which aims to alleviate the computational burden of the macro base station, the sub-channel allocation and transmit power are optimized by using the cooperation of all the small base stations and the macro base station. Finally, numerical results are presented to demonstrate the convergence of the proposed algorithms and the effectiveness of the sub-channel transfer. More importantly, it is shown that the centralized algorithm can achieve the higher total throughput, while the distributed algorithm greatly decreases the resource allocation time.

关键词： Reconfigurable intelligent surface heterogeneous network resource allocation distributed computing amplitude coefficient optimization phase shift optimization

来源：评论

学校读者我要写书评

暂无评论

DiGTreeS: a distributed resilient framework for generalized tree search

引用

JOURNAL OF SUPERcomputing 2024年第10期80卷 15006-15037页

作者： Jamal, Md Arshad Kailasam, Sriram Goyal, Bhumanyu Singh, Varun Indian Inst Technol Sch Comp & Elect Engn Mandi India Natl Inst Technol Dept Comp Sci & Engn Warangal India Microsoft Corp Bengaluru India Microsoft Corp Hyderabad India

Exact combinatorial search algorithms have applications in several areas of computational algebra, AI, discrete optimization, etc. These problems are compute-intensive and have a highly irregular search tree. Most of the earlier efforts to parallelize these algorithms used a fixed degree of parallelism during runtime. We show that such an approach leads to poor resource utilization as the parallel run-time efficiency of an irregular search application varies over time. We propose DiGTreeS, a distributed resilient framework for generalized tree search that supports elastic scaling. It features an easy-to-use API for expressing combinatorial search and hides away the system concerns such as load balancing, fault tolerance, and elastic scaling. We evaluate the DiGTreeS framework for different scaling strategies and show its effectiveness on four representative problem instances: Traveling Salesman Problem, 0-1 Knapsack, N-queens, and Generic State Space Search Application.

关键词： distributed computing Elastic scaling Irregular tree search Fault tolerance

来源：评论

学校读者我要写书评

暂无评论

L_(1)-Smooth SVM with distributed Adaptive Proximal Stochastic Gradient Descent with Momentum for Fast Brain Tumor Detection

引用

Computers, Materials & Continua 2024年第5期79卷 1975-1994页

作者： Chuandong Qin Yu Cao Liqun Meng School of Mathematics and Information Science North Minzu UniversityYinchuan750021China Ningxia Key Laboratory of Intelligent Information and Big Data Processing Yinchuan750021China

Brain tumors come in various types,each with distinct characteristics and treatment approaches,making manual detection a time-consuming and potentially ambiguous *** tumor detection is a valuable tool for gaining a deeper understanding of tumors and improving treatment *** learning models have become key players in automating brain tumor *** descent methods are the mainstream algorithms for solving machine learning *** this paper,we propose a novel distributed proximal stochastic gradient descent approach to solve the L_(1)-Smooth Support Vector Machine(SVM)classifier for brain tumor ***,the smooth hinge loss is introduced to be used as the loss function of *** avoids the issue of nondifferentiability at the zero point encountered by the traditional hinge loss function during gradient descent ***,the L_(1) regularization method is employed to sparsify features and enhance the robustness of the ***,adaptive proximal stochastic gradient descent(PGD)with momentum,and distributed adaptive PGDwithmomentum(DPGD)are proposed and applied to the L_(1)-Smooth *** computing is crucial in large-scale data analysis,with its value manifested in extending algorithms to distributed clusters,thus enabling more efficient processing ofmassive amounts of *** DPGD algorithm leverages Spark,enabling full utilization of the computer’s multi-core *** to its sparsity induced by L_(1) regularization on parameters,it exhibits significantly accelerated convergence *** the perspective of loss reduction,DPGD converges faster than *** experimental results show that adaptive PGD withmomentumand its variants have achieved cutting-edge accuracy and efficiency in brain tumor ***-trained models,both the PGD andDPGD outperform other models,boasting an accuracy of 95.21%.

关键词： Support vector machine proximal stochastic gradient descent brain tumor detection distributed computing

来源：评论

学校读者我要写书评

暂无评论

distributed computing Building Blocks for Rational Agents 14

Distributed Computing Building Blocks for Rational Agents

引用

ACM Symposium on Principles of distributed computing (PODC)

作者： Afek, Yehuda Ginzberg, Yehonatan Feibish, Shir Landau Sulamy, Moshe Tel Aviv Univ Blavatnik Sch Comp Sci Tel Aviv Israel

ISBN: (纸本)9781450329446

Following [4] we extend and generalize the game-theoretic model of distributed computing, identifying different utility functions that encompass different potential preferences of players in a distributed system. A good distributed algorithm in the game-theoretic context is one that prohibits the agents (processors with interests) from deviating from the protocol;any deviation would result in the agent losing, i.e., reducing its utility at the end of the algorithm. We distinguish between different utility functions in the context of distributed algorithms, e.g., utilities based on communication preference, solution preference, and output preference. Given these preferences we construct two basic building blocks for game theoretic distributed algorithms, a wake-up building block resilient to any preference and in particular to the communication preference (to which previous wake-up solutions were not resilient), and a knowledge sharing building block that is resilient to any and in particular to solution and output preferences. Using the building blocks we present several new algorithms for consensus, and renaming as well as a modular presentation of the leader election algorithm of [4].

关键词： distributed computing game theory message passing consensus renaming leader election knowledge sharing rational agents

来源：评论

学校读者我要写书评

暂无评论

Energy-Efficient distributed Spiking Neural Network for Wireless Edge Intelligence

引用

IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS 2024年第9期23卷 10683-10697页

作者： Liu, Yanzhen Qin, Zhijin Li, Geoffrey Ye Imperial Coll London Dept Elect & Elect Engn London England Tsinghua Univ Dept Elect Engn Beijing 100084 Peoples R China Beijing Natl Res Ctr Informat Sci & Technol Beijing 100084 Peoples R China

The spiking neural network (SNN) is distinguished by its ultra-low power consumption, making it attractive for resource-limited edge intelligence. This paper investigates an energy-efficient (EE) distributed SNN, where multiple edge nodes, each containing a subset of spiking neurons, collaborate to gather and process information through wireless channels. To leverage the benefits of the joint design of neuromorphic computing and wireless communications, we develop quantitative system models and formulate the problem of minimizing the energy consumption of edge devices under constraints of limited bandwidth and spike loss probability. Particularly, a simplified homogeneous SNN is first explored, where the system is proved to have stationary states with a constant firing rate and an alternating optimization based algorithm is proposed for jointly allocating the computation and communication resources. The algorithms are further extended to heterogeneous SNNs by exploiting the statistics of spikes. Extensive simulation results on neuromorphic datasets demonstrate that the developed algorithms can significantly reduce the power consumption of edge systems while ensuring inference accuracy. Moreover, SNNs achieve comparable performance with state-of-the-art recurrent neural networks (RNNs) but are much more bandwidth-efficient and energy-saving.

关键词： Neurons Wireless communication Delays Computational modeling Membrane potentials Bandwidth Firing Spiking neural network energy-efficient distributed computing resource allocation edge intelligence

来源：评论

学校读者我要写书评

暂无评论

Sentiment analysis based distributed recommendation system

引用

MULTIMEDIA TOOLS AND APPLICATIONS 2024年第25期83卷 66539-66563页

作者： Singh, Tinku Rajput, Vinarm Sharma, Nikhil Satakshi Kumar, Manish Chungbuk Natl Univ Sch Informat & Commun Engn Cheongju Chungcheongbug South Korea Indian Inst Informat Technol Allahabad Dept IT Prayagraj Uttar Pradesh India Dept Maths SHUATS Prayagraj Uttar Pradesh India

The recommendation system assists in selecting the best product among the millions of products available on various e-commerce sites. An effective recommendation system can save the user's time while looking for a suitable product. People are spending more time on the internet these days, and their interests vary over time;as a result, their preference data is accumulating at a rapid pace. The recommendation algorithms should be scalable enough to process this large-scale data efficiently. It should also preserve the relationship between time and user preferences. Matrix factorization is a useful technique in this case. It is, however, dependent on the users' previous transactions and confronts data sparsity and scalability issues. In this study, we proposed a recommendation model utilizing distributed alternating least square matrix factorization that incorporates product ratings along with user reviews. It utilizes the up-vote technique to justify user reviews and the weighted rating normalization approach to assign a normalized rating to the reviews, which helps in improving recommendations. The proposed method is scalable enough to efficiently process the large amount of data generated due to user reviews and ratings. Extensive experiments were performed to validate the outcomes on the well-known Amazon reviews dataset utilizing the Apache Spark cluster. The proposed methodology outperformed state-of-the-art models in product recommendations, achieving an average precision of 89.1% and an average recall of 84.1%.

关键词： Sentiment analysis distributed computing Recommendation systems Big data Apache spark

来源：评论

学校读者我要写书评

暂无评论

Edge/Cloud Infinite-Time Horizon Resource Allocation for distributed Machine Learning and General Tasks

引用

IEEE TRANSACTIONS ON NETWORK AND SERVICE MANAGEMENT 2024年第1期21卷 697-713页

作者： Sartzetakis, Ippokratis Soumplis, Polyzois Pantazopoulos, Panagiotis Katsaros, Konstantinos V. Sourlas, Vasilis Varvarigos, Emmanouel Natl Tech Univ Athens Sch Elect & Comp Engn Athens 15773 Greece Natl Tech Univ Athens Inst Commun & Comp Syst Athens 15773 Greece

Edge computing has emerged as a computing paradigm where the application and data processing takes place close to the end devices. It decreases the distances over which data transfers are made, offering reduced delay and fast speed of action for general data processing and store/retrieve jobs. The benefits of edge computing can also be reaped for distributed computation algorithms, where the cloud also plays an assistive role. In this context, an important challenge is to allocate the required resources at both edge and cloud to carry out the processing of data that are generated over a continuous ("infinite") time horizon. This is a complex problem due to the variety of requirements (resource needs, accuracy, delay, etc.) that may be posed by each computation algorithm, as well as the heterogeneous resources' features (e.g., processing, bandwidth). In this work, we develop a solution for serving weakly coupled general distributed algorithms, with emphasis on machine learning algorithms, at the edge and/or the cloud. We present a dual-objective Integer Linear Programming formulation that optimizes monetary cost and computation accuracy. We also introduce efficient heuristics to perform the resource allocation. We examine various distributed ML allocation scenarios using realistic parameters from actual vendors. We quantify trade-offs related to accuracy, performance and cost of edge/cloud bandwidth and processing resources. Our results indicate that among the many parameters of interest, the processing costs seem to play the most important role for the allocation decisions. Finally, we explore interesting interactions between target accuracy, monetary cost and delay.

关键词： Cloud and edge computing distributed computing distributed machine learning inference training resource allocation

来源：评论

学校读者我要写书评

暂无评论

Near-optimal distributed dominating set in bounded arboricity graphs

引用

distributed computing 2024年第4期37卷 387-398页

作者： Dory, Michal Ghaffari, Mohsen Ilchi, Saeed Univ Haifa Dept Comp Sci Haifa Israel MIT Dept Elect Engn & Comp Sci Cambridge MA USA Swiss Fed Inst Technol Dept Comp Sci Zurich Switzerland

We describe a simple deterministic O(e-1 log A) round distributed algorithm for (2a + 1)(1 + e) approximation of minimum weighted dominating set on graphs with arboricity at most a. Here A denotes the maximum degree. We also show a lower bound proving that this round complexity is nearly optimal even for the unweighted case, via a reduction from the celebrated KMW lower bound on distributed vertex cover approximation (Kuhn et al. in JACM 63:116, 2016). Our algorithm improves on all the previous results (that work only for unweighted graphs) including a randomized O(a(2)) approximation in O(log n) rounds (Lenzen et al. in International symposium on distributed computing, Springer, 2010), a deterministic O(a log ?) approximation in O(log A) rounds (Lenzen et al. in international symposium on distributed computing, Springer, 2010), a deterministic O(a) approximation in O(log(2)?) rounds (implicit in Bansal et al. in Inform Process Lett 122:21-24, 2017;Proceeding 17th symposium on discrete algorithms (SODA), 2006), and a randomized O(a) approximation in O(a log n) rounds (Morgan et al. in 35th International symposiumon distributed computing, 2021). We also provide a randomized O(a log ?) round distributed algorithm that sharpens the approximation factor to a (1 + o(1)). If each node is restricted to do polynomial-time computations, our approximation factor is tight in the first order as it is NP-hard to achieve a - 1 - e approximation (Bansal et al. in Inform Process Lett 122:21-24, 2017).

关键词： distributed computing Dominating set Arboricity Approximation algorithms

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：