检索结果-内蒙古大学图书馆

International Symposium on parallel and distributed Processing with Applications, ISPA

作者： Yanqi Hao Zhiquan Lai Wei Wang Shengwei Li Weijie Liu Keshi Ge Dongsheng Li National Key Laboratory of Parallel and Distributed Computing(PDL) College of Computer National University of Defense Technology Changsha China

ISBN: (数字)9798331509712

ISBN: (纸本)9798331509729

Large deep neural network (DNN) models have demonstrated exceptional performance across diverse downstream tasks. Sharded data parallelism (SDP) has been widely used to reduce the memory footprint of model states. In a DNN training cluster, a device usually has multiple inter-device links that connect to other devices, like NVLink and InfiniBand. However, existing SDP approaches employ a single link at any given time, encountering challenges in efficient training due to significant communication overheads. We observe that the inter-device links can work independently without affecting each other. To reduce the fatal communication overhead of distributed training of large DNNs, this paper introduces HSDP, an efficient SDP training approach that enables the simultaneous utilization of multiple inter-device links. HSDP partitions models in a novel fine-grained manner and orchestrates the communication processes of partitioned parameters while considering inter-device links. This design enables concurrent communication execution and reduces communication overhead. To further optimize the training performance of HSDP, we propose a HSDP planner. The HSDP planner first abstracts the model partition and execution of HSDP into a communication parallel strategy, and builds a cost model to estimate the performance of each strategy. We then formulate the strategy searching as an optimization problem and solve it with an off-the-shelf solver. Evaluations on representative DNN workloads demonstrate that HSDP achieves up to 1.30× speedup compared to the state-of-the-art SDP training approaches.

关键词： Training Performance evaluation Costs distributed databases Artificial neural networks parallel processing Throughput Search problems Data models Optimization

来源：评论

学校读者我要写书评

暂无评论

3D parallelism for Transformers via Integer Programming

3D Parallelism for Transformers via Integer Programming

引用

International Conference on Acoustics, Speech, and Signal Processing (ICASSP)

作者： Hao Zheng Peng Liang Yu Tang Yanqi Shi Linbo Qiao Dongsheng Li National Key Laboratory of Parallel and Distributed Computing National University Of Defense Technology Changsha P.R.China

Transformer models, such as BERT, GPT, and ViT, have been applied to a wide range of areas in recent years, due to their efficacy. In order to improve the training efficiency of Transformer models, different distributed training approaches have been proposed, like Megatron-LM [8]. However, when multi-dimensional parallelism strategies are considered, due to the complexity, existing works can not harmonize the different strategies well enough to obtain a globally optimal solution. In this paper, we propose a parallelism strategy searching algorithm PTIP, which generates operator-level parallelism strategies consisting of three schemes: data parallelism, tensor parallelism, and pipeline parallelism. PTIP abstracts these three parallelism schemes simultaneously into an auxiliary graph, reformulates the searching problem into a mixed-integer programming (MIP) problem, and uses a MIP solver to obtain a high-quality multi-dimensional strategy. Experiments conducted on Transformers demonstrate that PTIP obtains 13.9% − 24.7% performance improvement compared to Megatron-LM [8].

关键词：

来源：评论

学校读者我要写书评

暂无评论

Noncontiguous I/O accesses through MPI-IO 03

Noncontiguous I/O accesses through MPI-IO

引用

IEEE/ACM International Symposium on Cluster computing and the Grid (CCGRID)

作者： A. Ching A. Choudhary K. Coloma Wei-keng Liao R. Ross W. Gropp Center for Parallel and Distributed Computing Northwestern University Evanston IL USA Argonne National Laboratory Argonne IL USA

ISBN: (纸本)9780769519197

I/O performance remains a weakness of parallel computing systems today. While this weakness is partly attributed to rapid advances in other system components, I/O interfaces available to programmers and the I/O methods supported by file systems have traditionally not matched efficiently with the types of I/O operations that scientific applications perform, particularly noncontiguous accesses. The MPI-IO interface allows for rich descriptions of the I/O patterns desired for scientific applications and implementations such as ROMIO have taken advantage of this ability while remaining limited by underlying file system methods. A method of noncontiguous data access, list I/O, was recently implemented in the parallel Virtual File System (PVFS). We implement support for this interface in the ROMIO MPI-IO implementation. Through a suite of noncontiguous I/O tests we compared ROMIO list I/O to current methods of ROMIO noncontiguous access and found that the list I/O interface provides performance benefits in many noncontiguous cases.

关键词： File systems Testing distributed computing Mathematics Computer science Laboratories parallel processing Programming profession Tiles Checkpointing

来源：评论

学校读者我要写书评

暂无评论

An Adaptive Quality of Service Based Scheduling Algorithm for Wide Area Large Scale Problems

An Adaptive Quality of Service Based Scheduling Algorithm fo...

引用

IEEE International Symposium on Network computing and Applications

作者： W. Lozano W. Rivera Parallel and Distributed Computing Laboratory Electrical and Computer Engineering Department University of Puerto Rico MayagÜez Puerto Rico

This paper explores the problem of dynamically scheduling large scale applications over wide area networks and then proposes an adaptive scheduling algorithm to provide quality of service. Our adaptive algorithm takes into account unexpected events and priority fluctuations and gives high priority to jobs with low probability of failure. Experimental results show that the new approach outperforms traditional scheduling approaches in grid computing environments

关键词： Quality of service Scheduling algorithm Large-scale systems Dynamic scheduling Wide area networks Adaptive scheduling Adaptive algorithm Fluctuations Processor scheduling Grid computing

来源：评论

学校读者我要写书评

暂无评论

Assessment of two pressure-velocity coupling strategies for local meshless numerical method

Assessment of two pressure-velocity coupling strategies for ...

引用

作者： Kosec, G. Vertnik, R. Šarler, B. Jožef Stefan Institute Laboratory for Parallel and Distributed Computing Slovenia University of Nova Gorica Laboratory for Multiphase Processes Slovenia Štore Steel D.o.o. Research Slovenia Centre of Excellence for Biosensors Automation and Process Control Laboratory for Advanced Materials Systems Slovenia

The performance of two pressure-velocity coupling strategies, associated with the primitive variable solution of incompressible Newtonian fluid with the meshless Local Radial Basis Function Collocation Method (LRBFCM), is compared with respect to computational efficiency, stability, accuracy and spatial convergence. The LRBFCM is structured with multiquadrics on five noded support domains. The explicit time stepping is used. The Backward-Facing Step problem (BFS) has been selected as a benchmark problem, previously tackled by several numerical methods. The semi-local fractional step method (FSM) and completely local pressure-velocity couplings (LPVC) are compared. The numerical results are validated against previously published data. The results are represented and compared in terms of a convergence plot of the reattachment position. We show that both approaches provide reasonable results;however, LVPC is less computationally complex and less stable in comparison to FSM. © 2012 WIT Press.

关键词： Computational efficiency

来源：评论

学校读者我要写书评

暂无评论

Crowd intelligence in AI 2.0 era

引用

Frontiers of Information Technology & Electronic Engineering 2017年第1期18卷 15-43页

作者： Wei LI Wen-jun WU Huai-min WANG Xue-qi CHENG Hua-jun CHEN Zhi-hua ZHOU Rong DING State Key Laboratory of Software Development Beihang University National Laboratory for Parallel and Distributed Processing College of ComputerNational University of Defense Technology Institute of Computing Technology Chinese Academy of Sciences College of Computer Science and Technology Zhejiang University National Key Laboratory for Novel Software Technology Nanjing University

The Internet based cyber-physical world has profoundly changed the information environment for the development of artificial intelligence(AI), bringing a new wave of AI research and promoting it into the new era of AI 2.0. As one of the most prominent characteristics of research in AI 2.0 era, crowd intelligence has attracted much attention from both industry and research communities. Specifically, crowd intelligence provides a novel problem-solving paradigm through gathering the intelligence of crowds to address challenges. In particular, due to the rapid development of the sharing economy, crowd intelligence not only becomes a new approach to solving scientific challenges, but has also been integrated into all kinds of application scenarios in daily life, e.g., online-tooffline(O2O) application, real-time traffic monitoring, and logistics management. In this paper, we survey existing studies of crowd intelligence. First, we describe the concept of crowd intelligence, and explain its relationship to the existing related concepts, e.g., crowdsourcing and human computation. Then, we introduce four categories of representative crowd intelligence platforms. We summarize three core research problems and the state-of-the-art techniques of crowd intelligence. Finally, we discuss promising future research directions of crowd intelligence.

关键词： Crowd intelligence Artificial intelligence 2.0 Crowdsourcing Human computation

来源：评论

学校读者我要写书评

暂无评论

Pro-Prophet: A Systematic Load Balancing Method for Efficient parallel Training of Large-scale MoE Models

arXiv

引用

arXiv 2024年

作者： Wang, Wei Lai, Zhiquan Li, Shengwei Liu, Weijie Ge, Keshi Shen, Ao Su, Huayou Li, Dongsheng The National Key Laboratory of Parallel and Distributed Computing College of Computer National University of Defense Technology Hunan Changsha China

he size of deep learning models has been increasing to enhance model quality. The linear increase in training computation budget with model size means that training an extremely large-scale model is exceedingly time-consuming. Recently, the Mixture of Expert (MoE) has drawn significant attention as it can scale models to extra-large sizes with a stable computation budget. However, inefficient distributed training of large-scale MoE models hinders their broader application. Specifically, a considerable dynamic load imbalance occurs among devices during training, significantly reducing throughput. Several load-balancing works have been proposed to address the challenge. System-level solutions draw more attention for their hardware affinity and non-disruption of model convergence compared to algorithm-level ones. However, they are troubled by high communication costs and poor communication-computation overlapping. To address these challenges, we propose a systematic load-balancing method, Pro-Prophet, which consists of a planner and a scheduler for efficient parallel training of large-scale MoE models. To adapt to the dynamic load imbalance, we profile training statistics and use them to design Pro-Prophet. For lower communication volume, Pro-Prophet planner determines a series of lightweight load-balancing strategies and efficiently searches for a communication-efficient one for training based on the statistics. For sufficient overlapping of communication and computation, Pro-Prophet scheduler schedules the data-dependent operations based on the statistics and operation features, further improving the training throughput. We conduct extensive experiments in four clusters and five MoE models. The results indicate that Pro-Prophet achieves up to 2.66x speedup compared to two popular MoE frameworks including Deepspeed-MoE and FasterMoE. Furthermore, Pro-Prophet has demonstrated a load-balancing improvement of up to 11.01x compared to a representative load-balancing work,

关键词： Budget control

来源：评论

学校读者我要写书评

暂无评论

Using Grid computing to Enable distributed Radar Data Retrieval and Processing

Using Grid Computing to Enable Distributed Radar Data Retrie...

引用

IEEE International Conference on parallel and distributed computing, Applications and Technologies (PDCAT)

作者： Diego Arias Wilson Rivera Parallel and Distributed Computing Laboratory Electrical and Computer Engineering Department University of Puerto Rico MayagÜez Puerto Rico

We describe a tool that implements a set of services to manipulate and store data from a radar network in a transparent way to end users. A major requirement of this system is data availability and reliability. Consequently, we have implemented a redundancy schema based on the information dispersal algorithm (IDA). Preliminary results show that the IDA based replication provides better reliability and less storage spending than traditional replication

关键词： Grid computing Radar Information retrieval Atmosphere Earth distributed computing Computer networks Concurrent computing Reliability engineering Availability

来源：评论

学校读者我要写书评

暂无评论

On implementation of an enhanced and integrated parallel programming web-based toolkit

On implementation of an enhanced and integrated parallel pro...

引用

ITRE 2005 - 3rd International Conference on Information Technology: Research and Education

作者： Li, Kuan-Ching Yang, Chao-Tung Parallel and Distributed Processing Center Dept. of Computer Science and Information Management Providence University Shalu Taichung 43301 Taiwan High Performance Computing Laboratory Dept. of Computer Science and Information Engineering Tunghai University Taichung City Taichung 40704 Taiwan

ISBN: (纸本)0780389328

The computing power provided by high performance and low cost PC-based clusters and Grid computing platforms are attractive and they are equal or superior to supercomputers and mainframes. In parallel, discussions on how to obtain more computing power from these computing platforms become an interesting issue. The development of applications for these high-performance computing platforms is complicated for several reasons: The complexity of applications themselves, which combines aspects of supercomputing and distributed computing, and by the need to achieve higher performance. This paper describes the design rationale and implementation of a parallel programming web-based toolkit, to ease the parallel programming learning process, with the use of web-based interface. The toolkit has widely been used in MPI parallel programming courses (both in graduate and undergraduate levels) and industry trainings. © 2005 IEEE.

关键词： Computer programming

来源：评论

学校读者我要写书评

暂无评论

Performance problems online detection in cloud computing systems via analyzing request execution paths

Performance problems online detection in cloud computing sys...

引用

International Conference on Dependable Systems and Networks Workshops (DSN-W)

作者： Haibo Mi Huaimin Wang Gang Yin Hua Cai Qi Zhou Tingtao Sun National Laboratory for Parallel and Distributed Processing National University of Defense Technology Changsha China Computing Platform Alibaba Cloud Computing Company Hangzhou China

It is quite a headache for developers to online detect performance problems in large-scale cloud computing systems. The behavior and the hidden connections among the huge amount of runtime request execution paths in cloud computing systems usually contain useful information for performance problem detection. In this paper, we propose an approach to rapidly diagnose the source of performance degradation in large-scale non-stop cloud computing systems. The approach first groups the user requests into categories with a fast clustering algorithm; then applies the principal components analysis to extract the primary methods; finally compares the normal and abnormal behaviors of the primary methods to localize the main cause of performance problems. We conduct extensive experiments over a real-world enterprise system providing services for the public. The results show that our approach can locate the prime causes of performance problems accurately and efficiently.

关键词： Cloud computing Principal component analysis Instruments Degradation Servers Data mining Accuracy

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：