检索结果-内蒙古大学图书馆

International Conference on Quality Software

作者： Ji Wang Rui Shen H. Zhu National Laboratory for Parallel and Distributed Processing Changsha China Department of Computing Oxford Brookes University Oxford UK

The paper presents a caste-centric approach to agent-oriented programming by introducing SLABSp language. The fundamental concepts of caste-centric methodology, caste and scenario, as well as environment descriptions, are available as language facilities in SLABSp in a coherent way. In SLABSp programming, agents are organized into castes to represent their structure and behavior characteristics, and their behaviors are defined by scenarios and rules in the context of their environment. The relations between agents and castes are bound at runtime, and the perceptions and interactions between agents are supported with scenarios and behavior rules. Two selected SLABSp programs are demonstrated to illustrate the programming style.

关键词： Programming profession Computer languages Object oriented modeling Runtime Multiagent systems Research and development Laboratories distributed processing Concurrent computing distributed computing

来源：评论

学校读者我要写书评

暂无评论

An Approach for Modeling and Ranking Node-Level Stragglers in Cloud Datacenters

An Approach for Modeling and Ranking Node-Level Stragglers i...

引用

IEEE International Conference on Services computing (SCC)

作者： Xue Ouyang Peter Garraghan Changjian Wang Paul Townend Jie Xu Parallel and Distributed Laboratory National University of Defense Technology Changsha China School of Computing University of Leeds Leeds UK

The ability of servers to effectively execute tasks within Cloud datacenters varies due to heterogeneous CPU and memory capacities, resource contention situations, network configurations and operational age. Unexpectedly slow server nodes (node-level stragglers) result in assigned tasks becoming task-level stragglers, which dramatically impede parallel job execution. However, it is currently unknown how slow nodes directly correlate to task straggler manifestation. To address this knowledge gap, we propose a method for node performance modeling and ranking in Cloud datacenters based on analyzing parallel job execution tracelog data. By using a production Cloud system as a case study, we demonstrate how node execution performance is driven by temporal changes in node operation as opposed to node hardware capacity. Different sample sets have been filtered in order to evaluate the generality of our framework, and the analytic results demonstrate that node abilities of executing parallel tasks tend to follow a 3-parameter-loglogistic distribution. Further statistical attribute values such as confidence interval, quantile value, extreme case possibility, etc. can also be used for ranking and identifying potential straggler nodes within the cluster. We exploit a graph-based algorithm for partitioning server nodes into five levels, with 0.83% of node-level stragglers identified. Our work lays the foundation towards enhancing scheduling algorithms by avoiding slow nodes, reducing task straggler occurrence, and improving parallel job performance.

关键词： Servers Production Data models Computational modeling Analytical models Time factors Calculators

来源：评论

学校读者我要写书评

暂无评论

HSDP: Accelerating Large-scale Model Training via Efficient Sharded Data parallelism

HSDP: Accelerating Large-scale Model Training via Efficient ...

引用

International Symposium on parallel and distributed Processing with Applications, ISPA

作者： Yanqi Hao Zhiquan Lai Wei Wang Shengwei Li Weijie Liu Keshi Ge Dongsheng Li National Key Laboratory of Parallel and Distributed Computing(PDL) College of Computer National University of Defense Technology Changsha China

ISBN: (数字)9798331509712

ISBN: (纸本)9798331509729

Large deep neural network (DNN) models have demonstrated exceptional performance across diverse downstream tasks. Sharded data parallelism (SDP) has been widely used to reduce the memory footprint of model states. In a DNN training cluster, a device usually has multiple inter-device links that connect to other devices, like NVLink and InfiniBand. However, existing SDP approaches employ a single link at any given time, encountering challenges in efficient training due to significant communication overheads. We observe that the inter-device links can work independently without affecting each other. To reduce the fatal communication overhead of distributed training of large DNNs, this paper introduces HSDP, an efficient SDP training approach that enables the simultaneous utilization of multiple inter-device links. HSDP partitions models in a novel fine-grained manner and orchestrates the communication processes of partitioned parameters while considering inter-device links. This design enables concurrent communication execution and reduces communication overhead. To further optimize the training performance of HSDP, we propose a HSDP planner. The HSDP planner first abstracts the model partition and execution of HSDP into a communication parallel strategy, and builds a cost model to estimate the performance of each strategy. We then formulate the strategy searching as an optimization problem and solve it with an off-the-shelf solver. Evaluations on representative DNN workloads demonstrate that HSDP achieves up to 1.30× speedup compared to the state-of-the-art SDP training approaches.

关键词： Training Performance evaluation Costs distributed databases Artificial neural networks parallel processing Throughput Search problems Data models Optimization

来源：评论

学校读者我要写书评

暂无评论

DaMSTF: Domain Adversarial Learning Enhanced Meta Self-Training for Domain Adaptation

arXiv

引用

arXiv 2023年

作者： Lu, Menglong Huang, Zhen Zhao, Yunxiang Tian, Zhiliang Liu, Yang Li, Dongsheng National Key Laboratory of Parallel and Distributed Computing National University of Defense Technology China Beijing Institute of Biotechnology China

Self-training emerges as an important research line on domain adaptation. By taking the model’s prediction as the pseudo labels of the unlabeled data, self-training bootstraps the model with pseudo instances in the target domain. However, the prediction errors of pseudo labels (label noise) challenge the performance of self-training. To address this problem, previous approaches only use reliable pseudo instances, i.e., pseudo instances with high prediction confidence, to retrain the model. Although these strategies effectively reduce the label noise, they are prone to miss the hard examples. In this paper, we propose a new self-training framework for domain adaptation, namely Domain adversarial learning enhanced Self-Training Framework (DaMSTF). Firstly, DaMSTF involves meta-learning to estimate the importance of each pseudo instance, so as to simultaneously reduce the label noise and preserve hard examples. Secondly, we design a meta constructor for constructing the meta validation set, which guarantees the effectiveness of the meta-learning module by improving the quality of the meta validation set. Thirdly, we find that the meta-learning module suffers from the training guidance vanishment and tends to converge to an inferior optimal. To this end, we employ domain adversarial learning as a heuristic neural network initialization method, which can help the meta-learning module converge to a better optimal. Theoretically and experimentally, we demonstrate the effectiveness of the proposed DaMSTF. On the cross-domain sentiment classification task, DaMSTF improves the performance of BERT with an average of nearly 4%. © 2023, CC BY-NC-SA.

关键词： Forecasting

来源：评论

学校读者我要写书评

暂无评论

3D parallelism for Transformers via Integer Programming

3D Parallelism for Transformers via Integer Programming

引用

International Conference on Acoustics, Speech, and Signal Processing (ICASSP)

作者： Hao Zheng Peng Liang Yu Tang Yanqi Shi Linbo Qiao Dongsheng Li National Key Laboratory of Parallel and Distributed Computing National University Of Defense Technology Changsha P.R.China

Transformer models, such as BERT, GPT, and ViT, have been applied to a wide range of areas in recent years, due to their efficacy. In order to improve the training efficiency of Transformer models, different distributed training approaches have been proposed, like Megatron-LM [8]. However, when multi-dimensional parallelism strategies are considered, due to the complexity, existing works can not harmonize the different strategies well enough to obtain a globally optimal solution. In this paper, we propose a parallelism strategy searching algorithm PTIP, which generates operator-level parallelism strategies consisting of three schemes: data parallelism, tensor parallelism, and pipeline parallelism. PTIP abstracts these three parallelism schemes simultaneously into an auxiliary graph, reformulates the searching problem into a mixed-integer programming (MIP) problem, and uses a MIP solver to obtain a high-quality multi-dimensional strategy. Experiments conducted on Transformers demonstrate that PTIP obtains 13.9% − 24.7% performance improvement compared to Megatron-LM [8].

关键词：

来源：评论

学校读者我要写书评

暂无评论

Noncontiguous I/O accesses through MPI-IO 03

Noncontiguous I/O accesses through MPI-IO

引用

IEEE/ACM International Symposium on Cluster computing and the Grid (CCGRID)

作者： A. Ching A. Choudhary K. Coloma Wei-keng Liao R. Ross W. Gropp Center for Parallel and Distributed Computing Northwestern University Evanston IL USA Argonne National Laboratory Argonne IL USA

ISBN: (纸本)9780769519197

I/O performance remains a weakness of parallel computing systems today. While this weakness is partly attributed to rapid advances in other system components, I/O interfaces available to programmers and the I/O methods supported by file systems have traditionally not matched efficiently with the types of I/O operations that scientific applications perform, particularly noncontiguous accesses. The MPI-IO interface allows for rich descriptions of the I/O patterns desired for scientific applications and implementations such as ROMIO have taken advantage of this ability while remaining limited by underlying file system methods. A method of noncontiguous data access, list I/O, was recently implemented in the parallel Virtual File System (PVFS). We implement support for this interface in the ROMIO MPI-IO implementation. Through a suite of noncontiguous I/O tests we compared ROMIO list I/O to current methods of ROMIO noncontiguous access and found that the list I/O interface provides performance benefits in many noncontiguous cases.

关键词： File systems Testing distributed computing Mathematics Computer science Laboratories parallel processing Programming profession Tiles Checkpointing

来源：评论

学校读者我要写书评

暂无评论

An Adaptive Quality of Service Based Scheduling Algorithm for Wide Area Large Scale Problems

An Adaptive Quality of Service Based Scheduling Algorithm fo...

引用

IEEE International Symposium on Network computing and Applications

作者： W. Lozano W. Rivera Parallel and Distributed Computing Laboratory Electrical and Computer Engineering Department University of Puerto Rico MayagÜez Puerto Rico

This paper explores the problem of dynamically scheduling large scale applications over wide area networks and then proposes an adaptive scheduling algorithm to provide quality of service. Our adaptive algorithm takes into account unexpected events and priority fluctuations and gives high priority to jobs with low probability of failure. Experimental results show that the new approach outperforms traditional scheduling approaches in grid computing environments

关键词： Quality of service Scheduling algorithm Large-scale systems Dynamic scheduling Wide area networks Adaptive scheduling Adaptive algorithm Fluctuations Processor scheduling Grid computing

来源：评论

学校读者我要写书评

暂无评论

Assessment of two pressure-velocity coupling strategies for local meshless numerical method

Assessment of two pressure-velocity coupling strategies for ...

引用

作者： Kosec, G. Vertnik, R. Šarler, B. Jožef Stefan Institute Laboratory for Parallel and Distributed Computing Slovenia University of Nova Gorica Laboratory for Multiphase Processes Slovenia Štore Steel D.o.o. Research Slovenia Centre of Excellence for Biosensors Automation and Process Control Laboratory for Advanced Materials Systems Slovenia

The performance of two pressure-velocity coupling strategies, associated with the primitive variable solution of incompressible Newtonian fluid with the meshless Local Radial Basis Function Collocation Method (LRBFCM), is compared with respect to computational efficiency, stability, accuracy and spatial convergence. The LRBFCM is structured with multiquadrics on five noded support domains. The explicit time stepping is used. The Backward-Facing Step problem (BFS) has been selected as a benchmark problem, previously tackled by several numerical methods. The semi-local fractional step method (FSM) and completely local pressure-velocity couplings (LPVC) are compared. The numerical results are validated against previously published data. The results are represented and compared in terms of a convergence plot of the reattachment position. We show that both approaches provide reasonable results;however, LVPC is less computationally complex and less stable in comparison to FSM. © 2012 WIT Press.

关键词： Computational efficiency

来源：评论

学校读者我要写书评

暂无评论

Crowd intelligence in AI 2.0 era

引用

Frontiers of Information Technology & Electronic Engineering 2017年第1期18卷 15-43页

作者： Wei LI Wen-jun WU Huai-min WANG Xue-qi CHENG Hua-jun CHEN Zhi-hua ZHOU Rong DING State Key Laboratory of Software Development Beihang University National Laboratory for Parallel and Distributed Processing College of ComputerNational University of Defense Technology Institute of Computing Technology Chinese Academy of Sciences College of Computer Science and Technology Zhejiang University National Key Laboratory for Novel Software Technology Nanjing University

The Internet based cyber-physical world has profoundly changed the information environment for the development of artificial intelligence(AI), bringing a new wave of AI research and promoting it into the new era of AI 2.0. As one of the most prominent characteristics of research in AI 2.0 era, crowd intelligence has attracted much attention from both industry and research communities. Specifically, crowd intelligence provides a novel problem-solving paradigm through gathering the intelligence of crowds to address challenges. In particular, due to the rapid development of the sharing economy, crowd intelligence not only becomes a new approach to solving scientific challenges, but has also been integrated into all kinds of application scenarios in daily life, e.g., online-tooffline(O2O) application, real-time traffic monitoring, and logistics management. In this paper, we survey existing studies of crowd intelligence. First, we describe the concept of crowd intelligence, and explain its relationship to the existing related concepts, e.g., crowdsourcing and human computation. Then, we introduce four categories of representative crowd intelligence platforms. We summarize three core research problems and the state-of-the-art techniques of crowd intelligence. Finally, we discuss promising future research directions of crowd intelligence.

关键词： Crowd intelligence Artificial intelligence 2.0 Crowdsourcing Human computation

来源：评论

学校读者我要写书评

暂无评论

Pro-Prophet: A Systematic Load Balancing Method for Efficient parallel Training of Large-scale MoE Models

arXiv

引用

arXiv 2024年

作者： Wang, Wei Lai, Zhiquan Li, Shengwei Liu, Weijie Ge, Keshi Shen, Ao Su, Huayou Li, Dongsheng The National Key Laboratory of Parallel and Distributed Computing College of Computer National University of Defense Technology Hunan Changsha China

he size of deep learning models has been increasing to enhance model quality. The linear increase in training computation budget with model size means that training an extremely large-scale model is exceedingly time-consuming. Recently, the Mixture of Expert (MoE) has drawn significant attention as it can scale models to extra-large sizes with a stable computation budget. However, inefficient distributed training of large-scale MoE models hinders their broader application. Specifically, a considerable dynamic load imbalance occurs among devices during training, significantly reducing throughput. Several load-balancing works have been proposed to address the challenge. System-level solutions draw more attention for their hardware affinity and non-disruption of model convergence compared to algorithm-level ones. However, they are troubled by high communication costs and poor communication-computation overlapping. To address these challenges, we propose a systematic load-balancing method, Pro-Prophet, which consists of a planner and a scheduler for efficient parallel training of large-scale MoE models. To adapt to the dynamic load imbalance, we profile training statistics and use them to design Pro-Prophet. For lower communication volume, Pro-Prophet planner determines a series of lightweight load-balancing strategies and efficiently searches for a communication-efficient one for training based on the statistics. For sufficient overlapping of communication and computation, Pro-Prophet scheduler schedules the data-dependent operations based on the statistics and operation features, further improving the training throughput. We conduct extensive experiments in four clusters and five MoE models. The results indicate that Pro-Prophet achieves up to 2.66x speedup compared to two popular MoE frameworks including Deepspeed-MoE and FasterMoE. Furthermore, Pro-Prophet has demonstrated a load-balancing improvement of up to 11.01x compared to a representative load-balancing work,

关键词： Budget control

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：