检索结果-内蒙古大学图书馆

2023 Forum on Specification and Design languages, FDL 2023

ISBN: (纸本)9798350307375

The proceedings contain 14 papers. The topics discussed include: PLiNIO: a user-friendly library of gradient-based methods for complexity-aware DNN optimization;hybrid PTX analysis for GPU accelerated CNN inferencing aiding computer architecture design;neuro-symbolic empowered denoising diffusion probabilistic models for real-time anomaly detection in industry 4.0;satellite payload design for optimized thermal management using a distributed processor system;virtual prototype driven application specific hardware optimization;secure programming platform for edge-based IoT;formal methods-based optimization of dataflow models with translation to synchronous models;VIR2EM: virtualization and remotization for resilient and efficient manufacturing;and enhancing compiler-driven HDL design with automatic waveform analysis.

关键词：

来源：评论

学校读者我要写书评

暂无评论

Sound and Partially-Complete Static Analysis of Data-Races in GPU Programs

引用

proceedings OF THE ACM ON programming languages-PACMPL 2024年第OOPSLA期8卷 2434-2461页

作者： Liew, Dennis Cogumbreiro, Tiago Lange, Julien Univ Massachusetts Boston Boston MA 02125 USA Royal Holloway Univ London Egham England

GPUs are progressively being integrated into modern society, playing a pivotal role in Artificial Intelligence and High-Performance computing. Programmers need a deep understanding of the GPU programming model to avoid subtle data-races in their codes. Static verification that is sound and incomplete can guarantee data-race freedom, but the alarms it raises may be spurious and need to be validated. In this paper, we establish a True Positive Theorem for a static data-race detector for GPU programs, i.e., a result that identifies a class of programs for which our technique only raises true alarms. Our work builds on the formalism of memory access protocols, that models the concurrency operations of CUDA programs. The crux of our approach is an approximation analysis that can correctly identify true alarms, and pinpoint the conditions that make an alarm imprecise. Our approximation analysis detects when the reported locations are reachable (control independence, or CI), and when the reported locations are precise (data independence, or DI), as well identify inexact values in an alarm. In addition to a True Positive result for programs that are CI and DI, we establish the root causes of spurious alarms depending on whether CI or DI are present. We apply our theory to introduce FaialAA, the first sound and partially complete data-race detector. We evaluate FaialAA in three experiments. First, in a comparative study with the state-of-the-art tools, we show that FaialAA confirms more DRF programs than others while emitting 1.9x fewer potential alarms. Importantly, the approximation analysis of FaialAA detects 10 undocumented data-races. Second, in an experiment studying 6 commits of data-race fixes in open source projects OpenMM and Nvidia's MegaTron, FaialAA confirmed the buggy and fixed versions of 5 commits, while others were only able to confirm 2. Third, we show that 59.5% of 2,770 programs are CI and DI, quantifying when the approximation analysis of FaialAA is comp

关键词： data-race detection GPU programming static analysis true positives

来源：评论

学校读者我要写书评

暂无评论

Unsupervised Translation of programming Language - A Survey Paper 4

Unsupervised Translation of Programming Language - A Survey ...

引用

4th International Conference on Advances in computing, Communication Control and Networking, ICAC3N 2022

作者： Kusum, Kusum Ahmed, Abrar Bhuvana, C. Vivek, V. India

ISBN: (纸本)9781665474368

Unsupervised translation of programming languages has seen a recent boom because of the need for legacy systems to port to a much more efficient processing system. This has given rise to various techniques for translation of programming languages. The translation of programming languages is required to ensure that systems can easily communicate and operate with the latest and avant-garde technology. The task of translation of programming languages becomes complex due to the differences in syntaxes that are prevalent between various programming languages. There is also the issue of diverse programming languages and different methods to achieve a common goal. Hence it is imperative to understand the working and challenges of current top notch models for translation of programming languages and to ponder on how their accuracy and efficiency can be improved. In this study, the different models that are employed for translation of programming languages and their effectiveness in their task shall be discussed. © 2022 IEEE.

关键词： Legacy systems

来源：评论

学校读者我要写书评

暂无评论

BeeHive: Sub-second Elasticity forWeb Services with Semi-FaaS Execution 2023

BeeHive: Sub-second Elasticity forWeb Services with Semi-Faa...

引用

28th ACM International Conference on Architectural Support for programming languages and Operating Systems (ASPLOS)

作者： Zhao, Ziming Wu, Mingyu Tang, Jiawei Zang, Binyu Wang, Zhaoguo Chen, Haibo Shanghai Jiao Tong Univ SEIEE Inst Parallel & Distributed Syst Shanghai Peoples R China Shanghai AI Lab Shanghai Peoples R China Minist Educ China Engn Res Ctr Domain Specif Operating Syst Shanghai Peoples R China

ISBN: (纸本)9781450399166

Function-as-a-service (FaaS), an emerging cloud computing paradigm, is expected to provide strong elasticity due to its promise to auto-scale fine-grained functions rapidly. Although appealing for applications with good parallelism and dynamic workload, this paper shows that it is non-trivial to adapt existing monolithic applications (like web services) to FaaS due to their complexity. To bridge the gap between complicated web services and FaaS, this paper proposes a runtime-based Semi-FaaS execution model, which dynamically extracts time-consuming code snippets (closures) from applications and offloads them to FaaS platforms for execution. It further proposes BeeHive, an offloading framework for Semi-FaaS, which relies on the managed runtime to provide a fallback-based execution model and addresses the performance issues in traditional offloading mechanisms for FaaS. Meanwhile, the runtime system of BeeHive selects offloading candidates in a user-transparent way and supports efficient object sharing, memory management, and failure recovery in a distributed environment. The evaluation using various web applications suggests that the Semi-FaaS execution supported by BeeHive can reach sub-second resource provisioning on commercialized FaaS platforms like AWS Lambda, which is up to two orders of magnitude better than other alternative scaling approaches in cloud computing.

关键词： Cloud computing Function-as-a-Service Java Virtual Machine

来源：评论

学校读者我要写书评

暂无评论

Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference 38

Exploiting Inter-Layer Expert Affinity for Accelerating Mixt...

引用

International Parallel and distributed Processing Symposium (IPDPS)

作者： Yao, Jinghan Anthony, Quentin Shafi, Aamir Subramoni, Hari Panda, Dhabaleswar K. (DK) Ohio State Univ Columbus Dept Comp Sci & Engn Columbus OH 43210 USA

ISBN: (纸本)9798350387117;9798350387124

In the realm of large language models (LLMs) like the Generative Pre-trained Transformer (GPT), the Mixture of Experts (MoE) paradigm has emerged as a powerful technique for enhancing model expressiveness and accuracy. However, the deployment of GPT MoE models for parallel inference on distributed systems presents significant challenges, primarily due to the extensive Alltoall communication required for expert routing and aggregation. This communication bottleneck exacerbates the already complex computational landscape, hindering the efficient utilization of high-performance computing resources. In this paper, we propose a lightweight optimization technique called ExFlow, to largely accelerate the inference of these MoE models. We take a new perspective on alleviating the communication overhead by exploiting the inter-layer expert affinity. Unlike previous methods, our solution can be directly applied to pre-trained MoE models without any fine-tuning or accuracy degradation. By proposing a context-coherent expert parallelism on distributed systems, our ExFlow design only uses one Alltoall communication to deliver the same functionality while previous methods all require two Alltoalls. By carefully examining the conditional probability in tokens' routing across multiple layers, we proved that pre-trained GPT MoE models implicitly exhibit a strong inter-layer expert affinity. We then design an efficient integer programming model to precisely capture such features and show that by properly placing the experts on corresponding GPUs, we can reduce up to 67% of tokens' cross-GPU routing latency on various hardware configurations and topologies. Our solution beats the cutting-edge Deepspeed-MoE in GPT MoE models with experts from 8 to 64, with up to 2.2x improvement in inference throughput. To the best of our knowledge, this is the first work in leveraging inter-layer expert affinity to accelerate the inference of GPT MoE models. We further provide a detailed study of how

关键词： Mixture of experts Parallel inference Collective communication Generative models distributed system

来源：评论

学校读者我要写书评

暂无评论

A Multi-Level Platform-Independent GPU API for High-Level programming models 37th

A Multi-Level Platform-Independent GPU API for High-Level Pr...

引用

37th International Supercomputing Conference on High Performance computing (ISC High Performance computing)

作者： Hayashi, Akihiro Paul, Sri Raj Sarkar, Vivek Georgia Inst Technol Atlanta GA 30332 USA Intel Corp Austin TX USA

ISBN: (纸本)9783031232190;9783031232206

While there has been a growing interest in supporting accelerators, especially GPU accelerators, in large-scale systems, the user typically has to work with low-level GPU programming models such as CUDA along with the low-level message passing interface (MPI). We believe higher-level programming models such as Partitioned Global Address Space (PGAS) programming models enable productive parallel programming at both the intra-node and inter-node levels in homogeneous and heterogeneous nodes. However, GPU programming with PGAS languages in practice is still limited since there is still a big performance gap between compiler-generated GPU code and hand-tuned GPU code;hand-optimization of CPU-GPU data transfers is also an important contributor to this performance gap. Thus, it is not rare that the user eventually writes a fully external GPU program that includes the host part -i.e., GPU memory (de)allocation, host-device/device-host data transfer, and the device part - i.e., GPU kernels, and calls it from their primary language, which is not very productive. Our key observation is that the complexity of writing the external GPU program comes not only from writing GPU kernels in the device part, but also from writing the host part. In particular, interfacing objects in the primary language to raw C/C++ pointers is tedious and error-prone, especially because high-level languages usually have a well-defined type system with type inference. In this paper, we introduce the GPUAPI module, which offers multiple abstraction levels of low-level GPU API routines for high-level programming models with a special focus on PGAS languages, which allows the user to choose an appropriate abstraction level depending on their tuning scenarios. The module is also designed to work with multiple standard low-level GPU programming models: CUDA, HIP, DPC++, and SYCL, thereby significantly improving productivity and portability. We use Chapel as the primary example and our preliminary performanc

关键词： GPUs Chapel PGAS languages distributed programming model GPU API library

来源：评论

学校读者我要写书评

暂无评论

FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts models 25

FSMoE: A Flexible and Scalable Training System for Sparse Mi...

引用

30th International Conference on Architectural Support for programming languages and Operating Systems-ASPLOS

作者： Pan, Xinglin Lin, Wenxiang Zhang, Lin Shi, Shaohuai Tang, Zhenheng Wang, Rui Li, Bo Chu, Xiaowen Hong Kong Univ Sci & Technol Guangzhou Guangzhou Peoples R China Harbin Inst Technol Shenzhen Peoples R China Hong Kong Univ Sci & Technol Hong Kong Peoples R China

ISBN: (纸本)9798400706981

Recent large language models (LLMs) have tended to leverage sparsity to reduce computations, employing the sparsely activated mixture-of-experts (MoE) technique. MoE introduces four modules, including token routing, token communication, expert computation, and expert parallelism, that impact model quality and training efficiency. To enable versatile usage of MoE models, we introduce FSMoE, a flexible training system optimizing task scheduling with three novel techniques: 1) Unified abstraction and online profiling of MoE modules for task scheduling across various MoE implementations. 2) Co-scheduling intra-node and inter-node communications with computations to minimize communication overheads. 3) To support near-optimal task scheduling, we design an adaptive gradient partitioning method for gradient aggregation and a schedule to adaptively pipeline communications and computations. We conduct extensive experiments with configured MoE layers and real-world MoE models on two GPU clusters. Experimental results show that 1) our FSMoE supports four popular types of MoE routing functions and is more efficient than existing implementations (with up to a 1.42x speedup), and 2) FSMoE outperforms the state-of-the-art MoE training systems (DeepSpeed-MoE and Tutel) by 1.18x-1.22x on 1458 MoE layers and 1.19x-3.01x on real-world MoE models based on GPT-2 and Mixtral using a popular routing function.

关键词： distributed Deep Learning Large Language Model Mixture-of-Experts Training System Scheduling

来源：评论

学校读者我要写书评

暂无评论

Optimizing Search and Sort Algorithms: Harnessing Parallel programming for Efficient Processing of Large Datasets 2

Optimizing Search and Sort Algorithms: Harnessing Parallel P...

引用

2nd International Conference on Automation, computing and Renewable Systems, ICACRS 2023

作者： Maurya, Ipsit Sinha, Subhratha Suguna, M. School of Computer Science and Engineering Vellore Institute of Technology Chennai India

ISBN: (纸本)9798350340235

This research study investigates the impact of parallel programming techniques on the performance of searching and sorting algorithms. Traditional sequential algorithms have been the foundation of data processing for decades, but the increasing availability of parallel computing resources opens new possibilities for improving efficiency and reducing execution times. By leveraging parallel programming models, such as shared memory and distributed memory systems, this study explores how the parallelization of search and sort algorithms can enhance their speed and scalability. Through comprehensive experimentation and analysis, this study aims to provide valuable insights into the benefits and challenges of parallel programming in the context of searching and sorting algorithms. © 2023 IEEE.

关键词： Parallel programming Performance optimization Scalability analysis Searching algorithms Sorting algorithms

来源：评论

学校读者我要写书评

暂无评论

25th IFIP WG 6.1 International Conference on Coordination models and Language, COORDINATION 2023, held as part of the 18th International Federated Conference on distributed computing Techniques, DisCoTec 2023

25th IFIP WG 6.1 International Conference on Coordination Mo...

引用

25th IFIP WG 6.1 International Conference on Coordination models and Language, COORDINATION 2023, held as part of the 18th International Federated Conference on distributed computing Techniques, DisCoTec 2023

ISBN: (纸本)9783031353604

The proceedings contain 15 papers. The special focus in this conference is on Coordination models and Language. The topics include: JoT: A Jolie Framework for Testing Microservices;rollback Recovery in Session-Based programming;safe Asynchronous Mixed-Choice for Timed Interactions;A Formal MDE Framework for Inter-DSL Collaboration;legal Contracts Amending with;toward Run-time Coordination of Reconfiguration Requests in Cloud computing Systems;macroSwarm: A Field-Based Compositional Framework for Swarm programming;ScaRLib: A Framework for Cooperative Many Agent Deep Reinforcement Learning in Scala;programming distributed Collective Processes for Dynamic Ensembles and Collective Tasks;shelley: A Framework for Model Checking Call Ordering on Hierarchical Systems;stark: A Software Tool for the Analysis of Robustness in the unKnown Environment;RSC to the ReSCu: Automated Verification of Systems of Communicating Automata;reasoning About Choreographic Programs;caos: A Reusable Scala Web Animator of Operational Semantics.

关键词：

来源：评论

学校读者我要写书评

暂无评论

FASOP: Fast yet Accurate Automated Search for Optimal Parallelization of Transformers on Heterogeneous GPU Clusters 24

FASOP: Fast yet Accurate Automated Search for Optimal Parall...

引用

33rd International Symposium on High-Performance Parallel and distributed computing (HPDC)

作者： Hwang, Sunyeol Lee, Eungyeong Oh, Hongseok Yi, Youngmin Univ Seoul Seoul South Korea

ISBN: (纸本)9798400704130

Transformer-based large language models have recently shown remarkable performance, but their significantly large parameters require efficient training, which is commonly realized by utilizing both data- and model-parallel deep learning on a GPU cluster. To minimize the training time, the optimal degrees of data and model parallelisms and the optimal model partitioning should be searched. When heterogeneous GPU clusters are used to utilize as many GPUs as possible, it becomes more challenging. In this work, we propose a framework named FASOP that automatically and rapidly finds the (near-)optimal degrees of parallelisms and model partitioning of Transformer-based models on heterogeneous GPU clusters, with an accurate estimation of pipelining latency and communications. Moreover, it can search for optimal cluster configurations that minimize the training time while satisfying the cost of GPU clusters. The proposed model partitioning algorithm in FASOP is three orders of magnitude faster than Dynamic programming in the state-of-the-art for GPT-2 1.5B on a mixed set of 32 GPUs with A100 and A10, leading to a few seconds instead of several hours. And, FASOP shows only 8.7% mean absolute error in training time estimation for GPT-2 1.5B. With a fast yet accurate search, FASOP achieved up to 1.37x speedup compared to Megatron-LM.

关键词： Heterogeneous cluster distributed deep learning Parallelization strategy Large language model

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：