检索结果-内蒙古大学图书馆

30th Symposium on Principles and Practice of Parallel programming

作者： Lin, Junfeng Liu, Ziming You, Yang Wang, Jun Zhang, Weihao Zhao, Rong Tsinghua Univ Beijing Peoples R China Natl Univ Singapore Singapore Singapore CETHIK Grp Co Ltd Hangzhou Peoples R China Lynxi Technol Co Ltd Beijing Peoples R China

ISBN: (纸本)9798400714436

Training large language models (LLMs) has become increasingly expensive due to the rapid expansion in model size. Pipeline parallelism is a widely used distributed training technique. However, as LLMs with larger context become prevalent and memory optimization techniques advance, traditional PP methods encounter greater communication challenges due to the increased size of activations and gradients of activations. To address this issue, we introduce weight-pipeline parallelism (WeiPipe) that transitions from an activation-passing pipeline to a weight-passing pipeline. WeiPipe reduces communication costs and achieves a more balanced utilization by transmitting only weights and their gradients between workers in a pipeline manner. WeiPipe does not rely on collective communication primitives, thus ensuring scalability. We present four variations of WeiPipe parallelism, including WeiPipe-Interleave, which emphasizes communication efficiency, and WeiPipe-zero-bubble, discussing the potential for minimal bubble ratios. Our implementation of WeiPipe-Interleave, performed on up to 32 GPUs and tested in various model configurations, including large-context LLM training, demonstrates a significant improvement in throughput compared to state-of-the-art pipeline parallelism and fully sharded data parallelism with different underlying infrastructures, including NVLink connections within cluster with Ethernet among cluster, and PCIe within cluster and Ethernet among cluster. Additionally, WeiPipe also shows greater scalability in communicationconstrained scenarios compared to state-of-art strategies.

关键词： distributed deep learning pipeline parallelism large-scale training long-context training

来源：评论

学校读者我要写书评

暂无评论

T3: Transparent Tracking & Triggering for Fine-grained Overlap of Compute & Collectives 24

T3: Transparent Tracking & Triggering for Fine-grained Overl...

引用

29th ACM International Conference on Architectural Support for programming languages and Operating Systems (ASPLOS)

作者： Pati, Suchita Aga, Shaizeen Islam, Mahzabeen Jayasena, Nuwan Sinclair, Matthew D. Univ Wisconsin Madison Adv Micro Devices Inc Madison WI USA Adv Micro Devices Inc Santa Clara CA USA

ISBN: (纸本)9798400703850

Large Language models increasingly rely on distributed techniques for their training and inference. These techniques require communication across devices which can reduce scaling efficiency as the number of devices increases. While some distributed techniques can overlap, and thus, hide this communication with independent computations, techniques such as Tensor Parallelism (TP) inherently serialize communication with model execution. One approach to hide this serialized communication is to interleave it with the producer operation (of the communicated data) in a fine-grained manner. However, this fine-grained interleaving of communication and computation in software can be difficult. Furthermore, as with any concurrent execution, it requires compute and memory resources to be shared between computation and communication, causing resource contention that reduces overlapping efficacy. To overcome these challenges, we propose T3 which applies hardware-software co-design to transparently overlap serialized communication while minimizing resource contention with compute. T3 transparently fuses producer operations with the subsequent communication via a simple configuration of the producer's output address space and requires minor software changes. At the hardware level, T3 adds a lightweight track and trigger mechanism to orchestrate the producer's compute, and communication. It further uses compute-enhanced memories for communication's attendant compute. As a result, T3 reduces resource contention, and efficiently overlaps serialized communication with computation. For important Transformer models like T-NLG, T3 speeds up communication-heavy sublayers by 30% geomean (max 47%) and reduces data movement by 22% geomean (max 36%). Furthermore, T3's benefits persist as models scale: geomean 29% for sublayers in similar to 500-billion parameter models, PALM and MT-NLG.

关键词： distributed Machine Learning Collective Communication Transformers GPUs Fusion Fine-grained Overlap Near-memory computing

来源：评论

学校读者我要写书评

暂无评论

Adaptive Parallel Training for Graph Neural Networks 25

Adaptive Parallel Training for Graph Neural Networks

引用

30th Symposium on Principles and Practice of Parallel programming

作者： Ma, Kaihao Liu, Renjie Yan, Xiao Cai, Zhenkun Song, Xiang Wang, Minjie Li, Yichao Cheng, James Chinese Univ Hong Kong Hong Kong Peoples R China Southern Univ Sci & Technol Shenzhen Peoples R China Ctr Perceptual & Interact Intelligence Hong Kong Peoples R China Amazon Seattle WA USA AWS Shanghai AI Lab Shanghai Peoples R China

ISBN: (纸本)9798400714436

There are several strategies to parallelize graph neural network (GNN) training over multiple GPUs. We observe that there is no consistent winner (i.e., with the shortest running time), and the optimal strategy depends on the graph dataset, GNN model, training algorithm, and hardware configurations. As such, we design the APT system to automatically select efficient parallelization strategies for GNN training tasks. To this end, we analyze the trade-offs of the strategies and design simple yet effective cost models to compare their execution time and facilitate strategy selection. Moreover, we also propose a general abstraction of the strategies, which allows to implement a unified execution engine that can be configured to run different strategies. Our experiments show that APT usually chooses the optimal or a close to optimal strategy, and the training time can be reduced by over 2x compared with always using a single strategy. APT is open-source at https://***/kaihaoma/APT.

关键词： Graph Neural Networks distributed and Parallel Training Network Communication

来源：评论

学校读者我要写书评

暂无评论

Breaking the programming Language Barrier: Multilingual Prompting to Empower Non-Native English Learners 25

Breaking the Programming Language Barrier: Multilingual Prom...

引用

27th Australasian computing Education Conference-ACE-Annual

作者： Prather, James Reeves, Brent N. Denny, Paul Leinonen, Juho MacNeil, Stephen Luxton-Reilly, Andrew Orvalho, Joao Alipour, Amin Alfageeh, Ali Amarouche, Thezyrie Kimmel, Bailey Wright, Jared Blake, Musa Barbre, Gweneth Abilene Christian Univ Abilene TX 79699 USA Univ Auckland Auckland New Zealand Aalto Univ Espoo Finland Temple Univ Philadelphia PA USA Polytech Univ Coimbra Coimbra Portugal Univ Houston Houston TX USA Univ Toronto Scarborough Toronto ON Canada

ISBN: (纸本)9798400714252

Non-native English speakers (NNES) face multiple barriers to learning programming. These barriers can be obvious, such as the fact that programming language syntax and instruction are often in English, or more subtle, such as being afraid to ask for help in a classroom full of native English speakers. However, these barriers are frustrating because many NNES students know more about programming than they can articulate in English. Advances in generative AI (GenAI) have the potential to break down these barriers because state of the art models can support interactions in multiple languages. Moreover, recent work has shown that GenAI can be highly accurate at code generation and explanation. In this paper, we provide the first exploration of NNES students prompting in their native languages (Arabic, Chinese, and Portuguese) to generate code to solve programming problems. Our results show that students are able to successfully use their native language to solve programming problems, but not without some difficulty specifying programming terminology and concepts. We discuss the challenges they faced, the implications for practice in the short term, and how this might transform computing education globally in the long term.

关键词： AI Artificial Intelligence Automatic Code Generation Codex Copilot CS1 GenAI GitHub GPT GPT-4 ChatGPT HCI Introductory programming Large Language models LLM Non-Native English Speakers Novice programming OpenAI Prompt Problems

来源：评论

学校读者我要写书评

暂无评论

PipeLLM: Fast and Confidential Large Language Model Services with Speculative Pipelined Encryption 25

PipeLLM: Fast and Confidential Large Language Model Services...

引用

30th International Conference on Architectural Support for programming languages and Operating Systems-ASPLOS

作者： Tan, Yifan Tan, Cheng Mi, Zeyu Chen, Haibo Shanghai Jiao Tong Univ SEIEE Inst Parallel & Distributed Syst Shanghai Peoples R China Northeastern Univ Boston MA USA

ISBN: (纸本)9798400706981

Confidential computing on GPUs, like NVIDIA H100, mitigates the security risks of outsourced Large Language models (LLMs) by implementing strong isolation and data encryption. Nonetheless, this encryption incurs a significant performance overhead, reaching up to 52.8% and 88.2% throughput drop when serving OPT-30B and OPT-66B, respectively. To address this challenge, we introduce PipeLLM, a user-transparent runtime system. PipeLLM removes the overhead by overlapping the encryption and GPU computation through pipelining-an idea inspired by the CPU instruction pipelining-thereby effectively concealing the latency increase caused by encryption. The primary technical challenge is that, unlike CPUs, the encryption module lacks prior knowledge of the specific data needing encryption until it is requested by the GPUs. To this end, we propose speculative pipelined encryption to predict the data requiring encryption by analyzing the serving patterns of LLMs. Further, we have developed an efficient, low-cost pipeline relinquishing approach for instances of incorrect predictions. Our experiments show that compared with vanilla systems without confidential computing (e.g., vLLM, PEFT, and FlexGen), PipeLLM incurs modest overhead (<19.6% in throughput) across various LLM sizes, from 13B to 175B. PipeLLM's source code is available at https://***/SJTU-IPADS/PipeLLM.

关键词： Nvidia Confidential computing Large Language Model Confidential Virtual Machine

来源：评论

学校读者我要写书评

暂无评论

Invited Paper: DeepSLOs for the computing Continuum

Invited Paper: DeepSLOs for the Computing Continuum

引用

2024 Workshop on Advanced Tools, programming languages, and PLatforms for Implementing and Evaluating algorithms for distributed Systems, ApPLIED 2024, co-located with PODC 2024

作者： Casamayor Pujol, Victor Sedlak, Boris Xu, Yanwei Donta, Praveen Kumar Dustdar, Schahram Distributed Systems Group TU Wien Vienna1040 Austria

ISBN: (纸本)9798400706707

The advent of the computing continuum, i.e., the blending of all existing computational tiers, calls for novel techniques and methods that consider its complex dynamics. This work presents the DeepSLO as a novel design paradigm to define and structure Service Level Objectives (SLOs) for distributed computing continuum systems. Hence, when multiple stakeholders are involved, the DeepSLO allows them to plan the overarching behaviors of the system. Further, the techniques employed (Bayesian networks, Markov blanket, Active inference) provide autonomy and decentralization to each SLO while the DeepSLO hierarchy remains to account for objectives dependencies. Finally, DeepSLOs are represented graphically, as well as individual SLOs enabling a human interpretation of the system performance. Copyright © 2024 held by the owner/author(s). Publication rights licensed to ACM.

关键词： Bayesian networks

来源：评论

学校读者我要写书评

暂无评论

programming Tools for High-Performance Data Analysis 24

Programming Tools for High-Performance Data Analysis

引用

33rd International Symposium on High-Performance Parallel and distributed computing (HPDC)

作者： Talia, Domenico Trunfio, Paolo Univ Calabria DIMES Arcavacata Di Rende CS Italy

ISBN: (纸本)9798400704130

In this paper we discuss and compare some of the most popular big data programming frameworks for high performance computing (HPC) systems, such as Hadoop, Spark, Storm, Airflow, MPI, Hive, Pig and UPC++. We highlight here their support to different classes of applications, such as batch, streaming, graph-based, and query-based applications. The main features of such frameworks according to their programming model, type of parallelism, level of abstraction, verbosity, and main classes of applications are summarized. We also discuss the main factors that can influence the choice of the most appropriate framework to process and analyze big data. These include the characteristics of input data, the class of application and the infrastructure requirements.

关键词： Big data programming models MapReduce Workflows Message passing SQL-like PGAS

来源：评论

学校读者我要写书评

暂无评论

TNIC: A Trusted NIC Architecture A hardware-network substrate for building high-performance trustworthy distributed systems 25

TNIC: A Trusted NIC Architecture A hardware-network substrat...

引用

30th International Conference on Architectural Support for programming languages and Operating Systems-ASPLOS

作者： Giantsidi, Dimitra Pritzi, Julian Gust, Felix Katsarakis, Antonios Koshiba, Atsushi Bhatotia, Pramod Univ Edinburgh Edinburgh Midlothian Scotland Tech Univ Munich Munich Germany Huawei Res Edinburgh Midlothian Scotland

ISBN: (纸本)9798400710797

We introduce TNIC, a trusted NIC architecture for building trustworthy distributed systems deployed in heterogeneous, untrusted (Byzantine) cloud environments. TNIC builds a minimal, formally verified, silicon root-of-trust at the network interface level. We strive for three primary design goals: (1) a host CPU-agnostic unified security architecture by providing trustworthy network-level isolation;(2) a minimalistic and verifiable TCB based on a silicon root-of-trust by providing two core properties of transferable authentication and non-equivocation;and (3) a hardware-accelerated trustworthy network stack leveraging SmartNICs. Based on the TNIC architecture and associated network stack, we present a generic set of programming APIs and a recipe for building high-performance, trustworthy, distributed systems for Byzantine settings. We formally verify the safety and security properties of our TNIC while demonstrating its use by building four trustworthy distributed systems. Our evaluation of TNIC shows up to 6x performance improvement compared to CPU-centric TEE systems.

关键词： trusted computing hardware-software co-design

来源：评论

学校读者我要写书评

暂无评论

COMPSO: Optimizing Gradient Compression for distributed Training with Second-Order Optimizers 25

COMPSO: Optimizing Gradient Compression for Distributed Trai...

引用

30th Symposium on Principles and Practice of Parallel programming

作者： Sun, Baixi Liu, Weijin Pauloski, J. Gregory Tian, Jiannan Jia, Jinda Wang, Daoce Zhang, Boyuan Zheng, Mingkai Di, Sheng Jin, Sian Zhang, Zhao Yu, Xiaodong Iskra, Kamil A. Beckman, Pete Tan, Guangming Tao, Dingwen Indiana Univ Bloomington IN USA Stevens Inst Technol Hoboken NJ USA Univ Chicago Chicago IL USA Rutgers State Univ New Brunswick NJ USA Argonne Natl Lab Lemont IL USA Temple Univ Philadelphia PA USA Northwestern Univ Evanston IL USA Univ Chinese Acad Sci Beijing Peoples R China

ISBN: (纸本)9798400714436

Second-order optimization methods have been developed to enhance convergence and generalization in deep neural network (DNN) training compared to first-order methods like Stochastic Gradient Descent (SGD). However, these methods face challenges in distributed settings due to high communication overhead. Gradient compression, a technique commonly used to accelerate communication for first-order approaches, often results in low communication reduction ratios, decreased model accuracy, and/or high compression overhead when applied to second-order methods. To address these limitations, we introduce a novel gradient compression method for second-order optimizers called COMPSO. This method effectively reduces communication costs while preserving the advantages of second-order optimization. COMPSO employs stochastic rounding to maintain accuracy and filters out minor gradients to improve compression ratios. Additionally, we develop GPU optimizations to minimize compression overhead and performance modeling to ensure end-to-end performance gains across various systems. Evaluation of COMPSO on different DNN models shows that it achieves a compression ratio of 22.1x, reduces communication time by 14.2x, and improves overall performance by 1.9x, all without any drop in model accuracy.

关键词： Deep learning distributed training second-order optimization K-FAC data compression

来源：评论

学校读者我要写书评

暂无评论

AdoB: Bridging Benign and Byzantine Consensus with Atomic distributed Objects

引用

proceedings OF THE ACM ON programming languages-PACMPL 2024年第OOPSLA期8卷 419-448页

作者： Honore, Wolf Qiu, Longfei Kim, Yoonseung Shin, Ji-Yong Kim, Jieung Shao, Zhong Yale Univ New Haven CT 06520 USA Northeastern Univ Boston MA USA Inha Univ Incheon South Korea

Achieving consensus is a challenging and ubiquitous problem in distributed systems that is only made harder by the introduction of malicious byzantine servers. While significant effort has been devoted to the benign and byzantine failure models individually, no prior work has considered the mechanized verification of both in a generic way. We claim this is due to the lack of an appropriate abstraction that is capable of representing both benign and byzantine consensus without either losing too much detail or becoming impractically complex. We build on recent work on the atomic distributed object model to fill this void with a novel abstraction called AdoB. In addition to revealing important insights into the essence of consensus, this abstraction has practical benefits for easing distributed system verification. As a case study, we proved safety and liveness properties for AdoB in Coq, which are the first such mechanized proofs to handle benign and byzantine consensus in a unified manner. We also demonstrate that AdoB faithfully models real consensus protocols by proving it is refined by standard network-level specifications of Fast Paxos and a variant of Jolteon.

关键词： distributed systems consensus protocols byzantine liveness formal verification refinement proof assistants

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：