检索结果-内蒙古大学图书馆

EfficientMoE: Optimizing Mixture-of-Experts Model Training With Adaptive Load Balance

IEEE TRANSACTIONS ON parallel AND distributed systems 2025年第4期36卷 677-688页

作者： Zeng, Yan Huang, Chengchuang Mei, Yipeng Zhang, Lifu Su, Teng Ye, Wei Shi, Wenqi Wang, Shengnan Hangzhou Dianzi Univ Hangzhou 310018 Peoples R China Huawei Technol Co Ltd Distributed Data Lab Shenzhen 518000 Peoples R China

Mixture-of-Experts (MoE) efficiently trains large models by using sparse activation to lower costs, selecting a few experts based on data characteristics. However, it faces challenges such as All-to-All communication overhead and load imbalance, with most optimizations targeting dynamic graphs rather than the more efficient static graphs. This study identifies two key challenges in training MoE on static graphs: 1) excessive All-to-All communication (up to 75% of iteration time) and load imbalance (70% of tokens handled by two experts) between experts due to the sparse structure of the MoE model and the token distribution;and 2) inefficient zero-padding for static shapes, leading to unnecessary computational overhead(wasting approximately 50% of resources). Thus, EfficientMoE, a scheduling method based on expert load and data characteristics, is introduced. EfficientMoE first designs a sampler to collect real-time information about token distribution, expert load, etc. It constructs a load prediction model to evaluate expert load. Subsequently, EfficientMoE proposes a dynamic schedule strategy for experts with evaluated expert load, reducing All-to-All communication and addressing load-balancing issues. Additionally, an expert capacity model is proposed to set different capacities for replicas of hot experts before static graph compilation, minimizing computation and storage overhead caused by significant padding. This study implements EfficientMoE in MindSpore and uses 32 Ascend AI accelerators to train an MoE model with 21 billion parameters and evaluate its validity. EfficientMoE demonstrated an improvement of 30% in model training time, approximately 12% reduction in communication time, and saved 35% computational resources across different clusters, compared with Switch transformers, and the Fastermoe method for static graphs.

关键词： Computational modeling Load modeling Training parallel processing AI accelerators Data models Computational efficiency Tensors Costs Transformers distributed training parallelism load balance

来源：评论

学校读者我要写书评

暂无评论

HiPART: High-Performance Technology for Advanced real-time systems 16

HiPART: High-Performance Technology for Advanced Real-Time S...

引用

16th workshop on parallel Programming and Run-time Management Techniques for Many-Core Architectures and 14th workshop on Design Tools and Architectures for Multicore Embedded Computing Platforms, PARMA-DITAM 2025

作者： Royuela, Sara Munera, Adrian Yu, Chenle Pinot, Josep Barcelona Supercomputing Center Spain

ISBN: (纸本)9783959773638

Cyber-physical systems (CPS) attempt to meet real-time and safety requirements by using hypervisors that provide isolation via virtualisation and real-time Operating systems that manage the concurrency of system tasks. However, the operating system's (OS) decisions may hinder the efficiency of tasks because it needs more awareness of their specific intricacies. Hence, one critical limitation to efficiently developing CPSs is the lack of tailored parallel programming models that can harness the capabilities of advanced heterogeneous architectures while meeting the requirements integral to CPSs, such as real-time behaviour and safety requirements. While conventional HPC languages, like OpenMP and CUDA, cannot accommodate critical non-functional properties, safety languages, like Rust and Ada, are limited in their capabilities to exploit complex systems efficiently. On top of that, accessibility to the programming task is essential to making the system usable to different domain experts. HiPART tackles these challenges by developing a comprehensive framework holistically addressing efficiency, interoperability, reliability, and sustainability. The HiPART framework, based on OpenMP, provides tailored support for (1) real-time behaviour and safety requirements and (2) the efficient exploitation of advanced parallel and heterogeneous processor architectures. This support is exposed to users through extensions to the OpenMP specification and its implementation in the LLVM framework, including the compiler and the OpenMP runtime library. With this framework, HiPART will contribute to realising more capable and reliable autonomous systems across various domains, from autonomous mobility to space exploration. © Sara Royuela, Adrian Munera, Chenle Yu, and Josep Pinot.

关键词： parallel architectures

来源：评论

学校读者我要写书评

暂无评论

16th workshop on parallel Programming and Run-time Management Techniques for Many-Core Architectures and 14th workshop on Design Tools and Architectures for Multicore Embedded Computing Platforms, PARMA-DITAM 2025

16th Workshop on Parallel Programming and Run-Time Managemen...

引用

ISBN: (纸本)9783959773638

The proceedings contain 6 papers. The topics discussed include: analysis of GPU memory allocation characteristics;custom floating-point computations for the optimization of ODE solvers on FPGA;system-level timing performance estimation based on a unifying HW/SW performance metric;towards studying the effect of compiler optimizations and software randomization on GPU reliability;evaluation of the parallel features of rust for space systems;and HiPART: high-performance technology for advanced real-time systems.

关键词：

来源：评论

学校读者我要写书评

暂无评论

ChunkFunc: Dynamic SLO-Aware Configuration of Serverless Functions

引用

IEEE TRANSACTIONS ON parallel AND distributed systems 2025年第6期36卷 1237-1252页

作者： Pusztai, Thomas Nastic, Stefan TU Wien Distributed Syst Grp A-1040 Vienna Austria

Serverless computing promises to be a cost effective form of on demand computing. To fully utilize its cost saving potential, workflows must be configured with the appropriate amount of resources to meet their response time Service Level Objective (SLO), while keeping costs at a minimum. Since determining and updating these configuration models manually is a nontrivial and error prone task, researchers have developed solutions for automatically finding configurations that meet the aforementioned requirements. However, our initial experiments show that even when following best practices and using state-of-the-art configuration tools, resources may still be considerably over- or underprovisioned, depending on the size of functions' input payload. In this paper we present ChunkFunc, an SLO- and input data-aware framework for tuning serverless workflows. Our main contributions include: i) an SLO- and input size-aware function performance model for optimized configurations in serverless workflows, ii) ChunkFunc Profiler, an auto-tuned, Bayesian Optimization-guided profiling mechanism for profiling serverless functions with typical input data sizes to build a performance model, and iii) ChunkFunc Workflow Optimizer, which uses these models to determine an input size dependent configuration for each serverless function in a workflow to meet the SLO, while keeping costs to a minimum. We evaluate ChunkFunc on real-life serverless workflows and compare it to two state-of-the-art solutions, showing that it increases SLO adherence by a factor of 1.04 to 2.78, depending on the workflow, and reduces costs by up to 61% .

关键词： Costs Data models time factors Measurement Tuning Optimization Monitoring Memory management Buildings Bayes methods Serverless workflows serverless functions configuration tuning SLOs profiling

来源：评论

学校读者我要写书评

暂无评论

GreenFlow: A Carbon-Efficient Scheduler for Deep Learning Workloads

引用

IEEE TRANSACTIONS ON parallel AND distributed systems 2025年第2期36卷 168-184页

作者： Gu, Diandian Zhao, Yihao Sun, Peng Jin, Xin Liu, Xuanzhe Peking Univ Sch Comp Sci Beijing 100871 Peoples R China Peking Univ Minist Educ Key Lab High Confidence Software Technol Beijing 100871 Peoples R China Peking Univ Ctr Data Space Technol & Syst Beijing 100871 Peoples R China Shanghai AI Lab Shanghai 200232 Peoples R China

Deep learning (DL) has become a key component of modern software. Training DL models leads to huge carbon emissions. In data centers, it is important to reduce carbon emissions while completing DL training jobs early. In this article, we propose GreenFlow, a GPU cluster scheduler that reduces the average Job Completion time (JCT) under a carbon emission budget. We first present performance models for DL training jobs to predict the throughput and energy consumption performance under different configurations. Based on the performance models and the carbon intensity of the grid, GreenFlow dynamically allocates GPUs, and adjusts the GPU-level and job-level configurations of DL training jobs. GreenFlow applies network packing and buddy allocation to job placement, thus avoiding extra carbon incurred by resource fragmentations. Evaluations on a real testbed show that when emitting the same amount of carbon, GreenFlow can improve the average JCT by up to 2.15x, compared to competitive baselines.

关键词： GPU cluster distributed deep learning carbon efficiency carbon efficiency cluster scheduling cluster scheduling cluster scheduling

来源：评论

学校读者我要写书评

暂无评论

distributed Online Optimization Of Consensus-Based Adaptive Gradients Over time-Varying Networks

IEEE Transactions on Industrial Cyber-Physical Systems

引用

IEEE Transactions on Industrial Cyber-Physical systems 2025年 3卷 190-197页

作者： Sheng, Qingyang Shi, Xiasheng Su, Yanxu Anhui University School of Artificial Intelligence Hefei230601 China Ministry of Education Engineering Research Center of Autonomous Unmanned System Technology Hefei230601 China

The application of distributed optimization to ICPSs has advantages and challenges. Recently, a consensus-based distributed adaptive moment estimation method, referred to as DAdam (distributed Adam), has been proposed as a variant of Adam specifically tailored for distributed and parallel computing environments. DAdam integrates the benefits of adaptive learning rate and moment estimation, but its application scenarios are limited. The assumption of static networks in existing literatures is conservative in real environments. To overcome this limitation, we propose DAdam-TV which can solve the optimization problems in time-varying networks. After rigorous analysis, we exam the convergence of proposed algorithm. For convex and non-convex problems, we bound the dynamic regret and local regret, respectively. Numerical simulations show that DAdam-TV has better performance in solving optimization problems in dynamic networks. DAdam-TV breaks through the limitation of static application scenarios, which makes the algorithm more general and effective in practical applications such as ICPSs. © 2023 IEEE.

关键词： Optimization algorithms

来源：评论

学校读者我要写书评

暂无评论

Efficient parallel Sparse Tensor Contraction

引用

IEEE Transactions on parallel and distributed systems 2025年第6期36卷 1206-1219页

作者： Singh, Somesh Uçar, Bora LyonF-69364 France LyonF-69364 France AtlantaGA30332 United States

We investigate the performance of algorithms for sparse tensor-sparse tensor multiplication (SpGETT). This operation, also called sparse tensor contraction, is a higher order analogue of the sparse matrix-sparse matrix multiplication (SpGEMM) operation. Therefore, SpGETT can be performed by first converting the input tensors into matrices, then invoking high performance variants of SpGEMM, and finally reconverting the resultant matrix into a tensor. Alternatively, one can carry out the scalar operations underlying SpGETT in the realm of tensors without matrix formulation. We discuss the building blocks in both approaches and formulate a hashing-based method to avoid costly search or redirection operations. We present performance results with the current state-of-the-art SpGEMM-based approaches, existing SpGETT approaches, and a carefully implemented SpGETT approach with a new fine-tuned hashing method, proposed in this article. We evaluate the methods on real world tensors by contracting a tensor with itself along varying dimensions. Our proposed hashing-based method for SpGETT consistently outperforms the state-of-the-art method, achieving a 25% reduction in sequential execution time on average and a 21% reduction in parallel execution time on average across a variety of input instances. © 1990-2012 IEEE.

关键词： Tensors

来源：评论

学校读者我要写书评

暂无评论

Enhancing performance of machine learning tasks on edge-cloud infrastructures: A cross-domain Internet of Things based framework

引用

FUTURE GENERATION COMPUTER systems-THE INTERNATIONAL JOURNAL OF ESCIENCE 2025年 166卷

作者： Almurshed, Osama Kaushal, Ashish Meshoul, Souham Muftah, Asmail Almoghamis, Osama Petri, Ioan Auluck, Nitin Rana, Omer Cardiff Univ Cardiff Wales Indian Inst Technol Ropar Rupnagar India Princess Nourah bint Abdulrahman Univ Riyadh Saudi Arabia Azzaytuna Univ Tarhuna Libya King Saud Univ Riyadh Saudi Arabia Prince Sattam Bin Abdulaziz Univ Al Kharj Saudi Arabia

The Internet of Things (IoT) and Edge-Cloud Computing have been trending technologies over the past few years. In this work, we introduce the Enhanced Optimized-Greedy Nominator Heuristic (EO-GNH), a framework designed to optimize machine learning (ML) and artificial intelligence (AI) application placement in edge environments, aiming to improve Quality of Service (QoS). Developed specifically for sectors such as smart agriculture, industry, and healthcare, EO-GNH integrates asynchronous MapReduce and parallel meta- heuristics to effectively manage AI applications, focusing on execution performance, resource utilization, and infrastructure resilience. The framework carefully addresses the distribution challenges of AI applications, especially Service Function Chains (SFCs), in edge-cloud infrastructures. It contains Data Flow Management, which covers aspects of data storage and data privacy, and also considers factors like regional adaptations, mobile access, and AI model refinement. EO-GNH ensures high availability for forecasting, prediction, and training AI models, operating efficiently within a geo-distributed infrastructure. The proposed strategies within EO-GNH emphasize concurrent multi-node execution, enhancing AI application placement by improving execution time, dependability, and cost-effectiveness. The efficiency of EO-GNH is demonstrated through its impact on QoS in real-time resource management across three application domains, highlighting its adaptability and potential in diverse cross-domain IoT-based environments.

关键词： Artificial intelligence distributed systems Edge computing Internet of things Machine learning

来源：评论

学校读者我要写书评

暂无评论

EFFICIENT PARTITION-OF-UNITY RADIAL-BASIS-FUNCTION INTERPOLATION FOR COUPLED PROBLEMS

引用

SIAM Journal on Scientific Computing 2025年第2期47卷 B558-B582页

作者： Schneider, David Uekermann, Benjamin Institute for Parallel and Distributed Systems University of Stuttgart Stuttgart70569 Germany

Mapping of data between nonmatching meshes is a key ingredient of multiphysics simulations. Black-box data mapping, which only operates on clouds of mesh vertices without connectivity, enables modular software environments. In this paper, we develop such a black-box approach that is capable of handling very large data sets on parallel systems. More precisely, we implement partition-of-unity radial-basis-function interpolation into the coupling library preCICE. The method tackles the data mapping problem by decomposing it into smaller, independent subproblems, which makes it well-suited for parallel computing. To this end, we develop a tailor-made clustering algorithm and study numerical details to ensure robustness and accuracy. We, moreover, deduce user-friendly mapping parameters for which we determine robust default values. Tests on real-world geometries show that the method is scalable and orders of magnitude more efficient than previous data mapping in preCICE. Consequently, the implementation greatly extends the applicability of preCICE, benefiting the library's large user community. © 2025 Society for Industrial and Applied Mathematics.

关键词： Mesh generation

来源：评论

学校读者我要写书评

暂无评论

Driver Behavior Recognition in Complex Driving Scenarios via Multicollinear Fusion Network With Optimizations

引用

IEEE SENSORS JOURNAL 2025年第8期25卷 13300-13315页

作者： Deng, Wenhao Zhao, Chihang Shen, Peng Zhang, Ziyi Southeast Univ Sch Transportat Nanjing 210096 Peoples R China CCCC First Highway Consultants Co Ltd Xian 710068 Peoples R China

Distracted driving is a primary contributor to traffic accidents, and driver behavior recognition (DBR) research holds immense potential in reducing the frequency of accidents caused by such distractions. However, existing DBR networks trained on single datasets exhibit severe limitations, and they will overfit to the dataset's specific scenarios and vehicle environments, thereby compromising their generalization abilities in complex scenarios. Therefore, we integrate a diverse comprehensive dataset and propose a generalizable multicollinear fusion network termed MFN-DBR for highly accurate recognition in complex scenarios. The technical process includes: First, the State Farm, ASU, AUCV2, and 100-Driver datasets are integrated to form a comprehensive dataset featuring diverse drivers and scenarios. Then, multidimensional collaborative attention (MCA) is introduced to enhance the critical features weights, and orthogonal low-rank embedding (OLE) loss is utilized to optimize the feature space distributions of ResNet-DBR, ResNeXt-DBR, and ConvNeXt-DBR subnetworks. Furthermore, a linear parallel fusion strategy is adopted to fuse their feature representations, and thus construct the MFN. Specifically, the MFN-DBR incorporates the specific structural attributes and scenario adaptability of subnetworks, enabling it to generate more integrated and richer features and defining more discriminative classification boundaries. The experimental outcomes demonstrate that MFN-DBR achieves 91.11% accuracy, 91.38% F1-score, and 19.6 ms/frames on the comprehensive dataset, and even 91.01% accuracy in special scenarios with extremely poor lighting conditions, outperforming other state-of-the-art networks. Additionally, we provide real-time verifications in two real driving video streams, demonstrating that MFN-DBR is capable of generalizing in real, complex, and variable driving scenarios with high performance, effectiveness, and robustness.

关键词： Feature extraction Driver behavior distributed Bragg reflectors Image recognition Vehicles Data mining Sensors Robustness Accidents Lighting Complex driving scenarios driver behavior recognition (DBR) feature fusion intelligent transportation systems real-time verifications

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：