检索结果-内蒙古大学图书馆

data-intensive workflow scheduling strategy based on deep reinforcement learning in multi-clouds

JOURNAL OF CLOUD COMPUTING-ADVANCES SYSTEMS AND APPLICATIONS 2023年第1期12卷 125-125页

作者： Zhang, Shuo Zhao, Zhuofeng Liu, Chen Qin, Shenghui North China Univ Technol Sch Informat Beijing 100144 Peoples R China Beijing Key Lab Large Scale Stream Data Integrat & Beijing 100144 Peoples R China

With the increase development of Internet of Things devices, the data-intensive workflow has emerged as a new kinds of representation for IoT applications. Because most IoT systems are structured in multi-clouds environment and the data-intensive workflow has the characteristics of scattered data sources and distributed execution requirements at the cloud center and edge clouds, it brings many challenges to the scheduling of such workflow, such as data flow control management, data transmission scheduling, etc. Aiming at the execution constraints of business and technology and data transmission optimization of data-intensive workflow, a data-intensive workflow scheduling method based on deep reinforcement learning in multi-clouds is proposed. First, the execution constraints, edge node load and data transmission volume of IoT data workflow are modeled;then the data-intensive workflow is segmented with the consideration of business constraints and the first optimization goal of data transmission;besides, taking the workflow execution time and average load balancing as the secondary optimization goal, the improved DQN algorithm is used to schedule the workflow. Based on the DQN algorithm, the model reward function and action selection are redesigned and improved. The simulation results based on workflowSim show that, compared with MOPSO, NSGA-II, GTBGA and DQN, the algorithm proposed in this paper can effectively reduce the execution time of IoT data workflow under the condition of ensuring the execution constraints and load balancing of multi-clouds.

关键词： data-intensive workflow Deep-Q-network Multi-objective optimization intensive learning

来源：评论

学校读者我要写书评

暂无评论

Hypergraph-Based data Reduced Scheduling Policy for data-intensive workflow in Clouds 3rd

Hypergraph-Based Data Reduced Scheduling Policy for Data-Int...

引用

3rd International Conference of Pioneering Computer Scientists, Engineers and Educators (ICPCSEE)

作者： Hu, Zhigang Li, Jia Zheng, Meiguang Zhang, Xinxin Kang, Hui Tao, Yong Yang, Jiao Cent S Univ Sch Software Changsha 410075 Hunan Peoples R China

ISBN: (纸本)9789811063886;9789811063879

data-intensive computing is expected to be the next-generation IT computing paradigm. data-intensive workflows in clouds are becoming more and more popular. How to schedule data-intensive workflow efficiently has become the key issue. In this paper, first, we build a directed hypergraph model for data-intensive workflow, since Hypergraphs can more accurately model communication volume and better represent asymmetric problems, and the cut metric of hypergraphs is well suited for minimizing the total volume of communication. Second, we propose a concept data supportive ability to help the presentation of data-intensive workflow application and provide the merge operation details considering the data supportive ability. Third, we present an optimized hypergraph multi-level partitioning algorithm. Finally we bring a data reduced scheduling policy HEFT-P for data-intensive workflow. Through simulation, we compare HEFT-P with three typical workflow scheduling policies. The results indicate that HEFT-P could obtain reduced data scheduling and reduce the makespan of executing data-intensive workflows.

关键词： data-intensive workflow Directed hypergraph data reduced scheduling Cloud computing

来源：评论

学校读者我要写书评

暂无评论

A Dynamic Variable Exchange Method for data-intensive workflow 18

A Dynamic Variable Exchange Method for Data-Intensive Workfl...

引用

18th International Conference on Advanced Communication Technology (ICACT)

作者： Park, Minjae Ma, Jin Kim, Hyunah Ahn, Hyun Kim, Kwanghoon Pio BISTel Inc Engn Innovat Ctr Seoul South Korea KISTI Natl Inst Supercomp & Networking Daejeon South Korea Kyonggi Univ Dept Comp Sci Suwon Kyonggido South Korea

ISBN: (纸本)9788996865063

In order to create or modify workflow model which is validated, it is good to support real-time trace and debug for data-intensive workflow model design tool. And, if it is possible to change variables of workflow instance, when workflow model trace or debug, it is very useful. So, we propose dynamic variables exchange method, and attempt to exchange variables on a data-intensive workflow modelling tool with visual approach.

关键词： data-intensive workflow workflow trace or debug workflow variables exchange

来源：评论

学校读者我要写书评

暂无评论

ABS: Agent-based Scheduling for data-intensive workflow in Software-as-a-Service Environments 4

ABS: Agent-based Scheduling for Data-Intensive Workflow in S...

引用

4th International Conference on Advanced Cloud and Big data (CBD)

作者： Chen, Huangke Meng, Jiayang Zhu, Jianghan Wang, Jianjiang Natl Univ Def Technol Sci & Technol Informat Syst Engn Lab Changsha 410073 Hunan Peoples R China

ISBN: (纸本)9781509036776

Efficient allocation and execution of data-intensive scientific workflows to reduce execution time and the data transferring overhead still remains enormous challenge for the adoption of cloud computing to execute data-intensive scientific workflows. To face such a challenge, in this paper, we introduce an agent-based scheduling approach, namely ABS, that automates the deployment of data-intensive workflows in Softwareas-a-Service (SaaS) environments considering two objectives: makespan and data transfer overhead. The algorithm ABS includes two phases: 1) planning phase, task agents in the same workflow form groups through cooperation to reduce data transfer among tasks;2) scheduling phase, task groups will be assigned to VMs such that reducing makespans for workflows. The experiments conducted using real-world workflow applications demonstrate that our solution outperform the solution produced by the existing algorithm.

关键词： Cloud computing multi-agent systems scheduling data-intensive workflow Software-as-a-Service

来源：评论

学校读者我要写书评

暂无评论

Hypergraph-Based data Reduced Scheduling Policy for data-intensive workflow in Clouds

Hypergraph-Based Data Reduced Scheduling Policy for Data-Int...

引用

The Third International Conference of Pioneering Computer Scientists, Engineers and Educators,ICPCSEE 2017(originally ICYCSEE)

作者： Zhigang Hu Jia Li Meiguang Zheng Xinxin Zhang Hui Kang Yong Tao Jiao Yang School of Software Central South University

data-intensive computing is expected to be the next-generation IT computing paradigm. data-intensive workflows in clouds are becoming more and more popular. How to schedule data-intensive workflow efficiently has become the key issue. In this paper, first, we build a directed hypergraph model for data-intensive workflow, since Hypergraphs can more accurately model communication volume and better represent asymmetric problems, and the cut metric of hypergraphs is well suited for minimizing the total volume of communication. Second, we propose a concept data supportive ability to help the presentation of data-intensive workflow application and provide the merge operation details considering the data supportive ability. Third, we present an optimized hypergraph multi-level partitioning algorithm. Finally we bring a data reduced scheduling policy HEFT-P for data-intensive workflow. Through simulation, we compare HEFT-P with three typical workflow scheduling *** results indicate that HEFT-P could obtain reduced data scheduling and reduce the makespan of executing data-intensive workflows.

关键词： data-intensive workflow Directed hypergraph data reduced scheduling Cloud computing

来源：评论

学校读者我要写书评

暂无评论

A Dynamic Variable Exchange Method for data-intensive workflow

A Dynamic Variable Exchange Method for Data-Intensive Workfl...

引用

International Conference on Advanced Communication Technology

作者： Minjae Park Jin Ma Hyunah Kim Hyun Ahn Kwanghoon Pio Kim Engineering Innovation Center BISTel Inc. Seoul South Korea National Institute of Supercomputing and Networking KISTI Daejeon South Korea Dept. of Computer Science Kyonggi University Suwon Kyonggido South Korea

ISBN: (纸本)9781509016969

关键词： data-intensive workflow workflow trace or debug workflow variables exchange

来源：评论

学校读者我要写书评

暂无评论

ParaLite: A Parallel database System for data-intensive workflows

引用

IEICE TRANSACTIONS ON INFORMATION AND SYSTEMS 2014年第5期E97D卷 1211-1224页

作者： Chen, Ting Taura, Kenjiro Univ Tokyo Tokyo 1130033 Japan

To better support data-intensive workflows which are typically built out of various independently developed executables, this paper proposes extensions to parallel database systems called User-Defined eXecutables (UDX) and collective queries. UDX facilitates the description of workflows by enabling seamless integrations of external executables into SQL statements without any efforts to write programs confirming to strict specifications of databases. A collective query is an SQL query whose results are distributed to multiple clients and then processed by them in parallel, using arbitrary UDX. It provides efficient parallelization of executables through the data transfer optimization algorithms that distribute query results to multiple clients, taking both communication cost and computational loads into account. We implement this concept in a system called Para Lite, a parallel database system based on a popular lightweight database SQLite. Our experiments show that Para Lite has several times higher performance over Hive for typical SQL tasks and has 10x speedup compared to a commercial DBMS for executables. In addition, this paper studies a real-world text processing workflow and builds it on top of Para Lite, Hadoop, Hive and general files. Our experiences indicate that Para Lite outperforms other systems in both productivity and performance for the workflow.

关键词： data-intensive workflow parallel database system user-defined executable collective query

来源：评论

学校读者我要写书评

暂无评论

Improving the energy efficiency and performance of data-intensive workflows in virtualized clouds

引用

JOURNAL OF SUPERCOMPUTING 2018年第7期74卷 2935-2955页

作者： Qu, Xilong Xiao, Peng Huang, Lirong Hunan Univ Finance & Econ Sch Informat Technol & Management Changsha 410205 Hunan Peoples R China Hunan Inst Engn Sch Comp & Commun Xiangtan 411105 Peoples R China

In recent years, deploying and running data-intensive workflows in cloud platform has become more and more popular in many areas. Unlike computation-intensive applications, a data-intensive workflow typically requires to deal with bulk data transferring between different resource sites, which means some traditional energy-efficiency optimization technologies are difficult to be enforced when running data-intensive workflows. In this paper, we first formulate the power model of a data-intensive workflow, which takes into account power consumption caused by data transferring. Based on this power model, we introduce a novel metric called Shortest Path in terms of Energy Consumption and design an energy-efficient heuristic scheduling algorithm, which is aiming at reducing the extra energy consumption caused by delays of bulk data transferring. Extensive experiments and performance evaluations show that the proposed scheduling algorithm can significantly reduce the overall energy consumption of running data-intensive workflows comparing with several existing algorithms. In addition, the proposed algorithm also exhibits better adaptiveness and robustness when a cloud system is facing intensive and unpredicted workloads.

关键词： Cloud computing data-intensive workflow Quality of service Makespan Energy consumption

来源：评论

学校读者我要写书评

暂无评论

Risk-aware intermediate dataset backup strategy in cloud-based data intensive workflows

引用

FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE 2016年 55卷 524-533页

作者： Wang, Mingzhong Zhu, Liehuang Zhang, Zijian Beijing Inst Technol Sch Comp Sci Beijing 100081 Peoples R China

data-intensive workflows are generally computing- and data-intensive with large volume of data generated during their execution. Therefore, some of the data should be saved to avoid the expensive re-execution of tasks in case of exceptions. However, cloud-based data storage services come at some expense. In this paper, we introduce the risk evaluation model tailored for workflow structure to measure and achieve the trade-off between the overhead of backup storage and the cost of data regeneration in failure, making the service selection and execution more efficient and robust. The proposed method computes and compares the potential loss with and without data backup to achieve the trade-off between overhead of intermediate dataset backup and task re-execution after exceptions. We also design the utility function with the model and apply a genetic algorithm to find the optimized schedule. The results show that the robustness of the schedule is increased while the possible risk of failure is minimized, especially when the volume of generated data is not large in comparison with the input. (C) 2014 Elsevier B.V. All rights reserved.

关键词： data-intensive workflow Risk evaluation Robustness Intermediate dataset Checkpoint

来源：评论

学校读者我要写书评

暂无评论

A novel cloud model based data placement strategy for data-intensive application in clouds

引用

COMPUTERS & ELECTRICAL ENGINEERING 2019年 77卷 445-456页

作者： Zhang, Xinxin Hu, Zhigang Zheng, Meiguang Li, Jia Yang, Liu Cent S Univ Sch Software Changsha 410075 Hunan Peoples R China

Today, more and more data-intensive applications are deployed in cloud environments among geo-distributed data centers. It is a fundamental challenge to solve the data placement problem for data-intensive applications. The key problem is to present the uncertain and random process of data placement appropriately, while significantly reducing the transmission time across data centers and decreasing the frequency of data movement among data centers. In this paper, we introduce a new type of entity called a Virtual data Agent (VDA) and convert the data placement problem into two mapping processes, namely, mapping from the data set to the virtual data agent and mapping from the virtual data agent to the data center. We propose Cloud model based data Placement Algorithm with Virtual data Agent (CDPVDA). Through simulation using real workflow applications, we compare CDPVDA with two typical data placement strategies. The results indicate that CDPVDA could reduce unavoidable overhead in data transmission between the data centers by 5% to 20%. (C) 2018 Elsevier Ltd. All rights reserved.

关键词： data placement data-intensive workflow Cloud model Cloud computing

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：