Scheduling tasks in distributed computing infrastructures (DCIs) is challenging mainly because the scheduler is facing a number of more or less dependent parameters that characterize the hosts coming from a particular...
详细信息
ISBN:
(纸本)9781467344661;9781467344678
Scheduling tasks in distributed computing infrastructures (DCIs) is challenging mainly because the scheduler is facing a number of more or less dependent parameters that characterize the hosts coming from a particular computing environment and the tasks. In this paper we introduce a multicriteria scheduling method for DCIs, aiming a better matching between hosts, and tasks waiting in a priority queue at a pull-based scheduler. The novelty of the approach consists in employing the Promethee [1] decision aid for selecting tasks. In the aim of computing preference relationships (priorities) among tasks, this approach performs pairwise comparisons of values that characterize tasks. The method exhibits interesting advantages, such as allowing the user to choose the values for the computation of the priorities, like the expected completion time (ECT) and cost. The approach is also very flexible, allowing through a set of parameters the specification of particular scheduling policies. To validate this method we built an XtrebWeb-like simulator, which is capable of running on real traces. We experiment on internet desktop grid (IDG), cloud and best effort grid (BEG), with various workloads. The results show that the Promethee-based scheduling method obtains good performance especially on IDG when certain fractions of the tasks fail. We also prove that multi-criteria scheduling using Promethee performs better than single-criterion scheduling, improving both makespan and cost. Also, a simple definition of ECT is the most efficient in terms of makespan. In this work we also explain the challenges of using Promethee for scheduling in DCIs.
An efficient exploitation of the distributed computing infrastructures (DCIs) is specially needed to deal with the data deluge that the scientific community, in particular the Astrophysics one, is facing. This require...
详细信息
ISBN:
(纸本)9781467374590
An efficient exploitation of the distributed computing infrastructures (DCIs) is specially needed to deal with the data deluge that the scientific community, in particular the Astrophysics one, is facing. This requires a good understanding of the underlying DCIs. Science Gateways (SGs) provide the users with an environment that eases the interaction with the DCIs. As a previous step, IT skilled users should populate the SGs with friendly but advanced tools (e.g. workflows, visualization tools) that not only support the scientists to build their own experiments but also adapt them in an optimal way to the infrastructures. In Astronomy, the Virtual Observatory provides the community with services and tools for data access and sharing. However, state of the art telescopes and the coming Square Kilometre Array (SKA), able to reach data rates in the exa-scale domain, will also require advanced tools for data analysis and visualization that should be run on DCIs as well as shared on SGs. In the here presented work, we have selected as exemplar a set of analysis tasks of interest for some SKA use cases. These analysis tasks have been implemented as web services that use the COMPSs programming model in order to achieve a more efficient use of the DCIs. At the same time, the nature of the web services turns them into blocks that the astronomers can combine with VO services to build their own workflows. The web services and the workflows built upon them form a two-level workflow system that hides the technical details of the DCIs and exploits them efficiently. This approach is used for the first time in analytical tasks of interest for the SKA that benefits from the capabilities of the DCIs.
Even though the Italian Grid Infrastructure (IGI) is a general purpose distributed platform, in the past it has been used mainly for serial computations. Parallel applications have been typically executed on supercomp...
详细信息
ISBN:
(纸本)9780769549392;9781467353212
Even though the Italian Grid Infrastructure (IGI) is a general purpose distributed platform, in the past it has been used mainly for serial computations. Parallel applications have been typically executed on supercomputer facilities or, in case of "not high-end" HPC applications, on local commodity parallel clusters. Nowadays, with the availability of multiple cores processors, Grid computing is becoming very attractive also for parallel applications but some problems exist in supporting of HPC applications on Grid environment. Here we describe the work made to set up a HPC testbed for "not high-end" HPC applications, based on IGI Grid technologies, to find solutions to those problems. Participating sites have been selected among the ones running HPC clusters in Grid environment. Each of them contributed with their specific HPC experience and their available resources to the present test, which encompasses an unprecedented large set of applications from different disciplines in the fields of astronomy, astrophysics, chemistry, climatology, material science and oceanography. In addition to computing resources sharing, the main contribution of each participant was the identification of the real requirements of his application also related to the current middleware limitations and then the realization of a test platform enhanced with additional HPC solutions and configurations developed in a tight collaboration between HPC administrators, users and IGI managers. The main work was on computational resources selection, data management and the definition, the deployment and the documentation of the software execution environment. The outcoming results of the testbed represent the basis of the HPC support in the IGI production infrastructure.
Scientific workflows in clouds have been successfully used for automation of large-scale computations, but so far they were applied to the loosely-coupled problems, where most workflow tasks can be processed independe...
详细信息
Scientific workflows in clouds have been successfully used for automation of large-scale computations, but so far they were applied to the loosely-coupled problems, where most workflow tasks can be processed independently in parallel and do not require high volume of communication. The multi-frontal solver algorithm for finite element meshes can be represented as a workflow, but the fine granularity of resulting tasks and the large communication to computation ratio makes it hard to execute it efficiently in loosely-coupled environments such as the Infrastructure-as-a-Service clouds. In this paper, we hypothesize that there exists a class of meshes that can be effectively decomposed into a workflow and mapped onto a cloud infrastructure. To show that, we have developed a workflow-based multi-frontal solver using the HyperFlow workflow engine, which comprises workflow generation from the elimination tree, analysis of the workflow structure, task aggregation based on estimated computation costs, and distributed execution using a dedicated worker service that can be deployed in clouds or clusters. The results of our experiments using the workflows of over 10,000 tasks indicate that after task aggregation the resulting workflows of over 100 tasks can be efficiently executed, and the overheads are not prohibitive. These results lead us to conclusions that our approach is feasible and gives prospects for providing a generic workflow-based solution using clouds for problems typically considered as requiring HPC infrastructure.
Scientific workflows in clouds have been successfully used for automation of large-scale computations, but so far they were applied to the loosely-coupled problems, where most workflow tasks can be processed independe...
详细信息
Scientific workflows in clouds have been successfully used for automation of large-scale computations, but so far they were applied to the loosely-coupled problems, where most workflow tasks can be processed independently in parallel and do not require high volume of communication. The multi-frontal solver algorithm for finite element meshes can be represented as a workflow, but the fine granularity of resulting tasks and the large communication to computation ratio makes it hard to execute it efficiently in loosely-coupled environments such as the Infrastructure-as-a-Service clouds. In this paper, we hypothesize that there exists a class of meshes that can be effectively decomposed into a workflow and mapped onto a cloud infrastructure. To show that, we have developed a workflow-based multi-frontal solver using the HyperFlow workflow engine, which comprises workflow generation from the elimination tree, analysis of the workflow structure, task aggregation based on estimated computation costs, and distributed execution using a dedicated worker service that can be deployed in clouds or clusters. The results of our experiments using the workflows of over 10,000 tasks indicate that after task aggregation the resulting workflows of over 100 tasks can be efficiently executed, and the overheads are not prohibitive. These results lead us to conclusions that our approach is feasible and gives prospects for providing a generic workflow-based solution using clouds for problems typically considered as requiring HPC infrastructure.
暂无评论