检索结果-内蒙古大学图书馆

36th IEEE International Conference on Distributed Computing Systems (ICDCS)

作者： Thamsen, Lauritz Renner, Thomas Kao, Odej Tech Univ Berlin Berlin Germany

ISBN: (纸本)9781509014828

parallel dataflow systems like Apache Flink allow analysis of large datasets with iterative programs. However, allocating a cost-effective set of resources for such jobs is a difficult task as the resource utilization depends on many factors such as dataset size, key value distributions, computational complexity of programs, and the underlying hardware. What's more, some of these factors are not well known before the execution. There are, for example, often no data statistics such as key value distributions available beforehand. For this reason, we propose to improve the resource utilization at runtime using the repetitive nature of iterative dataflow programs. Based on runtime statistics gathered in previous iterations, the resource allocation is adapted dynamically at the synchronization barriers between iterations. This approach has two advantages: First, at barriers detailed statistics can be available, even for parallelly executed task pipelines. Second, at barriers dataflows can be adapted without complex handling of intermediate task state. This paper presents a prototype integrated with Apache Flink and an evaluation on a cluster with 480 cores. One experiment shows a 57% reduction of the job runtime by allocating more resources for a shorter time, another experiment a release of up to 40% surplus resources without significantly extending the job runtime.

关键词： parallel dataflows Scalable Data Processing Resource Utilization Dynamic Scaling

来源：评论

学校读者我要写书评

暂无评论

Emma in Action: Declarative dataflows for Scalable Data Analysis 16

Emma in Action: Declarative Dataflows for Scalable Data Anal...

引用

ACM SIGMOD International Conference on Management of Data

作者： Alexandrov, Alexander Salzmann, Andreas Krastev, Georgi Katsifodimos, Asterios Markl, Volker TU Berlin Berlin Germany

ISBN: (纸本)9781450335317

parallel dataflow APIs based on second-order functions were originally seen as a flexible alternative to SQL. Over time, however, their complexity increased due to the number of physical aspects that had to be exposed by the underlying engines in order to facilitate efficient execution. To retain a sufficient level of abstraction and lower the barrier of entry for data scientists, projects like Spark and Flink currently offer domain-specific APIs on top of their parallel collection abstractions. This demonstration highlights the benefits of an alternative design based on deep language embedding. We showcase Emma a programming language embedded in Scala. Emma promotes parallel collection processing through native constructs like Scala's for-comprehensions a declarative syntax akin to SQL. In addition, Emma also advocates quoting the entire data analysis algorithm rather than its individual dataflow expressions. This allows for decomposing the quoted code into (sequential) control flow and (parallel) dataflow fragments, optimizing the dataflows in context, and transparently offloading them to an engine like Spark or Flink. The proposed design promises increased programmer productivity due to avoiding an impedance mismatch, thereby reducing the lag times and cost of data analysis.

关键词： parallel dataflows emma scala macros large-scale data analysis monad comprehensions data-parallel execution mapreduce

来源：评论

学校读者我要写书评

暂无评论

When to Use a Distributed Dataflow Engine: Evaluating the Performance of Apache Flink 13

When to Use a Distributed Dataflow Engine: Evaluating the Pe...

引用

Conference on UIC/ATC/ScalCom/CBDCom/IoP/SmartWorld

作者： Verbitskiy, Ilya Thamsen, Lauritz Kao, Odej Tech Univ Berlin Berlin Germany

ISBN: (纸本)9781509027712

With the increasing amount of available data, distributed data processing systems like Apache Flink and Apache Spark have emerged that allow to analyze large-scale datasets. However, such engines introduce significant computational overhead compared to non-distributed implementations. Therefore, the question arises when using a distributed processing approach is actually beneficial. This paper helps to answer this question with an evaluation of the performance of the distributed data processing framework Apache Flink. In particular, we compare Apache Flink executed on up to 50 cluster nodes to single-threaded implementations executed on a typical laptop for three different benchmarks: TPC-H Query 10, Connected Components, and Gradient Descent. The evaluation shows that the performance of Apache Flink is highly problem dependent and varies from early outperformance in case of TPC-H Query 10 to slower runtimes in case of Connected Components. The reported results give hints for which problems, input sizes, and cluster resources using a distributed data processing system like Apache Flink or Apache Spark is sensible.

关键词： Apache Flink COST Distributed data processing parallel dataflows Performance evaluation

来源：评论

学校读者我要写书评

暂无评论

CoLoc: Distributed Data and Container Colocation for Data-Intensive Applications 4

CoLoc: Distributed Data and Container Colocation for Data-In...

引用

4th IEEE International Conference on Big Data (Big Data)

作者： Renner, Thomas Thamsen, Lauritz Kao, Odej Tech Univ Berlin Berlin Germany

ISBN: (纸本)9781467390057

The performance of scalable analytic frameworks supporting data-intensive parallel applications often depends significantly on the time it takes to read input data. Therefore, existing frameworks like Spark and Flink try to achieve a high degree of data locality by scheduling tasks on nodes where the input data resides. However, the set of nodes running a job and its tasks is chosen by a cluster resource management system like YARN, which schedules containers without taking the location of data into account. Yet, the scheduling of the frameworks is restricted to the set of nodes the containers are running on. At the same time, many jobs in productive clusters are recurring with predictable characteristics. For these jobs, it is possible to plan in advance on which nodes to place a job's input data and execution containers. In this paper we present CoLoc, a lightweight data and container scheduling assistant for recurring data-intensive analytic jobs. CoLoc allows users to define related files that serve as input for the same job. It colocates related files on a set of nodes and offers this scheduling hint to the cluster manager to also place the jobs container on these nodes. The main advantage of CoLoc is a reduction of network transfers due to a higher data locality and locally performed operators like grouping or joining two or more datasets. We implement CoLoc on Hadoop YARN and HDFS, then evaluate it on a 40 node cluster using workloads based on Apache Flink and the TPC-H benchmark suite. Compared to YARN's default scheduler and HDFS's block placement scheduler, CoLoc reduces the execution time up to 35% for the tested data-intensive workloads.

关键词： Resource Management Data Placement parallel dataflows Scheduling Data-Intensive Applications

来源：评论

学校读者我要写书评

暂无评论

Representations and Optimizations for Embedded parallel Dataflow Languages

引用

ACM TRANSACTIONS ON DATABASE SYSTEMS 2019年第1期44卷 4-4页

作者： Alexandrov, Alexander Krastev, Georgi Markl, Volker Tech Univ Berlin Database Syst & Informat Management Grp DIMA Bldg EN Off EN7Einsteinufer 17 D-10587 Berlin Germany TU Berlin Berlin Germany

parallel dataflow engines such as Apache Hadoop, Apache Spark, and Apache Flink are an established alternative to relational databases for modern data analysis applications. A characteristic of these systems is a scalable programming model based on distributed collections and parallel transformations expressed by means of second-order functions such as map and reduce. Notable examples are Flink's DataSet and Spark's RDD programming abstractions. These programming models are realized as EDSLs-domain specific languages embedded in a general-purpose host language such as Java, Scala, or Python. This approach has several advantages over traditional external DSLs such as SQL or XQuery. First, syntactic constructs from the host language (e.g., anonymous functions syntax, value definitions, and fluent syntax via method chaining) can be reused in the EDSL. This eases the learning curve for developers already familiar with the host language. Second, it allows for seamless integration of library methods written in the host language via the function parameters passed to the parallel dataflow operators. This reduces the effort for developing analytics dataflows that go beyond pure SQL and require domain-specific logic. At the same time, however, state-of-the-art parallel dataflow EDSLs exhibit a number of shortcomings. First, one of the main advantages of an external DSL such as SQL -the high-level, declarative Select-From-Where syntax-is either lost completely or mimicked in a non-standard way. Second, execution aspects such as caching, join order, and partial aggregation have to be decided by the programmer. Optimizing them automatically is very difficult due to the limited program context available in the intermediate representation of the DSL. In this article, we argue that the limitations listed above are a side effect of the adopted type-based embedding approach. As a solution, we propose an alternative EDSL design based on quotations. We present a DSL embedded in Scala a

关键词： parallel dataflows MapReduce monad comprehensions

来源：评论

学校读者我要写书评

暂无评论

When to Use a Distributed Dataflow Engine: Evaluating the Performance of Apache Flink

When to Use a Distributed Dataflow Engine: Evaluating the Pe...

引用

IEEE International Conference on Ubiquitous Intelligence and Computing

作者： Ilya Verbitskiy Lauritz Thamsen Odej Kao Technische Universitat Berlin

ISBN: (纸本)9781509027729

关键词： COST distributed data processing parallel dataflows Apache Flink performance evaluation

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：