检索结果-内蒙古大学图书馆

data stream processing via code annotations

JOURNAL OF SUPERCOMPUTING 2018年第11期74卷 5659-5673页

作者： Danelutto, Marco De Matteis, Tiziano Mencagli, Gabriele Torquati, Massimo Univ Pisa Dept Comp Sci Pisa Italy

Time-to-solution is an important metric when parallelizing existing code. The REPARA approach provides a systematic way to instantiate stream and data parallel patterns by annotating the sequential source code with attributes. Annotations are automatically transformed in a target parallel code that uses existing libraries for parallel programming (e.g., FastFlow). In this paper, we apply this approach for the parallelization of a data stream processing application. The description shows the effectiveness of the approach in easily and quickly prototyping several parallel variants of the sequential code by obtaining good overall performance in terms of both throughput and latency.

关键词： Code annotations Parallel patterns data stream processing

来源：评论

学校读者我要写书评

暂无评论

data stream processing in Factory Automation

Data Stream Processing in Factory Automation

引用

15th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA)

作者： Wolf, Bernhard Herzig, Philipp Behrens, Ingo Majumdar, Anirban Ameling, Michael SAP Res Ctr Dresden SAP AG Dresden Germany

ISBN: (纸本)9781424468508

data stream processing is a valuable technique to solve demanding problems that also occur in factory automation, such as continuous data processing with high throughput and real-time output, and distributed data acquisition and processing. However, the intricacies of data stream processing techniques make its application difficult in real-life scenarios. One particularly challenging situation arises when changing conditions necessitate a modification in processing logic of system operators. This is especially difficult in the presence of streaming data and transient internal states of the system. Since downtimes are expensive, an efficient solution has to be provided for updating the processing logic. In this paper, strategies for on-the-fly adaptation of data stream queries are presented and experimentally evaluated with examples from the domain of condition-based maintenance. Techniques for state preservation allow for a fast transition to new processing logic. The results show that our strategies are well suited for demanding applications in factory environments.

关键词： condition based maintenance data acquisition data stream processing distributed data acquisition distributed data processing factory automation factory automation

来源：评论

学校读者我要写书评

暂无评论

Energy consumption analysis of data stream processing: a benchmarking approach

引用

SOFTWARE-PRACTICE & EXPERIENCE 2017年第10期47卷 1443-1462页

作者： Dayarathna, Miyuru Li, Yuanlong Wen, Yonggang Fan, Rui Nanyang Technol Univ Sch Comp Engn Singapore Singapore

Energy efficiency of data analysis systems has become a very important issue in recent times because of the increasing costs of data center operations. Although distributed streaming workloads have increasingly been present in modern data centers, energy-efficient scheduling of such applications remains as a significant challenge. In this paper, we conduct an energy consumption analysis of data stream processing systems in order to identify their energy consumption patterns. We follow stream system benchmarking approach to solve this issue. Specifically, we implement Linear Road benchmark on six stream processing environments (S4, Storm, ActiveMQ, Esper, Kafka, and Spark streaming) and characterize these systems' performance on a real-world data center. We study the energy consumption characteristics of each system with varying number of roads as well as with different types of component layouts. We also use a microbenchmark to capture raw energy consumption characteristics. We observed that S4, Esper, and Spark streaming environments had highest average energy consumption efficiencies compared with the other systems. Using a neural networkbased technique with the power/performance information gathered from our experiments, we developed a model for the power consumption behavior of a streaming environment. We observed that energy-efficient execution of streaming application cannot be specifically attributed to the system CPU usage. We observed that communication between compute nodes with moderate tuple sizes and scheduling plans with balanced system overhead produces better power consumption behaviors in the context of data stream processing systems. Copyright (c) 2016 John Wiley & Sons, Ltd.

关键词： data stream processing energy consumption analysis benchmarking workload characterization distributed systems linear road

来源：评论

学校读者我要写书评

暂无评论

Proactive elasticity and energy awareness in data stream processing

引用

JOURNAL OF SYSTEMS AND SOFTWARE 2017年 127卷 302-319页

作者： De Matteis, Tiziano Mencagli, Gabriele Univ Pisa Dept Comp Sci Largo B Pontecorvo 3 I-56127 Pisa Italy

data stream processing applications have a long running nature (24 hr/7 d) with workload conditions that may exhibit wide variations at run-time. Elasticity is the term coined to describe the capability of applications to change dynamically their resource usage in response to workload fluctuations. This paper focuses on strategies for elastic data stream processing targeting multicore systems. The key idea is to exploit Model Predictive Control, a control-theoretic method that takes into account the system behavior over a future time horizon in order to decide the best reconfiguration to execute. We design a set of energy aware proactive strategies, optimized for throughput and latency QoS requirements, which regulate the number of used cores and the CPU frequency through the Dynamic Voltage and Frequency Scaling (DVFS) support offered by modern multicore CPUs. We evaluate our strategies in a high-frequency trading application fed by synthetic and real-world workload traces. We introduce specific properties to effectively compare different elastic approaches, and the results show that our strategies are able to achieve the best outcome. (C) 2016 Elsevier Inc. All rights reserved.

关键词： data stream processing Elasticity Model predictive control Frequency scaling

来源：评论

学校读者我要写书评

暂无评论

A Hybrid Method for Fast Predicate Matching in data stream processing

引用

JOURNAL OF INFORMATION SCIENCE AND ENGINEERING 2012年第2期28卷 365-377页

作者： Kim, Hyeon Gyu Kim, Myoung Ho Korea Adv Inst Sci & Technol Dept Comp Sci Taejon 305701 South Korea

Efficient predicate matching is essential to provide real-time responses in data stream processing. Many predicate indexes have been introduced to address the predicate matching problem. Among them, the IBS-tree handles predicates with open and closed intervals in O0(log(2)n) time. However, it was found that its insertion algorithm is not optimal when processing open intervals. In this paper, a predicate indexing method is proposed that supports fast insertion of predicates based on the IBS-tree scheme without degrading the search performance. The basic idea of the proposed method is to use shared lists when processing open intervals, and the insertion time is reduced to O(log n) in this case. When dealing with closed intervals, an algorithm of the IBS-tree was used. Consequently, the insertion performance of the proposed method improves as the ratio of open intervals increases, which we observed through our experiments.

关键词： predicate matching predicate index shared list IBS-tree data stream processing

来源：评论

学校读者我要写书评

暂无评论

Pec: Proactive Elastic Collaborative Resource Scheduling in data stream processing

引用

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS 2019年第7期30卷 1628-1642页

作者： Wei, Xiaohui Li, Lina Li, Xiang Wang, Xingwang Gao, Shang Li, Hongliang Jilin Univ Symbol Computat & Knowledge Engineer Minist Educ Coll Comp Sci & Technol Changchun 130012 Jilin Peoples R China Jilin Univ Coll Comp Sci & Technol Changchun 130012 Jilin Peoples R China Jilin Univ High Performance Comp Ctr Changchun 130012 Jilin Peoples R China

In the Distributed Parallel stream processing Systems (DPSPS), elastic resource allocation allows applications to dynamically response to workload fluctuations. However, resource provisioning can be particularly challenging, due to the unpredictability of the workload. In addition, unlike CPU resources, bandwidth resources are often ignored in resource allocation. Moreover, resource allocation and resource placement are considered separately. In this paper, we investigate the proactive elastic resource scheduling problem for computation-intensive and communication-intensive applications, which aims at meeting the latency requirement with the minimal energy cost, and propose a dynamic collaborative strategy from the systemic perspective. Specifically, we first model a collaborative workload prediction pattern to accurately predict the upcoming workload, and construct a latency estimation model to estimate the latency of the application. Then, we design an energy-efficient resource pre-allocation method, in which the CPU frequency adjustment and the stability of resource reconfigurations are both considered. Finally, we present a communication-aware resource placement approach. Simulation results show that, compared with the reactive strategies, our strategy achieves an obviously better latency performance, and effectively avoids unnecessary resource adjustments. Meanwhile, the energy consumption is about saved by 50 percent on average, and the communication cost is maintained at a very low level of 4 percent.

关键词： data stream processing resource scheduling workload prediction elastic resource allocation proactive strategy stable reconfiguration

来源：评论

学校读者我要写书评

暂无评论

Elastic Scaling for data stream processing

引用

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS 2014年第6期25卷 1447-1463页

作者： Gedik, Bugra Schneider, Scott Hirzel, Martin Wu, Kun-Lung Bilkent Univ Comp Engn Deparment TR-06800 Ankara Turkey IBM Thomas J Watson Res Ctr Yorktown Hts NY 10598 USA

This article addresses the profitability problem associated with auto-parallelization of general-purpose distributed data stream processing applications. Auto-parallelization involves locating regions in the application's data flow graph that can be replicated at run-time to apply data partitioning, in order to achieve scale. In order to make auto-parallelization effective in practice, the profitability question needs to be answered: How many parallel channels provide the best throughput? The answer to this question changes depending on the workload dynamics and resource availability at run-time. In this article, we propose an elastic auto-parallelization solution that can dynamically adjust the number of channels used to achieve high throughput without unnecessarily wasting resources. Most importantly, our solution can handle partitioned stateful operators via run-time state migration, which is fully transparent to the application developers. We provide an implementation and evaluation of the system on an industrial-strength data stream processing platform to validate our solution.

关键词： data stream processing parallelization elasticity

来源：评论

学校读者我要写书评

暂无评论

DSPBench: A Suite of Benchmark Applications for Distributed data stream processing Systems

引用

IEEE ACCESS 2020年 8卷 222900-222917页

作者： Bordin, Maycon Viana Griebler, Dalvan Mencagli, Gabriele Geyer, Claudio F. R. Fernandes, Luiz Gustavo L. Fed Univ Rio Grande Sul UFRGS Inst Informat BR-91509900 Porto Alegre RS Brazil Pontifcal Catholic Univ Rio Grande Sul PUCRS Sch Technol BR-90619900 Porto Alegre RS Brazil SETREM Tres De Maio Fac Lab Adv Res Cloud Comp LARCC BR-98910000 Tres De Maio Brazil Univ Pisa Dept Comp Sci I-56127 Pisa Italy

Systems enabling the continuous processing of large data streams have recently attracted the attention of the scientific community and industrial stakeholders. data stream processing Systems (DSPSs) are complex and powerful frameworks able to ease the development of streaming applications in distributed computing environments like clusters and clouds. Several systems of this kind have been released and currently maintained as open source projects, like Apache Storm and Spark streaming. Some benchmark applications have often been used by the scientific community to test and evaluate new techniques to improve the performance and usability of DSPSs. However, the existing benchmark suites lack of representative workloads coming from the wide set of application domains that can leverage the benefits offered by the stream processing paradigm in terms of near real-time performance. The goal of this article is to present a new benchmark suite composed of 15 applications coming from areas like Finance, Telecommunications, Sensor Networks, Social Networks and others. This article describes in detail the nature of these applications, their full workload characterization in terms of selectivity, processing cost, input size and overall memory occupation. In addition, it exemplifies the usefulness of our benchmark suite to compare real DSPSs by selecting Apache Storm and Spark streaming for this analysis.

关键词： Benchmark testing Storms Sparks Task analysis Throughput Distributed databases Tools data stream processing big data benchmarking apache storm spark streaming

来源：评论

学校读者我要写书评

暂无评论

Paving the way towards high-level parallel pattern interfaces for data stream processing

引用

FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE 2018年 87卷 228-241页

作者： del Rio Astorga, David Dolz, Manuel F. Fernandez, Javier Daniel Garcia, J. Univ Carlos III Madrid Comp Sci & Engn Dept Leganes 28911 Spain

The emergence of the Internet of Things (IoT) data stream applications has posed a number of new challenges to existing infrastructures, processing engines, and programming models. In this sense, high-level interfaces, encapsulating algorithmic aspects in pattern-based constructions, have considerably reduced the development and parallelization efforts of this type of applications. An example of parallel pattern interface is GRPPI, a C++ generic high-level library that acts as a layer between developers and existing parallel programming frameworks, such as C++ threads, OpenMP and Intel TBB. In this paper, we complement the basic patterns supported by GRPPI with the new stream operators Split-Join and Window, and the advanced parallel patterns stream-Pool, Windowed-Farm and stream-Iterator for the aforementioned back ends. Thanks to these new stream operators, complex compositions among streaming patterns can be expressed. On the other hand, the collection of advanced patterns allows users to tackle some domain-specific applications, ranging from the evolutionary to the real-time computing areas, where compositions of basic patterns are not capable of fully mimicking the algorithmic behavior of their original sequential codes. The experimental evaluation of the new advanced patterns and the stream operators on a set of domain-specific use-cases, using different back ends and pattern-specific parameters, reports considerable performance gains with respect to the sequential versions. Additionally, we demonstrate the benefits of the GRPPI pattern interface from the usability, flexibility and readability points of view. (C) 2018 Elsevier B.V. All rights reserved.

关键词： data stream processing Parallel programming framework stream operator Domain-specific parallel pattern High-level API

来源：评论

学校读者我要写书评

暂无评论

Optimizing data stream processing for large-scale applications

引用

SOFTWARE-PRACTICE & EXPERIENCE 2018年第9期48卷 1607-1641页

作者： Cappellari, Paolo Roantree, Mark Chun, Soon Ae CUNY New York NY 10017 USA Dublin City Univ Sch Comp Insight Ctr Data Analyt Dublin Ireland

stream processing systems are designed to analyze data arriving in real time and using continuous queries and respond when a specific event or sequence of events are detected. An important aspect of these systems is streaming Analytics, which facilitates statistical calculations on continuous data within the stream. These systems must be designed to handle high volumes of data, be scalable, and accommodate a multitude of long-lived concurrently running analytics. The challenges involved in the development of stream processing include on-the-fly transformation of data streams to match the query needs of users and the ability to model stream transformations to detect overlaps and possibilities for optimizations and to specify a methodology to deliver optimizations. In particular, this work focuses on exposing data stream application internals in order to detect reusable parts and then consolidate applications to optimize computational resource usage. The streaming data Analytics Model presented in this paper adopts a declarative approach that enables processing and manipulation of data streams in a simple manner while facilitating powerful optimizations necessary for managing high volumes of streaming data in real time. An evaluation is provided to demonstrate in both theoretical and quantitative aspects the high performance offered by our approach.

关键词： continuous queries data stream processing distributed systems optimization

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：