检索结果-内蒙古大学图书馆

SOOM: Sort-Based Optimizer for Big Data multi-query

BIG DATA 2020年第1期8卷 38-61页

作者： Sahal, Radhya Khafagy, Mohammed H. Omara, Fatma A. Cairo Univ Fac Comp & Informat Cairo Egypt Natl Univ Ireland CONFIRM Ctr Smart Mfg Galway Ireland Fayoum Univ Fac Comp & Informat Al Fayyum Egypt

Mostly, sorting of data is a common operation in many applications, which causes the consumption of resources and thus leads to computation overheads. Regarding the context of Big Data multi-query, the shared sort operations are fairly large, which incur high-cost I/Os whether explicit or implicit. In particular, Big Data multi-query, including aggregation and sort operations, takes long execution time due to reshuffle of the same data multiple times using similar tasks. Therefore, exploiting the sharing data and the sharing sort opportunities of similar tasks can offer the possibility of reusing the previous results to optimize multi-query. For considering sharing data, our previous work, multi-query optimization Using Tuple Size and Histogram (MOTH) system, has been introduced to consider the granularity of the sharing data opportunities among multi-query. However, time overheads regarding redundant data in-network movement (i.e., shuffling time to transfer intermediate data for sort operations) have not been considered. Therefore, the MOTH system has been extended to SOOM (Sort-Based Optimizer over MOTH) system to exploit sharing sort opportunities, including explicit sorts of sort queries and implicit sorts of aggregation queries. The proposed SOOM system consists of two additional modules to exploit sharing sort opportunities, namely query explorer and sort exploiter, which leverage our existing MOTH system to fulfill optimizing multiple aggregation and sort queries. The experimental evaluation has shown that the SOOM system outperforms the naive and the state-of-art techniques regarding query execution time among queries by 45% and 30%, respectively, while introducing maximal intermediate data size reduction by 67% and 61% in average, respectively, over Hadoop-like infrastructures.

关键词： aggregation Big Data multi-query optimization sharing opportunity sort

来源：评论

学校读者我要写书评

暂无评论

Efficient processing of enumerative set-based queries

引用

INFORMATION SYSTEMS 2016年 55卷 54-72页

作者： Wang, Guoping Chan, Chee-Yong Huawei Shannon Lab Hangzhou 310051 Zhejiang Peoples R China Natl Univ Singapore Sch Comp Dept Comp Sci Comp 1 13 Comp Dr Singapore 117417 Singapore

Many applications often require finding sets of entities of interest that meet certain constraints. Such set-based queries (SQs) can be broadly classified into two types: optimization SQs that involve some optimization constraint and enumerative SQs that do not have any optimization constraint. While there has been much research on the evaluation of optimization SQs, there is very little work on the evaluation of enumerative SQs, which represent the most fundamental fragment of set-based queries. In this paper, we address the problem of evaluating enumerative SQs using RDBMS. While enumerative SQs can be expressed using SQL, existing relational engines, unfortunately, are not able to efficiently evaluate such queries due to their complexity. In this paper, we propose a novel evaluation approach for enumerative SQs. Our experimental results on PostgreSQL demonstrate that our proposed approach outperforms the conventional approach by up to three orders of magnitude. (C) 2015 Elsevier Ltd. All rights reserved.

关键词： Set-based queries multi-query optimization Relational database systems

来源：评论

学校读者我要写书评

暂无评论

Smart scheme: an efficient query execution scheme for event-driven stream processing

引用

KNOWLEDGE AND INFORMATION SYSTEMS 2019年第2期58卷 341-370页

作者： Shaikh, Salman Ahmed Watanabe, Yousuke Wang, Yan Kitagawa, Hiroyuki Univ Tsukuba Ctr Computat Sci Tsukuba Ibaraki Japan Nagoya Univ Inst Innovat Future Soc Nagoya Aichi Japan Univ Tsukuba Grad Sch Syst & Informat Engn Tsukuba Ibaraki Japan

With the increase in stream data, a demand for stream processing has become diverse and complicated. To meet this demand, several stream processing engines (SPEs) have been developed which execute continuous queries (CQs) to process continuous data streams. Event-driven stream processing, which is one of the important requirements, continuously gets the incoming stream data and, however, generates query results only on the occurrence of specified events. In the basic query execution scheme, even when no event is raised, input stream tuples are continuously processed by query operators, though they do not generate any query result. This results in increased system load and wastage of system resources. For this problem, we propose a smart event-driven stream processing scheme, which makes use of smart windows to buffer the stream tuples during the absence of an event. When the event is raised, the buffered tuples are flushed and processed by the downstream operators. If the buffered tuples in the smart window expire due to the window size before the occurrence of an event, they are deleted directly from the smart window. Since CQs once registered are executed for several weeks, months or even years, SPEs usually execute several CQs in parallel and merge their query plans whenever possible to save processing cost. Due to the presence of smart window, existing multi-query optimization techniques cannot work for smart event-driven stream processing. Hence, this work proposes a multi-query optimization for the proposed smart scheme to cover the cases where multiple continuous queries are registered. Extensive experiments are performed on real and synthetic data streams to show the effectiveness of the proposed smart scheme and its multi-query optimization.

关键词： Data stream processing Event-driven processing Smart query execution Smart window multi-query optimization Gate operator

来源：评论

学校读者我要写书评

暂无评论

Resource-efficient Shared query Execution via Exploiting Time Slackness 21

Resource-efficient Shared Query Execution via Exploiting Tim...

引用

ACM SIGMOD International Conference on Management of Data (SIGMOD)

作者： Tang, Dixin Shang, Zechao Ma, William W. Elmore, Aaron J. Krishnan, Sanjay Univ Calif Berkeley Berkeley CA 94720 USA Snowflake Comp Bozeman MT USA Univ Chicago Chicago IL 60637 USA

ISBN: (纸本)9781450383431

Shared query execution can reduce resource consumption by sharing common sub-expressions across concurrent queries. We show that this is not always the case when regularly querying a dataset under change. Depending on latency goals, how eagerly to incrementally process the new data differs. Naively sharing the execution of queries with different latency goals will push the whole shared plan to meet the lowest latency goal and execute more eagerly than each participating query. The overhead introduced by the eager execution can even offset the benefit of shared query execution. We propose an optimization framework iShare to exploit the benefit of shared execution and avoid the overhead of eager execution. iShare judiciously shares queries with different latency goals and selectively executes parts of the share plan lazily. iShare can significantly reduce resource consumption compared to eagerly executing share plans from the state-of-the-art multi-query optimizer or approaches that execute queries separately.

关键词： shared query execution multi-query optimization incremental view maintenance scheduled queries cloud database

来源：评论

学校读者我要写书评

暂无评论

query optimization over Distributed Data Stream

Query Optimization over Distributed Data Stream

引用

9th International Conference on Hybrid Intelligent Systems (HIS 2009)

作者： Wang, Shuang Tan, Zhenhua Gao, Xiaoxing Northeastern Univ Software Coll Shenyang Peoples R China

ISBN: (纸本)9780769537450

Recent research efforts in the fields of data stream processing show the increasing importance of processing data streams, e.g., in the e-science domain. Together with the advent of peer-to-peer (P2P) networks and grid computing, this leads to the necessity of developing new techniques for distributing and processing continuous queries over data streams in such networks. These systems often have to process multiple similar but different continuous aggregation queries simultaneously. Since executing each query separately can lead to significant scalability and performance problems, it is vital to share resources by exploiting similarities in the queries. The challenge is to identify overlapping computations that may not be obvious In the queries themselves. In this paper, we propose a novel algorithmic solution for problem of finding the minimum number of queries in such a distributed-streams setting, in order to optimize the communicate cost across the network. The experiment result show that our approach gives us as much as magnitude performance improvement over the no-share settings.

关键词： multi-query optimization group by query data stream distributed system aggregation

来源：评论

学校读者我要写书评

暂无评论

The research on Data Flow Technology in Computer Network Security Monitoring

The research on Data Flow Technology in Computer Network Sec...

引用

IEEE Workshop on Advanced Research and Technology in Industry Applications (WARTIA)

作者： Yin XiaoHong Nanchang Inst Sci & Technol Nanchang 330108 Jiangxi Peoples R China

ISBN: (纸本)9781479969890

With the rapid development of computer technology and application of Internet is becoming more and more widely, the Internet plays a more and more important role in people's life. At the same time, all kinds of network security events emerge in endlessly, seriously threaten the application and development of the Internet. With the purpose of safety, network monitoring, have more and more important significance in the maintenance of normal efficiently network run, key facilities, information system security, etc.,. How to realize effective network transmission and efficient online analysis to a huge number of distributed network security monitoring data so as to provide further support for a variety of applications become a major challenge in the field of network security and data processing.

关键词： Network Security Monitoring Distributed Data Stream Cost Efficient Processing multi-query optimization Stream Cube

来源：评论

学校读者我要写书评

暂无评论

TAMPA: Tabu Search-based multiple Queries optimization for Wireless Sensor Networks

TAMPA: Tabu Search-based Multiple Queries Optimization for W...

引用

3rd International Conference on Wireless Communications, Networking and Mobile Computing (WiCOM 2007)

作者： Tang, MingJian Cao, Jinli Chilamkurti, Naveen K. La Trobe Univ Dept Comp Sci & Comp Engn Melbourne Vic 3086 Australia

ISBN: (纸本)9781424413119

In this paper, we address one of the wireless sensor network query;processing issues posed due to the lack of support for multiple sensor network queries. The objective of the paper is to provide efficient and effective support to multiple queries so that the set of queries are pre-processed before disseminating them into the sensor network. It is very important that only necessary works will be assigned to the sensor network by virtue of strict energy constraint. The problem is modeled by Minimum Set Cover, which is one of the NP-complete problems. We propose an optimization scheme called TAMPA - a TAbu search-based multiple queries optimization to find an optimal merge order. The final set of queries to be sent into the network then can be derived from that merge order. We evaluate the proposed algorithm by conducting extensive simulation studies. The results show that energy can be significantly saved while the overall workload still satisfies the user requirements.

关键词： Wireless Sensor Networks multi-query optimization Tabu Search

来源：评论

学校读者我要写书评

暂无评论

Real-Time multi-Pattern Detection over Event Streams 19

Real-Time Multi-Pattern Detection over Event Streams

引用

ACM SIGMOD International Conference on Management of Data (SIGMOD)

作者： Kolchinsky, Ilya Schuster, Assaf Technion Israel Inst Technol Haifa Israel

ISBN: (纸本)9781450356435

Rapid advances in data-driven applications over recent years have intensified the need for efficient mechanisms capable of monitoring and detecting arbitrarily complex patterns in massive data streams. This task is usually performed by complex event processing (CEP) systems. CEP engines are required to process hundreds or even thousands of user-defined patterns in parallel under tight real-time constraints. To enhance the performance of this crucial operation, multiple techniques have been developed, utilizing well-known optimization approaches such as pattern rewriting and sharing common subexpressions. However, the scalability of these methods is limited by the high computation overhead, and the quality of the produced plans is compromised by ignoring significant parts of the solution space. In this paper, we present a novel framework for real-time multi-pattern complex event processing. Our approach is based on formulating the above task as a global optimization problem and applying a combination of sharing and pattern reordering techniques to construct an optimal plan satisfying the problem constraints. To the best of our knowledge, no such fusion was previously attempted in the field of CEP optimization. To locate the best possible evaluation plan in the resulting hyperexponential solution space, we design efficient local search algorithms that utilize the unique problem structure. An extensive theoretical and empirical analysis of our system demonstrates its superiority over state-of-the-art solutions.

关键词： Complex Event Processing multi-query optimization

来源：评论

学校读者我要写书评

暂无评论

Adapting to application workflow in processing data integration queries

Adapting to application workflow in processing data integrat...

引用

6th International Conferernce on Grid and Cooperative Computing

作者： Wu, Yongwei Liu, Jia Chen, Gang Yang, Guangwen Cao, Bo Zheng, Weimin Tsinghua Univ Tsinghua Natl Lab Informat Sci & Technol Dept Comp Sci & Technol Beijing 100084 Peoples R China

ISBN: (纸本)9780769528717

Data integration has evolved to provide efficient data management across distributed and heterogeneous data sources in grid environment. However, existing works in data integration consider little knowledge about the above applications. In such settings, queries from the same application are processed independently. In this paper, application properties are noticed in order to improve query performance for data integration. We present a general-purpose and well-modular architecture for addressing data integration in grid (DIG) environment first, the modules of which can be flexibly deployed to adapt specific application. Moreover, special attention has been paid on query processing to accommodate application workflow with the component of query Flow Processor. It introduces multi-query optimization techniques to speed up overall response time for DIG.

关键词： data integration query Flow Processor multi-query optimization

来源：评论

学校读者我要写书评

暂无评论

A model for multiple-query processing based upon strong factoring

A model for multiple-query processing based upon strong fact...

引用

International Conference on Information Technology - Coding and Computing

作者： Alsabbagh, JR Raghavan, VV Grand Valley State Coll Dept Comp Sci & Informat Syst Allendale MI 49401 USA

ISBN: (纸本)0769521088

This paper proposes a model for interleaving the execution of multiple queries simultaneously. The queries are rewritten, in the context of each other, in order to induce common subexpressions (CSEs). A CSE will then need to be evaluated once only and its result is used in the evaluation of the queries that contain it, thereby leading to cost saving over the conventional query-at-a-time scenario. Two important aspects characterize the proposed model. First, it rewrites the queries in such a way that the isolated CSEs are, themselves, heuristically efficient to evaluate during query execution. Second, it reduces the optimization overhead (due to query rewriting) by using a divide-and-conquer approach to deduce relationships among inter-query selection conditions on base relations while using only pattern matching to find relationships among higher-order subexpressions (those involving an arbitrary number of joins).

关键词： multiple query optimization multi-query optimization global query optimization common subexpressions database management systems

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：