检索结果-内蒙古大学图书馆

IEEE 28th International Conference on Parallel and distributed systems (IEEE ICPADS)

作者： Wang, Qihang Zuo, Decheng Zhang, Zhan Chen, Siyuan Liu, Tianming Harbin Inst Technol Dept Comp Sci & Technol Harbin Peoples R China

ISBN: (纸本)9781665473156

In the field of real-time analytics, stream joins are the basis for complex queries and greatly affect system performance. In order to satisfy the real-time requirements of streaming applications, the system imposes high requirements on the latency and throughput of the stream join operator. In this paper, we model the latency and throughput of distributed stream join systems based on queuing theory. Based on the analysis of this model, we demonstrate the impact of indexing-related overhead on the latency and throughput of stream join systems and propose a new distributed stream join system, Sepjoin, which is oriented to the hash join problem. Sepjoin reduces the number of tuples stored in each processing unit belonging to each input stream by designing a novel partitioning scheme that uses as many processing units as possible to store tuples belonging to each input stream, thereby reducing the index-related overhead of each processing unit when performing join operations and ultimately achieving performance benefits in terms of latency and throughput. We provide both theoretical analysis and extensive experimental evaluations to evaluate the processing latency and max throughput of Sepjoin.

关键词： big data distributed stream join system queuing theory partitioning scheme

来源：评论

学校读者我要写书评

暂无评论

An adaptive non-migrating load-balanced distributed stream window join system

引用

JOURNAL OF SUPERCOMPUTING 2023年第8期79卷 8236-8264页

作者： Wang, Qihang Zuo, Decheng Zhang, Zhan Chen, Siyuan Liu, Tianming Harbin Inst Technol Fac Comp Harbin 150001 Peoples R China

stream processing systems are widely used to process large amounts of data generated by applications in real time due to their advantages in latency and throughput. In most streaming applications, the system requires a comprehensive analysis of data from multiple data sources, so stream joins are the basis of stream processing systems. Similar to other big data problems, stream joins suffer from load imbalance, where a few nodes responsible for handling most of the load can become bottlenecks, thereby increasing latency and reducing throughput. Therefore, how to obtain a good load-balancing effect with low overhead is a critical issue in designing stream join systems. To solve this problem, we propose an adaptive non-migrating load-balancing method, which is mainly oriented to the stream window join problem. Considering that the completeness of the stream join results during the splitting of state to multiple downstream instances can be guaranteed by replicating the input tuples into multiple replicas and sending them to those downstream instances, our method can control the replication and forwarding of input tuples by setting up routing tables, and then when the system becomes unbalanced, our method can change the load distribution of the system by directly changing the partitioning of the tuples arriving later instead of state migration, and thus achieving load balancing with very low overhead. Based on our method, we develop a distributed stream window join system, NM-join, which is built on Flink. We theoretically analyze the completeness and effectiveness of our method and provide extensive experimental evaluations of NM-join in terms of load-balancing effect, latency, and throughput. Experimental results show that our method is able to perform load balancing with very low additional overhead, and thus outperforms existing load-balancing methods in terms of latency and throughput.

关键词： Big data distributed stream join system Data skew Dynamic load balancing Non-migrating

来源：评论

学校读者我要写书评

暂无评论

Simois: A Scalable distributed stream join system with Skewed Workloads 39

Simois: A Scalable Distributed Stream Join System with Skewe...

引用

39th IEEE International Conference on distributed Computing systems (ICDCS)

作者： Zhang, Fan Chen, Hanhua Jin, Hai Huazhong Univ Sci & Technol Natl Engn Res Ctr Big Data Technol & Syst Serv Comp Technol & Syst Lab Cluster & Grid Comp LabSch Comp Sci & Technol Wuhan 430074 Peoples R China

ISBN: (纸本)9781728125190

Many Bigllata applications require to perform quick join operations on different large-scale real-time data streams. The key challenge to design an efficient stream join system is how to reasonably partition the streaming data among distributed processing nodes to avoid high density of join computation. However, the skewed distribution of real world streams raises great challenges for streaming data partitioning in distributed stream join systems. Existing hash based partitioning schemes incur significant load imbalance which leads to low system throughput and long processing latency, while shuffling based strategies incur redundant join computation and much more communication. To address this issue, in this paper, we propose and implement a scalable distributed stream join system, Simois, which shuffles the potential top heavy-load keys while hashing the others. However, how to identify the keys which lead to the heavy workload imbalance is challenging, because the heavy workload is determined by the current joint status of two streams, and the distribution of the two streams may change with time. To solve this problem, we design a novel efficient exponential counting scheme for identifying the keys with the heaviest workload in the two dynamic streams. The proposed exponential counting scheme needs extremely low computation and space cost, so that it can be well implemented in a stream processing system. Moreover, we design a popularity decline algorithm to make our design adaptive to the highly dynamic changes of streams. We implement Simois on top of Apache Storm and conduct comprehensive experiments using large-scale real world traces. Experiment results show that Simois improves the system throughput significantly by 52% and reduces the average latency by 37%, compared to existing state-of-the-art designs.

关键词： distributed stream join system skew distribution exponential counting

来源：评论

学校读者我要写书评

暂无评论

Fastjoin: A Skewness-Aware distributed stream join system 33

FastJoin: A Skewness-Aware Distributed Stream Join System

引用

33rd IEEE International Parallel and distributed Processing Symposium (IPDPS)

作者： Zhou, Shunjie Zhang, Fan Chen, Hanhua Jin, Hai Zhou, Bing Bing Huazhong Univ Sci & Technol Natl Engn Res Ctr Big Data Technol & Syst Cluster & Grid Comp Lab Serv Comp Technol & Syst LabSch Comp Sci & Techn Wuhan 430074 Peoples R China Univ Sydney Sch Informat Technol Sydney NSW 2006 Australia

ISBN: (纸本)9781728112466

In the bigdata era, many applications are required to perform quick and accurate join operations on large-scale real-time data streams, such as stock trading and online advertisement analysis. To achieve high throughput and low latency, distributed stream join systems explore efficient stream partitioning strategies to execute the complex stream join procedure in parallel. Existing systems mainly deploy two kinds of partitioning strategies, i.e., random partitioning and hash partitioning. Random partitioning strategy partitions one data stream uniformly while broadcasting all the tuples of the other data stream. This simple strategy may incur lots of unnecessary computations for low selectivity stream join. Hash partitioning strategy maps all the tuples of the two data streams according to their attributes for joining. However, hash partitioning strategy suffers from a serious load imbalance problem caused by the skew distribution of the attributes, which is common in real-world data. The skewed load may seriously affect the system performance. In this paper, we carefully model the load skewness problem in distributed join systems. We explore the key tuples which lead to the heavy load skewness, and propose an efficient key selection algorithm, GreedyFit to find out these key tuples. We design a lightweight tuple migration strategy to solve the load imbalance problem in real-time and implement a new distributed stream join system, &Wain. Experimental results using real-world data show that Fastjoin can significantly improve the system performance in terms of throughput and latency compared to the slate-of-the-art stream join systems.

关键词： distributed stream join system data skew dynamic load balancing load migration

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：