检索结果-内蒙古大学图书馆

作者：蔡凯臻东南大学

学位级别：硕士

随着互联网的飞速发展以及各类新型互联网应用的普及,企业与研究机构面临的数据规模已经高速膨胀到了TB级乃至PB级。近些年来,伴随着内存价格的下降,为了进一步提升大数据处理性能,基于内存的存储与计算逐步成为新的热点。spark作为其... 详细信息

随着互联网的飞速发展以及各类新型互联网应用的普及,企业与研究机构面临的数据规模已经高速膨胀到了TB级乃至PB级。近些年来,伴随着内存价格的下降,为了进一步提升大数据处理性能,基于内存的存储与计算逐步成为新的热点。spark作为其中的代表,以弹性分布式数据集RDD为核心,实现了一个轻量高速可拓展的分布式内存计算框架。然而当前spark上层的高级查询工具spark sql的查询翻译没有针对多查询进行优化,批处理提交sql查询语句被翻译为不同的spark作业,作业之间无法共享数据,无法充分发挥spark内存计算的优势。本文针对spark sql存在的问题,对sql到spark作业查询优化机制进行了研究。本文深入分析了spark sql的工作流程,结合sql查询的特点,在底层持久化文件系统与spark核心间加入存储中间层以解决查询间输入数据的共享问题,并在复用现有分布式文件系统设计的基础上,通过合理的内存资源分配、高效的数据存储结构、低消耗的容错恢复设计优化查询数据的输入过程。在此基础上,引入多查询任务数据管理模块对存储中间层进行管理,通过介入查询任务,实现数据的一次载入、多次使用、及时释放,并针对数据载入存储中间层时面临的数据载入节点选择问题,提出了基于代价模型的数据载入节点选择策略,建立了spark作业执行代价的数学模型,对查询作业的耗费进行预估,基于代价模型选择适当的数据载入节点,实现集群资源的高效利用,提高系统运行的效率,加快查询任务的执行速度。本文在现有spark sql的基础上进行改进,开发了sql2spark系统,实现了上述功能。通过搭建实验平台,使用基准测试工具TPC-H生成测试数据,与现有的spark sql就查询性能进行分析比较。实验结果表明,改进后的sql2spark系统能够有效提高查询速度,减少磁盘I/O开销,降低内存占用,具有明显的优势。

关键词： spark spark sql 查询优化存储中间层

来源：评论

学校读者我要写书评

暂无评论

基于spark平台的支持标准sql语法的数据分析工具

基于Spark平台的支持标准SQL语法的数据分析工具

引用

作者：张超上海交通大学

学位级别：硕士

近年来,大数据领域的技术发展已经吸引了来自学术界和科技产业界甚至开发者社区越来越多的目光。如何针对密集性大数据高效的完成从数据获取到数据处理的一系列步骤,成为了大数据发展的主要挑战之一。谷歌提出的MapReduce框架虽然已经... 详细信息

近年来,大数据领域的技术发展已经吸引了来自学术界和科技产业界甚至开发者社区越来越多的目光。如何针对密集性大数据高效的完成从数据获取到数据处理的一系列步骤,成为了大数据发展的主要挑战之一。谷歌提出的MapReduce框架虽然已经被证明可以解决一部分大数据问题,但是随着更深入的使用,更多的应用则将其处理过程效率低下的缺点暴露无遗。为了解决效率问题,有研究者提出了MapReduce的替代品spark。spark sql作为spark中新加入的模块,为用户提供结构化数据操作接口的同时,还集成了丰富的开发库以供编程人员调用。用户可以借助spark sql完成简单的数据操作,流程较为简便,因此spark sql迅速成为spark中使用范围最为广泛的模块之一。然而spark sql也存在很多的不足之处。它所支持的两种Context:sqlContext和HiveContext,也都或多或少地存在一定问题。其共有的问题主要包括两点。第一,不支持标准的sql语法。用户如果想进行数据操作,需要先花一定的时间去学习它所支持的特殊语法。一些调查表明,几乎所有的大数据研究者或者开发者都具有例如Mysql,Oracle等传统数据库的使用经验。他们对标准sql语法更为熟知,更为希望可以直接在spark平台上直接使用由标准sql语法组成的语句。第二,这两者目前还存在很多功能性上的缺失。用户需要花费很多的时间精力去寻找其他方式来克服这些问题。这些不足都会对开发效率造成影响。本文提出了FlintStone,一个基于spark构建的数据分析计算工具。用户可以借助FlintStone完成众多的数据计算查询操作。与spark sql相比,FlintStone主要有三方面的优势。第一,支持标准sql(sql:1999)语法,用户可以直接利用更为熟悉的语句来完成操作。第二,它支持更多的操作类型和数据类型。第三,在从语句分析到生成spark任务的过程中,FlintStone会做一系列的优化。FlintStone可以被视为一座连通spark与标准DML(数据操作语言)的桥梁。最后,本文将FlintStone提供的FlintContext与spark sql提供的两种Context分别进行了功能和性能上的对比。实验结果表明,在功能上,FlintStone可以支持标准sql语法以及更多的操作类型和数据类型;在性能上,FlintStone同样有一定的提升。在对数据集进行Join类型操作时,FlintStone相比原生spark sql,性能上的提升可以达到10%左右。现阶段FlintStone已经完成了第一个版本的开发,同时已经在github平台上开源并贡献给了spark社区。同时该项目也受到了来自Intel大数据部门专家的支持和维护。

关键词：大数据 Apache spark spark sql 标准sql语法工具

来源：评论

学校读者我要写书评

暂无评论

spark环境下sql优化的方法

引用

数字通信世界 2021年第7期 13-14页

作者：杨彦彬干祯辉中通服软件科技有限公司上海200000 杭州东方通信软件技术有限公司浙江杭州310013

将传统sql稍加改动后迁移到大数据平台是实现业务平稳过渡最佳办法,但是传统sql所处理的数据量、运行环境与大数据平台完全不同,因而迁移后如何优化就很重要了。以当前主流大数据处理组件spark sql为例,论述了迁移产生问题背景、解决方... 详细信息

将传统sql稍加改动后迁移到大数据平台是实现业务平稳过渡最佳办法,但是传统sql所处理的数据量、运行环境与大数据平台完全不同,因而迁移后如何优化就很重要了。以当前主流大数据处理组件spark sql为例,论述了迁移产生问题背景、解决方案,以及优化前后性能比对结果。

关键词： spark 大数据 spark sql

来源：评论

学校读者我要写书评

暂无评论

Fast and scalable inequality joins

引用

VLDB JOURNAL 2017年第1期26卷 125-150页

作者： Khayyat, Zuhair Lucia, William Singh, Meghna Ouzzani, Mourad Papotti, Paolo Quiane-Ruiz, Jorge-Arnulfo Tang, Nan Kalnis, Panos King Abdullah Univ Sci & Technol Thuwal Saudi Arabia HBKU Qatar Comp Res Inst Doha Qatar Arizona State Univ Tempe AZ USA

Inequality joins, which is to join relations with inequality conditions, are used in various applications. Optimizing joins has been the subject of intensive research ranging from efficient join algorithms such as sort-merge join, to the use of efficient indices such as -tree, -tree and Bitmap. However, inequality joins have received little attention and queries containing such joins are notably very slow. In this paper, we introduce fast inequality join algorithms based on sorted arrays and space-efficient bit-arrays. We further introduce a simple method to estimate the selectivity of inequality joins which is then used to optimize multiple predicate queries and multi-way joins. Moreover, we study an incremental inequality join algorithm to handle scenarios where data keeps changing. We have implemented a centralized version of these algorithms on top of Postgresql, a distributed version on top of spark sql, and an existing data cleaning system, Nadeef. By comparing our algorithms against well-known optimization techniques for inequality joins, we show our solution is more scalable and several orders of magnitude faster.

关键词： Inequality join Postgresql spark sql Selectivity estimation Incremental

来源：评论

学校读者我要写书评

暂无评论

Performance Analysis of Big Data ETL Process over CPU-GPU Heterogeneous Architectures 37

Performance Analysis of Big Data ETL Process over CPU-GPU He...

引用

37th IEEE International Conference on Data Engineering (IEEE ICDE)

作者： Lee, Suyeon Park, Sungyong Sogang Univ Seoul South Korea

ISBN: (纸本)9781665448901

While GPUs have been utilized in the analysis stage of big data processing, the demand for GPU use in the extract-transform-load (ETL) stage has recently been increasing. There have been several research efforts to use GPUs for query processing in database systems. However, most of them are mainly focused on batching input data as much as possible to increase the overall throughput with GPUs. Moreover, they try to execute a query plan by only using a single device (only CPU or only GPU), based on the assumption that the PCIe transfer overhead between CPU and GPU is significant. In contrast, this paper presents several interesting observations and reveals that it is better to use a CPU-GPU heterogeneous query plan in some cases. For example, the heterogeneous query plan proved to be efficient when the input data size is small. This is because the PCIe overhead is not as serious as expected for small-sized data and can be offset with performance gains by partially using the GPU. In addition, even when the data size is large, it is sometimes better to use the CPU-GPU heterogeneous query plan rather than only the GPU plan, since some relational operations prefer the CPU over the GPU. To demonstrate these uses, this paper presents several experimental results and their analyses with spark sql in a CPU-GPU heterogeneous computing environment.

关键词： Big Data ETL spark sql GPU

来源：评论

学校读者我要写书评

暂无评论

Simba: Efficient In-Memory Spatial Analytics 16

Simba: Efficient In-Memory Spatial Analytics

引用

ACM SIGMOD International Conference on Management of Data

作者： Xie, Dong Li, Feifei Yao, Bin Li, Gefei Zhou, Liang Guo, Minyi Univ Utah Salt Lake City UT 84112 USA Shanghai Jiao Tong Univ Shanghai Peoples R China

ISBN: (纸本)9781450335317

Large spatial data becomes ubiquitous. As a result, it is critical to provide fast, scalable, and high-throughput spatial queries and analytics for numerous applications in location-based services (LBS). Traditional spatial databases and spatial analytics systems are disk based and optimized for IO efficiency. But increasingly, data are stored and processed in memory to achieve low latency, and CPU time becomes the new bottleneck. We present the Simba (Spatial In-Memory Big data Analytics) system that offers scalable and efficient in-memory spatial query processing and analytics for big spatial data. Simba is based on spark and runs over a cluster of commodity machines. In particular, Simba extends the spark sql engine to support rich spatial queries and analytics through both sql and the DataFrame API. It introduces indexes over RDDs in order to work with big spatial data and complex spatial operations. Lastly, Simba implements an effective query optimizer, which leverages its indexes and novel spatial-aware optimizations, to achieve both low latency and high throughput. Extensive experiments over large data sets demonstrate Simba's superior performance compared against other spatial analytics system.

关键词： big spatial data spark spark sql main-memory DBMS spatial queries spatial analytics in-memory computing

来源：评论

学校读者我要写书评

暂无评论

Low Cost Big Data Solutions: The Case of Apache spark on Beowulf Clusters 5

Low Cost Big Data Solutions: The Case of Apache Spark on Beo...

引用

5th International Conference on Internet of Things, Big Data and Security (IoTBDS)

作者： Fotache, Marin Cluci, Marius-Iulian Greavu-Serban, Valerica Alexandru Ioan Cuza Univ Iasi Romania

ISBN: (纸本)9789897584268

With distributed computing platforms deployed on affordable hardware, Big Data technologies have democratised the processing of huge volumes of structured and semi-structured data. Still, the costs of installing and operating even relatively small cluster of commodity servers or the cost of hiring cloud resources could prove inaccessible for many companies and institutions. This paper builds two predictive models for estimating the main drivers of the data processing performance for one of the most popular Big Data system (Apache spark) deployed on gradually increased number of nodes of a Beowulf cluster. Data processing performance was estimated by randomly generated sparksql queries on TPC-H database schema, with variable number of joins (including self-joins), predicates, groups, aggregate functions and subqueries included in FROM clause. Using two machine learning techniques, random forest and extreme gradient boosting, predictive models tried to estimate the query duration on predictors related to cluster setup and query structure and also to assess the importance of predictors for the outcome variability. Results were positive and encouraging for extending the cluster number of nodes and the database scale.

关键词： Big Data Beowulf Clusters Apache spark spark sql Machine Learning Distributed Computing TCP-H

来源：评论

学校读者我要写书评

暂无评论

Relational Schemata for Distributed SPARQL Query Processing

Relational Schemata for Distributed SPARQL Query Processing

引用

International Workshop on Semantic Big Data (SBD)

作者： Arrascue, Victor Anthony A. Koleva, Polina Alzogbi, Anas Cossu, Matteo Faerber, Michael Philipp, Patrick Schievelbein, Guilherme Taxidou, Io Lausen, Georg Univ Freiburg Freiburg Germany Ultra Tendency GmbH Niels Bohr Str 10c D-39106 Magdeburg Germany FZI Forschungszentrum Informat KIT Haid Und Neu Str 10-14 D-76131 Karlsruhe Germany

ISBN: (纸本)9781450367660

To benefit from mature database technology RDF stores are built on top of relational databases and SPARQL queries are mapped into sql. Using a shared-nothing computer cluster is a way to achieve scalability by carrying out query processing on top of large RDF datasets in a distributed fashion. Aiming to this the current paper elaborates on the impact of relational schema design when queries are mapped into Apache spark sql. A single triple table, a set of tables resulting from partitioning by predicate, a single wide table covering all properties, and a set of tables based on the application model specification called domain-dependent-schema, are the considered designs. For each of the mentioned approaches, the rows of the corresponding tables are stored in the distributed file system HDFS using the columnar-store Parquet. Experiments using standard benchmarks demonstrate that the single wide property table approach, despite its simplicity, is superior to other approaches. Further experiments demonstrate that this single table approach continues to be attractive even when repartitioning by key (RDF subject) is applied before executing queries.

关键词： Relational Schema RDF SPARQL spark sql Parquet

来源：评论

学校读者我要写书评

暂无评论

When Rule Engine meets Big Data: Design and Implementation of a Distributed Rule Engine using spark 3

When Rule Engine meets Big Data: Design and Implementation o...

引用

3rd IEEE International Conference on Big Data Computing Service and Applications (BigDataService)

作者： Zhang, Jindou Yang, Jinxing Li, Jing Univ Sci & Technol China Sch Comp Sci & Technol Hefei 230026 Anhui Peoples R China

ISBN: (纸本)9781509063185

Rule Engines have been widely used both in industry and academia since they can separate rule knowledge from implementation logic conveniently and flexibly. However, traditional rule engine systems can not deal with big data, because of the limitations of memory and computing capacity of one single computer. Consequently, some researchers have proposed distributed rule engine to meet this challenge. But these solutions still do not work well when enormous amounts of facts are involved, for the reasons such as high cost of moving data, data imbalance and so on. We present the sparkRE system, a distributed rule engine based on spark, to support big data reasoning. In particular, sparkRE uses DataFrame representing distributed working memory which holds enormous amounts of facts, and implements the rule condition-testing mechanism by the query execution engine of spark sql. In addition, we design a rule language tailored for sparkRE and an efficient inference engine. An experimental evaluation shows sparkRE's capability of matching 26 million facts, and reveals its scalability and performance.

关键词： Big Data Inference engine Rule Engine Rule language spark spark sql

来源：评论

学校读者我要写书评

暂无评论

Unique Topic Query Processing On Cloud 5

Unique Topic Query Processing On Cloud

引用

5th IEEE International Conference on Cyber Security and Cloud Computing (CSCloud) / 4th IEEE International Conference on Edge Computing and Scalable Cloud (EdgeCom)

作者： Liu, Lvhong Yang, Zhihui He, Zhenying Jing, Yinan Wang, Xiaoyang Sean Fudan Univ Software Sch Shanghai Peoples R China Fudan Univ Sch Comp Sci Shanghai Peoples R China

ISBN: (纸本)9781538658505

With the rapid growth of text data, text analysis plays an increasingly important role for various applications. Tranditional database management systems provide basic data operations for various users. But most of them only provide data access and basic calculation for relational datasets, we can not perform text analysis operations on them directly. To support users analyzing unstructured text, this paper designs a set of sql-like information extraction and unique topic query language for users to extract information and find out unique topics on unstructured data. This paper proposes a unique topic query processing system on cloud based on relational information extraction with spark as cloud infrastructure.

关键词： unique topic query information extraction text analysis spark sql cloud service

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：