检索结果-内蒙古大学图书馆

8th IEEE International Conference on Big Data (Big Data)

作者： Escobar, Otoniel Jose Campos Misev, Dimitar Baumann, Peter Jacobs Univ Bremen Bremen Germany Rasdaman GmbH Bremen Germany

ISBN: (纸本)9781728162515

Server-side extensibility through dynamically linked external code is a common method in relational databases. In the field of array databases such User-Defined Functions (UDFs) sometimes even represent the architectural cornerstone for array functionality. On the downside, UDF implementation often suffers from high coding complexity. The rasdaman array DBMS is a full-stack C++ implementation, so does not rely on some generic UDF mechanism. This allowed designing such an API from scratch, with particular emphasis on UDF coder convenience. The rasdaman UDFs rely on the general C++ client API classes. Based on a straightforward UDF interface definition adapter code is generated automatically. Experimental evaluation shows encouraging results, and the mechanism is going to be used in research and under operational conditions. We present the approach and motivate it through practical use cases.

关键词： array database user-defined functions rasdaman

来源：评论

学校读者我要写书评

暂无评论

Scalable analysis of multitemporal images using an array database

Scalable analysis of multitemporal images using an array dat...

引用

作者： Joshi, Abhasha Universidade Nova de Lisboa

学位级别：硕士

Massive archives of earth observation data are now available and the size of this data is increasing at a tremendous rate. This data is a very important resource and has a variety of applications including monitoring change, forestry application, agricultural application and urban planning. At the same time, they also possess challenge of storage, management, and high computational needs. In this study SciDB, an array-based database is used to store, manage and process multitemporal satellite imagery. The major aim of this study is to investigate the performance of SciDB based scalable solution to run arithmetic operation, simple time series analysis and complex time series analysis on multitemporal satellite imagery. This study provides better insight of SciDB architecture and provides suggestions for better performance in SciDB for remote sensing jobs. The research also compared the performance of time series analysis on SciDB array with file-based analysis using multicore parallelization (Using „Parallel‟ Package of R). It is found that SciDB provides a faster solution for time series analysis. However, SciDB might not be the best solution if the data size is smaller. Also, relative immaturity of SciDB and limited inherent support of remote sensing operations increases effort for the scientist to develop SciDB based solution. Nevertheless, SciDB has the potential to meet the ever increasing storage, management and computational need of big remote sensing data.

关键词： array database SciDB High Performance Computing Remote Sensing Multitemporal Images

来源：评论

学校读者我要写书评

暂无评论

Running scientific algorithms as array database operators: Bringing the processing power to the data 4

Running scientific algorithms as array database operators: B...

引用

4th IEEE International Conference on Big Data (Big Data)

作者： Marcin, Simon Csillaghy, Andre Univ Appl Sci North Western Switzerland FHNW Inst Technol & DataSpaces 4D CH-5210 Windisch Switzerland

ISBN: (纸本)9781467390057

array databases are well suited for storing and processing large multidimensional data. However, they usually run rather simple operations which only represent single steps of scientific algorithms. A way to run more complex logic is needed. In this paper, we study and test how to run entire scientific algorithms as native array database operators in a SciDB cluster. We present as use case our implementation of an iterative algorithm that reconstructs the distribution of plasma density in the solar corona at specific temperatures. This algorithm uses images series of the NASA spacecraft Solar Dynamic Observatory (SDO), which we stack in a 3-dimensional array. We measure for our use case a decrease of the overall runtime by an order of magnitude. We discuss different parameters used to scale up the array database cluster.

关键词： algorithms array database database applications performance cluster

来源：评论

学校读者我要写书评

暂无评论

FASTDB: An array database System for Efficient Storing and Analyzing Massive Scientific Data 15th

FASTDB: An Array Database System for Efficient Storing and A...

引用

15th International Conference on Algorithms and Architectures for Parallel Processing (ICA3PP)

作者： Li, Hui Qiu, Nengjun Chen, Mei Li, Hongyuan Dai, Zhenyu Zhu, Ming Huang, Menglin Guizhou Univ Dept Comp Sci Guiyang 550025 Peoples R China ACMIS Guizhou Engn Lab Guiyang 550025 Peoples R China Chinese Acad Sci Natl Astron Observ Beijing 100016 Peoples R China

ISBN: (纸本)9783319271613;9783319271606

With the development of science and technology, the data size and complexity of scientific data are increased rapidly, which made efficient data storage and parallel analysis of scientific data become a big challenge. The previous techniques that combine the traditional relational database with analysis software tends cannot efficiently meet the performance requirement of large scale scientific data based analysis. In this paper, we present FASTDB, a distributed array database system that optimized for massive scientific data management and provide a share-nothing, parallel array processing analysis. In order to demonstrate the intrinsic performance characteristics of FASTDB, we applied it into the interactive analysis of data from astronomical surveys, and designed a series of experiments with scientific analysis tasks. According to the experimental results, we found FASTDB can be significantly fast than traditional database based SkyServer in many typical analytical scenarios.

关键词： Scientific database system array database Scientific analysis Performance measurement Massive data

来源：评论

学校读者我要写书评

暂无评论

array-database Model (SciDB) for Standardized Storing of Hyperspectral Satellite Images

Array-database Model (SciDB) for Standardized Storing of Hyp...

引用

作者： Hausen, Eias Universidade Nova de Lisboa

学位级别：硕士

Hyperspectral Imaging is a technique that collects information from the electromagnetic spectrum, storing the value of the spectrum band for each pixel of the image. This technique stands out for the contiguous wide range of wavelengths it covers; leading to the ability of accurate surface and material distinction. The big volumes of Hyperspectral Images datasets, which are called data cubes as the band value represent the third dimension, have been a barrier against exploiting the full potential of these images where there is no standardized way in storing them. On top of that, the classical relational databases proved to be an inconvenient storage space for such *** databases have been a serious choice for storing scientific and big volumes of data, and they represent a promising suitable environment for hyperspectral images. We aim to study the efficiency of storing hyperspectral images on an array-database by suggesting a convenient data model. Furthermore, in order to examine the feasibility of this model, we make a comparison with two relational databases using specific measurements in performance and query complexity.

关键词： array database database Comparison GeoTIFF Images Hyperspectral Imaging Hyperspectral Satellite Images Raster database Satellite Images Storage SciDB

来源：评论

学校读者我要写书评

暂无评论

DyVer: Dynamic Version Handling for array databases 23

DyVer: Dynamic Version Handling for Array Databases

引用

37th ACM International Conference on Supercomputing (ACM ICS)

作者： Zhou, Amelie Chi Ke, Zhoubin Lao, Jianming Shenzhen Univ Shenzhen Peoples R China Univ Calif Irvine Irvine CA USA

ISBN: (纸本)9798400700569

array databases are important data management systems for scientific applications. In array databases, version handling is an important problem due to the no-overwrite feature of scientific data. Existing studies for optimizing data versioning in array databases are relatively simple, which either focus on minimizing storage sizes or improving simple version chains. In this paper, we focus on two challenges: (1) how to balance the tradeoff between storage size and query time for numerous version data, which may have derivative relationships with each other;(2) how to dynamically maintain this balance with continuously added new versions. To address the above challenges, this paper presents DyVer, a versioning framework for SciDB which is one of the most well-known array databases. DyVer includes two techniques, including an efficient storage layout optimizer to quickly reduce data query time under storage capacity constraint and a version segment technique to cope with dynamic version additions. We evaluate DyVer using real-world scientific datasets. Results show that DyVer can achieve up to 95% improvement on the average query time compared to state-of-the-art data versioning techniques under the same storage capacity constraint.

关键词： Scientific Data Management array database Versioning

来源：评论

学校读者我要写书评

暂无评论

Incremental Window Aggregates over array database 2

Incremental Window Aggregates over Array Database

引用

IEEE International Conference on Big Data

作者： Jiang, Li Kawashima, Hideyuki Tatebe, Osamu Univ Tsukuba Grad Sch Syst & Informat Engn Tsukuba Ibaraki Japan Univ Tsukuba Fac Informat Syst & Engn Tsukuba Ibaraki Japan

ISBN: (纸本)9781479956661

We propose an efficient window aggregation method over multi-dimensional array data based on incremental computation. We improve several aggregations with different data structures exploited to achieve efficient computation: list for sum and avg, heap for max and min, and balanced binary search tree for percentile. We present time complexity analysis for the methods, and then evaluate performance with experiments in SciDB array database system with both synthetic and JRA55 meteorological dataset. Our analysis shows that performance improvement is proportional to the window size in the last dimension in theory, and the result of experiment is consistent with the analysis. In certain cases, it shows an acceleration factor more than 13 by the proposed method with percentile, while a factor over 28 with maximum.

关键词： Window Aggregates Multi-Dimensional array Incremental Computation array database

来源：评论

学校读者我要写书评

暂无评论

ADOPTING DATACUBES AS A SERVICE FOR BUILDING THE TAIWAN NATIONAL FEDERATED RASTER DATA REPOSITORY

ADOPTING DATACUBES AS A SERVICE FOR BUILDING THE TAIWAN NATI...

引用

IEEE International Geoscience and Remote Sensing Symposium (IGARSS)

作者： Hao, Chen-Yu Shih, Yi-Liang Chang, Jo-Yu Feng Chia Univ Taichung Taiwan Natl Ctr High Performance Comp Hsinchu Taiwan

ISBN: (数字)9781665427920

ISBN: (纸本)9781665427920

Recently, the technologies development on Earth observation has significantly improved the efficiency to retrieve large amount of image data. It well resolved the difficulties on collecting enough analysis material, however, the growing effort on image pre-processing, calculating, and storing generates separated issues. Datacube, the concept of large-scale data warehouse has been considered as the solution for solving the problems. In this study, instead of adapting a Datacube software/solution directly, we introduced a new perspective that utilizes Datacube as the core onto a cloud infrastructure and become a package from NCHC (National Center for High Performance Computing) PaaS. This solution is significantly recognized by various governmental agencies and academic institutions.

关键词： Datacube array database federation

来源：评论

学校读者我要写书评

暂无评论

Integrating memory-mapping and N-dimensional hash function for fast and efficient grid-based climate data query

引用

ANNALS OF GIS 2021年第1期27卷 57-69页

作者： Xu, Mengchao Zhao, Liang Yang, Ruixin Yang, Jingchao Sha, Dexuan Yang, Chaowei George Mason Univ Dept Geog & Geoinformat Sci Fairfax VA 22030 USA George Mason Univ Dept Informat Sci & Technol Fairfax VA 22030 USA

database systems are pervasive components in the current big data era. However, efficiently managing and querying grid-based or array-based multidimensional climate data are still beyond the capabilities of most databases. The mismatch between the array data model and relational data model limited the performance to query multidimensional data in a traditional database when data volume hits a cap. Even a trivial data retrieval on large multidimensional datasets in a relational database is time-consuming and requires enormous storage space. Given the scientific interests and application demands on time-sensitive spatiotemporal data query and analysis, there is an urgent need for efficient data storage and fast data retrieval solutions on large multidimensional datasets. In this paper, we introduce a method for multidimensional data storing and accessing, which includes a new hash function algorithm that works on a unified data storage structure and couples with the memory-mapping technology. A prototype database library, LotDB developed as an implementation, is described in this paper, which shows promising results on data query performance compared with SciDB, MongoDB, and PostgreSQL.

关键词： Multidimensional array gridded data array database memory-mapping MERRA data cube

来源：评论

学校读者我要写书评

暂无评论

Method for managing and querying geo-spatial data using a grid-code-array spatial index

引用

EARTH SCIENCE INFORMATICS 2019年第2期12卷 173-181页

作者： Li, Shuang Pu, Guoliang Cheng, Chengqi Chen, Bo Peking Univ Inst Remote Sensing & GIS Beijing 100871 Peoples R China Peking Univ Coll Engn Beijing 100871 Peoples R China

As geospatial data is increasingly massive and complex, large data volume and rich data source query retrieval are among the urgent issues in need of resolution. Spatial indices are widely used to organize data and optimize queries. However, tree-based indices are increasingly difficult to adapt to a high-efficiency query, and the combination of a grid index and space-filling curve can help decrease the dimensions to improve the query efficiency, but can also lead to data redundancy as one object can cover several grids. To solve the aforementioned problems, this paper proposes a method to manage and query data using a grid-code-array spatial index based on a GeoSOT global subdivision model. For the first time, a grid code was organized in a code-array format and an inverted index was constructed on the column of the code-array. By adding a grid-code-array data structure, we verified the feasibility and efficiency and compared the R-tree index in the Oracle Spatial system and the grid index in the ArcSDE geodatabase for Oracle, which are the most widely used. Experimental results showed that the spatial index we proposed has obvious advantages, which could solve the problem of storage redundancy and query results, and effectively improve spatial queries particularly when the data volume is large.

关键词： GeoSOT array database Global subdivision Spatial index Inverted index

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：