检索结果-内蒙古大学图书馆

IEEE International Geoscience and Remote Sensing Symposium (IGARSS)

作者： Zhou, Yan Wang, Xu Qin, MengDou Zhang, Cong Univ Elect Sci & Technol China Sch Resources & Environm 2006 Xiyuan Ave Chengdu 611731 Sichuan Peoples R China

ISBN: (纸本)9781665403696

As the volume of spatial data is increasing, distributed storage has become an effective storage management strategy for large amounts of data. The core of distributed storage is the data partitioning method, which partitions massive data onto different nodes evenly. Based on Geohash and Hilbert curve, this paper proposes a spatial data partitioning method of spatial data, which considers the characteristics of spatiotemporal and semantics, and then stores the partitioned spatio-temporal data onto each node to balance the data volume of each node. Experiments show that the spatial data partitioning method proposed in this paper can effectively take into account the balance of data volume, spatio-temporal proximity and semantic similarity. Compared with traditional data partitioning methods, the proposed method achieves the balance of data onto various nodes, and its data blocks have better spatio-temporal proximity and semantic similarity at the same time.

关键词： Spatio-temporal data spatial data partitioning Geohash encoding Hilbert curve

来源：评论

学校读者我要写书评

暂无评论

GIS spatial data partitioning method for distributed data processing

GIS spatial data partitioning method for distributed data pr...

引用

Symposium on Remote Sensing and GIS data Processing and Applications;and Innovative Multispectral Technology and Applications

作者： Zhou, Yan Zin, Qing Zhang, Yeting Wuhan Univ State Key Lab Informat Engn Surveying Mapping & R Wuhan 430079 Hubei Peoples R China

ISBN: (纸本)9780819469540

spatial data partitioning strategy plays an important role in GIS spatial data distributed storage and processing, its key problem is how to partition spatial data to distributed nodes in network environment. Existing main spatial data partitioning methods doesn't consider spatial locality and unstructured variable length characteristics of spatial data, these methods simply partition spatial data based on one or more attributes value that could result in storage capacity imbalance between distributed processing nodes. Aiming at these, we point out the two basic principles that spatial data partitioning should meet to in this paper. We propose a new spatial data partitioning method based on hierarchical decomposition method of low order Hilbert space-filling curve, which could avoid excessively intensive space partitioning by hierarchically decomposing subspaces. The proposed method uses Hilbert curve to impose a linear ordering on the multidimensional spatial objects, and partition the spatial objects according to this ordering. Experimental results show the proposed spatial data partitioning method not only achieves better storage load balance between distributed nodes, but also keeps well spatial locality of data objects after partitioning.

关键词： spatial data partitioning Hilbert space-filling curve hierarchical decomposition distributed data processing

来源：评论

学校读者我要写书评

暂无评论

Learned spatial data partitioning 6

Learned Spatial Data Partitioning

引用

6th International Workshop on Exploiting Artificial Intelligence Techniques for data Management (aiDM)

作者： Hori, Keizo Sasaki, Yuya Amagata, Daichi Murosaki, Yuki Onizuka, Makoto Osaka Univ Suita Osaka Japan

ISBN: (纸本)9798400701931

Due to the significant increase in the size of spatial data, it is essential to use distributed parallel processing systems to efficiently analyze spatial data. In this paper, we first study learned spatial data partitioning, which effectively assigns groups of big spatial data to computers based on locations of data by using machine learning techniques. We formalize spatial data partitioning in the context of reinforcement learning and develop a novel deep reinforcement learning algorithm. Our learning algorithm leverages features of spatial data partitioning and prunes ineffective learning processes to find optimal partitions efficiently. Our experimental study, which uses Apache Sedona and real-world spatial data, demonstrates that our method efficiently finds partitions for accelerating distance join queries and reduces the workload run time by up to 59.4%.

关键词： Reinforcement learning spatial data partitioning Deep Learning

来源：评论

学校读者我要写书评

暂无评论

An Improved Hilbert Curve for Parallel spatial data partitioning

引用

Geo-spatial Information Science 2007年第4期10卷 282-286页

作者： MENG Lingkui HUANG Changqing ZHAO Chunyu LIN Zhiyong School of Remote Sensing and Information Engineering Wuhan University 129 Luoyu Road Wuhan 430079 China

A novel Hilbert-curve is introduced for parallel spatial data partitioning, with consideration of the huge-amount property of spatial information and the variable-length characteristic of vector data items. Based on the improved Hilbert curve, the algorithm can be designed to achieve almost-uniform spatial data partitioning among multiple disks in parallel spatial databases. Thus, the phenomenon of data imbalance can be significantly avoided and search and query efficiency can be enhanced.

关键词： parallel spatial database spatial data partitioning data imbalance Hilbert curve

来源：评论

学校读者我要写书评

暂无评论

spatial coding-based approach for partitioning big spatial data in Hadoop

引用

COMPUTERS & GEOSCIENCES 2017年 106卷 60-67页

作者： Yao, Xiaochuang Mokbel, Mohamed F. Alarabi, Louai Eldawy, Ahmed Yang, Jianyu Yun, Wenju Li, Lin Ye, Sijing Zhu, Dehai China Agr Univ Coll Informat & Elect Engn Beijing 100083 Peoples R China Univ Minnesota Dept Comp Sci & Engn Minneapolis MN 55455 USA Univ Calif Riverside Dept Comp Sci & Engn Riverside CA 92521 USA Minist Land & Resources Land Consolidat & Rehabil Ctr Beijing 100035 Peoples R China Chinese Acad Sci Inst Remote Sensing & Digital Earth Beijing 100094 Peoples R China

spatial data partitioning (SDP) plays a powerful role in distributed storage and parallel computing for spatial data. However, due to skew distribution of spatial data and varying volume of spatial vector objects, it leads to a significant challenge to ensure both optimal performance of spatial operation and data balance in the cluster. To tackle this problem, we proposed a spatial coding-based approach for partitioning big spatial data in Hadoop. This approach, firstly, compressed the whole big spatial data based on spatial coding matrix to create a sensing information set (SIS), including spatial code, size, count and other information. SIS was then employed to build spatial partitioning matrix, which was used to spilt all spatial objects into different partitions in the cluster finally. Based on our approach, the neighbouring spatial objects can be partitioned into the same block. At the same time, it also can minimize the data skew in Hadoop distributed file system (HDFS). The presented approach with a case study in this paper is compared against random sampling based partitioning, with three measurement standards, namely, the spatial index quality, data skew in HDFS, and range query performance. The experimental results show that our method based on spatial coding technique can improve the query performance of big spatial data, as well as the data balance in HDFS. We implemented and deployed this approach in Hadoop, and it is also able to support efficiently any other distributed big spatial data systems.

关键词： spatial coding-based approach Big spatial data spatial data partitioning Hadoop

来源：评论

学校读者我要写书评

暂无评论

Efficient Parallel and Adaptive partitioning for Load-balancing in spatial Join 34

Efficient Parallel and Adaptive Partitioning for Load-balanc...

引用

34th IEEE International Parallel and Distributed Processing Symposium (IPDPS)

作者： Yang, Jie Puri, Satish Marquette Univ Dept Comp Sci Milwaukee WI 53233 USA

ISBN: (纸本)9781728168760

Due to the developments of topographic techniques, clear satellite imagery, and various means for collecting information, geospatial datasets are growing in volume, complexity, and heterogeneity. For efficient execution of spatial computations and analytics on large spatial data sets, parallel processing is required. To exploit fine-grained parallel processing in large scale compute clusters, partitioning in a load-balanced way is necessary for skewed datasets. In this work, we focus on spatial join operation where the inputs are two layers of geospatial data. Our partitioning method for spatial join uses Adaptive partitioning (ADP) technique, which is based on Quadtree partitioning. Unlike existing partitioning techniques, ADP partitions the spatial join workload instead of partitioning the individual datasets separately to provide better load-balancing. Based on our experimental evaluation, ADP partitions spatial data in a more balanced way than Quadtree partitioning and Uniform grid partitioning. ADP uses an output-sensitive duplication avoidance technique which minimizes duplication of geometries that are not part of spatial join output. In a distributed memory environment, this technique can reduce data communication and storage requirements compared to traditional methods. To improve the performance of ADP, an MPI+Threads based parallelization is presented. With ParADP, a pair of real world datasets, one with 717 million polylines and another with 10 million polygons, is partitioned into 65,536 grid cells within 7 seconds. ParADP performs well with both good weak scaling up to 4,032 CPU cores and good strong scaling up to 4,032 CPU cores.

关键词： spatial join spatial data partitioning load-balancing Adaptive partitioning Parallel partitioning MPI-GIS HPC

来源：评论

学校读者我要写书评

暂无评论

A Graph-Based spatial Cross-Validation Approach for Assessing Models Learned with Selected Features to Understand Election Results 20

A Graph-Based Spatial Cross-Validation Approach for Assessin...

引用

20th IEEE International Conference on Machine Learning and Applications (ICMLA)

作者： da Silva, Tiago Pinho Parmezan, Antonio R. S. Batista, Gustavo E. A. P. A. Univ Sao Paulo Sao Carlos Brazil Univ New South Wales Sydney NSW Australia

ISBN: (纸本)9781665443371

Elections are complex activities fundamental to any democracy. The contextualized analysis of election data allows us to understand electoral behavior and the factors that influence it. Multidisciplinary studies have been prioritized the predictive modeling of electoral features from thousands of explanatory features, considering geographic and spatial aspects inherent to the data. When building a model for such a purpose, it must be rigorously evaluated to understand its prediction error in future test cases. Although cross-validation is a widely used procedure for this task, it leads to optimistic results because the spatial independence between test and training data is not ensured in the resampling. On the other hand, alternatives to deal with spatial dependence may fall into a pessimistic scenario by assuming total spatial independence between the test and training sets regardless of the size of the first one, increasing the probability of overfitting. This paper addresses these issues by proposing a graph-based spatial cross-validation approach to assess models learned with selected features from spatially contextualized electoral datasets. Our approach takes advantage of the spatial graph structure provided by the lattice-type spatial objects to define a local training set to each test fold. We generate the local training sets by removing spatially close data that are highly correlated and irrelevant distant data that may interfere with error estimates. Experiments involving the second round of the 2018 Brazilian presidential election demonstrate that our approach contributes to the fair evaluation of models by enabling more realistic and local modeling.

关键词： spatial dependence spatial autocorrelation spatial data partitioning semivariogram prediction error

来源：评论

学校读者我要写书评

暂无评论

Efficient Bundled spatial Range Queries 19

Efficient Bundled Spatial Range Queries

引用

27th ACM SIGspatial International Conference on Advances in Geographic Information Systems (ACM SIGspatial GIS)

作者： Zacharatou, Eleni Tzirita Sidlauskas, Darius Tauheed, Farhan Heinis, Thomas Ailamaki, Anastasia Ecole Polytech Fed Lausanne Lausanne Switzerland Oracle Labs Zurich Zurich Switzerland Imperial Coll London London England RAW Labs SA Lausanne Switzerland

ISBN: (纸本)9781450369091

Efficiently querying multiple spatial data sets is a growing challenge for scientists. Astronomers query data sets that contain different types of stars (e.g., dwarfs, giants, stragglers) while neuroscientists query different data sets that model different aspects of the brain in the same space (e.g., neurons, synapses, blood vessels). The results of each query determine the combination of data sets to be queried next. Not knowing a priori the queried data sets makes it hard to choose an efficient indexing strategy. In this paper, we show that indexing and querying the data sets separately incurs considerable overhead but so does using one index for all data sets. We therefore develop STITCH, a novel index structure for the scalable execution of spatial range queries on multiple data sets. Instead of indexing all data sets separately or indexing all of them together, the key insight we use in STITCH is to partition all data sets individually and to connect them to the same reference space. By doing so, STITCH only needs to query the reference space and follow the links to the data set partitions to retrieve the relevant data. With experiments we show that STITCH scales with the number of data sets and outperforms the state-of-the-art by a factor of up to 12.3.

关键词： spatial Range Query Multiple spatial data Sets spatial Indexing spatial data partitioning spatial data Management

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：