检索结果-内蒙古大学图书馆

Infrastructures and services for remote sensing data production management across multiple satellite data centers

CLUSTER computing-THE JOURNAL OF NETWORKS SOFTWARE TOOLS AND APPLICATIONS 2016年第3期19卷 1243-1260页

作者： Zhang, Jie Yan, Jining Ma, Yan Xu, Dong Li, Pengfei Jie, Wei Chinese Acad Sci Inst Remote Sensing & Digital Earth Beijing Peoples R China Univ Chinese Acad Sci Beijing Peoples R China Yanshan Univ Inst Informat Sci & Engn Qinhuangdao Hebei Peoples R China Univ West London Sch Comp & Engn London England

With the number of satellite sensors and date centers being increased continuously, it is becoming a trend to manage and process massive remote sensing data from multiple distributed sources. However, the combination of multiple satellite data centers for massive remote sensing (RS) data collaborative processing still faces many challenges. In order to reduce the huge amounts of data migration and improve the efficiency of multi-datacenter collaborative process, this paper presents the infrastructures and services of the data management as well as workflow management for massive remote sensing data production. A dynamic data scheduling strategy was employed to reduce the duplication of data request and data processing. And by combining the remote sensing spatial metadata repositories and Gfarm grid file system, the unified management of the raw data, intermediate products and final products were achieved in the co-processing. In addition, multi-level task order repositories and workflow templates were used to construct the production workflow automatically. With the help of specific heuristic scheduling rules, the production tasks were executed quickly. Ultimately, the Multi-datacenter Collaborative Process System (MDCPS) were implemented for large-scale remote sensing data production based on the effective management of data and workflow. As a consequence, the performance of MDCPS in experiments environment showed that those strategies could significantly enhance the efficiency of co-processing across multiple data centers.

关键词： Multi-datacenter infrastructure Remote sensing data processing Distributed computing big data computing data management Workflow management

来源：评论

学校读者我要写书评

暂无评论

A Novel Pipeline Approach for Efficient big data Broadcasting

引用

IEEE TRANSACTIONS ON KNOWLEDGE AND data ENGINEERING 2016年第1期28卷 17-28页

作者： Wu, Chi-Jen Ku, Chin-Fu Ho, Jan-Ming Chen, Ming-Syan Natl Taiwan Univ Dept Elect Engn Taipei 10617 Taiwan Acad Sinica Inst Informat Sci Taipei 11529 Taiwan Acad Sinica Res Ctr Informat Technol Innovat Taipei 11529 Taiwan

big-data computing is a new critical challenge for the ICT industry. Engineers and researchers are dealing with data sets of petabyte scale in the cloud computing paradigm. Thus, the demand for building a service stack to distribute, manage, and process massive data sets has risen drastically. In this paper, we investigate the big data Broadcasting problem for a single source node to broadcast a big chunk of data to a set of nodes with the objective of minimizing the maximum completion time. These nodes may locate in the same datacenter or across geo-distributed datacenters. This problem is one of the fundamental problems in distributed computing and is known to be NP-hard in heterogeneous environments. We model the big-data broadcasting problem into a Lock-Step Broadcast Tree (LSBT) problem. The main idea of the LSBT model is to define a basic unit of upload bandwidth, r, such that a node with capacity c broadcasts data to a set of left perpendicular c/r right perpendicular children at the rate r. Note that r is a parameter to be optimized as part of the LSBT problem. We further divide the broadcast data into m chunks. These data chunks can then be broadcast down the LSBT in a pipeline manner. In a homogeneous network environment in which each node has the same upload capacity c, we show that the optimal uplink rate r* of LSBT is either c/2 or c/3, whichever gives the smaller maximum completion time. For heterogeneous environments, we present an O(nlog(2)) algorithm to select an optimal uplink rate r* and to construct an optimal LSBT. Numerical results show that our approach performs well with less maximum completion time and lower computational complexity than other efficient solutions in literature.

关键词： big data computing data delivery algorithm cloud computing distributed computing big data management

来源：评论

学校读者我要写书评

暂无评论

引用

CONCURRENCY AND COMPUTATION-PRACTICE & EXPERIENCE 2017年第4期29卷

作者： Subramanian, Sabitha Malli Soundarajan, Vijayalakshmi Bharathiar Univ Res & Dev Ctr Coimbatore 641046 Tamil Nadu India Thiagarajar Coll Engn Dept Comp Applicat Madurai 625015 Tamil Nadu India

big data is a new term used to identify the large scale and complex datasets. big data is now rapidly expanding in all science and engineering domains, owing to the fast development of networking, data storage, and data collection capacity. big data mining is the capability of extracting useful information from these large datasets. Nowadays, the integration of cloud computing with the data mining for the big data mining process is a challenging task. In order to process the huge amount of data, it is necessary to concentrate the improvement on the big data computation. Most of the existing approaches use the MapReduce to compute the big data. The increase in the computational cost and memory consumption are the main drawbacks of the existing approaches. To overcome these limitations, this paper proposes a similarity-based clustering and optimum cache replacement approach for big data computing applications. The job recovery process is initiated by copying the data in the cloud server and forwarding the data copy for further processing. Then, the job is divided into clusters based on the similarity-based clustering approach. Finally, the cache concept is introduced with the optimum cache replacement algorithm to avoid repeated execution of the jobs by queue management. The proposed approach is compared with the existing Spark and Hadoop approaches. The proposed approach achieves better performance in terms of iteration time, query response time, job completion time, and clustering accuracy. Copyright (C) 2016 John Wiley & Sons, Ltd.

关键词： big data computing data cleaning Hadoop MapReduce optimum cache replacement (OCR) algorithm similarity-based clustering (SC)

来源：评论

学校读者我要写书评

暂无评论

NFC For Dummies 1

引用

丛书名： -For dummies

2016年

作者： Robert P. Sabella

ISBN: (数字)9781119182931

ISBN: (纸本)9781119182924

Your no-nonsense guide to Near Field Communication Are you a newcomer to Near Field Communication and baffled by the scant documentation and online support available for this powerful new technology? You've come to the right place! Written in a friendly and easily accessible manner, NFC For Dummies takes the intimidation out of working with the features of NFC-enabled devices and tells you exactly what it is and what it doesand doesn't do. NFC is revolutionizing the way people interact on a daily basis. It enables big data and cloud-based computing through mobile devices and can be used by anyone with a smartphone or tablet every day! Soon to be as commonplace as using Wi-Fi or the camera on your smartphone, NFC is going to forever change the way we interact with people and the things around us. It simplifies the sending and receiving of information, makes monetary transactions simple and secureApple Pay already uses NFCand is a low-cost product to manufacture and use. As more developers create apps with NFC, you're going to see it used regularlyeverywhere from cash registers to your social media accounts to electronic identity systems. Don't get left behind; get up to speed on NFC today! Provides a plain-English overview of NFC Covers the history and technology behind NFC Helps you make sense of IoT and powered chips Explains proximity technologies and non-payment applications Whether you're a developer, investor, or a mobile phone user who is excited about the capabilities of this rapidly growing technology, NFC For Dummies is the reference you'll want to keep close at hand!

关键词： NFC For Dummies Robert Sabella Robert P. Sabella Near Field Communication big data cloud-based computing big data computing NFC cloud-based computing through mobile cloud-based computing through mobile devices overview of NFC history of NFC technology behind NFC IoT and powered chips proximity technology proximity technologies non-payment applications NFC book NFC books book on NFC books on NFC features of NFC-enabled devices NFC-enabled devices

来源：评论

学校读者我要写书评

暂无评论

Parallel Processing of Dynamic Continuous Queries over Streaming data Flows

引用

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS 2015年第3期26卷 834-846页

作者： Deng, Ze Wu, Xiaoming Wang, Lizhen Chen, Xiaodao Ranjan, Rajiv Zomaya, Albert Chen, Dan China Univ Geosci Sch Comp Sci Wuhan 430074 Peoples R China Chinese Acad Sci Inst Remote Sensing & Digital Earth Beijing Peoples R China CSIRO ICT Ctr Computat Informat Div Sydney NSW Australia Univ Sydney Sch Informat Technol Sydney NSW 2006 Australia Collaborat & Innovat Ctr Educ Technol Beijing Peoples R China

More and more real-time applications need to handle dynamic continuous queries over streaming data of high density. Conventional data and query indexing approaches generally do not apply for excessive costs in either maintenance or space. Aiming at these problems, this study first proposes a new indexing structure by fusing an adaptive cell and KDB-tree, namely CKDB-tree. A cell-tree indexing approach has been developed on the basis of the CKDB-tree that supports dynamic continuous queries. The approach significantly reduces the space costs and scales well with the increasing data size. Towards providing a scalable solution to filtering massive steaming data, this study has explored the feasibility to utilize the contemporary general-purpose computing on the graphics processing unit (GPGPU). The CKDB-tree-based approach has been extended to operate on both the CPU (host) and the GPU (device). The GPGPU-aided approach performs query indexing on the host while perform streaming data filtering on the device in a massively parallel manner. The two heterogeneous tasks execute in parallel and the latency of streaming data transfer between the host and the device is hidden. The experimental results indicate that (1) CKDB-tree can reduce the space cost comparing to the cell-based indexing structure by 60 percent on average, (2) the approach upon the CKDB-tree outperforms the traditional counterparts upon the KDB-tree by 66, 75 and 79 percent in average for uniform, skewed and hyper-skewed data in terms of update costs, and (3) the GPGPU-aided approach greatly improves the approach upon the CKDB-tree with the support of only a single Kepler GPU, and it provides real-time filtering of streaming data with 2.5M data tuples per second. The massively parallel computing technology exhibits great potentials in streaming data monitoring.

关键词： Streaming data cell-tree query indexing structure KDB-Tree big data computing data-intensive computing GPGPU

来源：评论

学校读者我要写书评

暂无评论

A scalable and fast OPTICS for clustering trajectory big data

引用

CLUSTER computing-THE JOURNAL OF NETWORKS SOFTWARE TOOLS AND APPLICATIONS 2015年第2期18卷 549-562页

作者： Deng, Ze Hu, Yangyang Zhu, Mao Huang, Xiaohui Du, Bo China Univ Geosci Sch Comp Sci Wuhan 430074 Hubei Peoples R China China Univ Geosci Sch Comp Sci Wuhan 430074 Hubei Peoples R China

Clustering trajectory data is an important way to mine hidden information behind moving object sampling data, such as understanding trends in movement patterns, gaining high popularity in geographic information and so on. In the era of 'big data', the current approaches for clustering trajectory data generally do not apply for excessive costs in both scalability and computing performance for trajectory big data. Aiming at these problems, this study first proposes a new clustering algorithm for trajectory big data, namely Tra-POPTICS by modifying a scalable clustering algorithm for point data (POPTICS). Tra-POPTICS has employed the spatiotemporal distance function and trajectory indexing to support trajectory data. Tra-POPTICS can process the trajectory big data in a distributed manner to meet a great scalability. Towards providing a fast solution to clustering trajectory big data, this study has explored the feasibility to utilize the contemporary general-purpose computing on the graphics processing unit (GPGPU). The GPGPU-aided clustering approach parallelized the Tra-POPTICS with the Hyper-Q feature of Kelper GPU and massive GPU threads. The experimental results indicate that (1) the Tra-POPTICS algorithm has a comparable clustering quality with T-OPTICS (the state of art work of clustering trajectories in a centralized fashion) and outperforms T-OPTICS by average four times in terms of scalability, and (2) the G-Tra-POPTICS has a comparable clustering quality with T-POPTICS as well and further gains about 30 speedup on average for clustering trajectories comparing to Tra-POPTICS with eight threads. The proposed algorithms exhibit great scalability and computing performance in clustering trajectory big data.

关键词： Trajectory big data Clustering big data computing GPGPU

来源：评论

学校读者我要写书评

暂无评论

An Empirical Study on Quality Issues of Production big data Platform 37

An Empirical Study on Quality Issues of Production Big Data ...

引用

IEEE/ACM 2nd International Workshop on Software Architecture and Metrics (SAM)

作者： Zhou, Hucheng Lou, Jian-Guang Zhang, Hongyu Lin, Haibo Lin, Haoxiang Qin, Tingting Microsoft Res Beijing Peoples R China Microsoft Beijing Peoples R China

ISBN: (纸本)9781479919345

big data computing platform has evolved to be a multi-tenant service. The service quality matters because system failure or performance slowdown could adversely affect business and user experience. To date, there is few study in literature on service quality issues of production big data computing platform. In this paper, we present an empirical study on the service quality issues of Microsoft ProductA, which is a company-wide multi-tenant big data computing platform, serving thousands of customers from hundreds of teams. ProductA has a well-defined escalation process (i.e., incident management process), which helps customers report service quality issues on 24/7 basis. This paper investigates the common symptom, causes and mitigation of service quality issues in big data platform. We conduct a comprehensive empirical study on 210 real service quality issues of ProductA. Our major findings include (1) 21.0% of escalations are caused by hardware faults;(2) 36.2% are caused by system side defects;(3) 37.2% are due to customer side faults. We also studied the general diagnosis process and the commonly adopted mitigation solutions. Our study results provide valuable guidance on improving existing development and maintenance practice of production big data platform, and motivate tool support.

关键词： empirical study big data computing quality issues escalations fault tolerance

来源：评论

学校读者我要写书评

暂无评论

Design and implementation of task scheduling strategies for massive remote sensing data processing across multiple data centers

引用

SOFTWARE-PRACTICE & EXPERIENCE 2014年第7期44卷 873-886页

作者： Zhang, Wanfeng Wang, Lizhe Ma, Yan Liu, Dingsheng Chinese Acad Sci Inst Remote Sensing & Digital Earth Beijing 100864 Peoples R China China Univ Geosci Sch Comp Beijing Peoples R China Univ Chinese Acad Sci Beijing Peoples R China

data intensive applications of remote sensing data processing are more and more widespread resulting from the evolutions in computer and network technologies. Especially, bags-of-tasks (BoTs) applications with a mass of sharing input files and directed acyclic graph (DAG) applications with data dependencies in a widely distributed computing environment bring new challenges. In this article, a strategy of partitioning group based on hypergraph (PGH) is introduced to formulate the model of sharing files. Within the PGH algorithm, BoTs applications would be partitioned into several groups to minimize the time of data transferring. We also adopted another scheduling policy, which is called optimized task tree (OTT) strategy to handle the DAG workflow of massive remote sensing data processing with data dependencies. A scheduling queue of DAG tasks would be updated according to the priorities changing. With the help of GridSim simulation environment, we designed the Gridlets within scheduler to test the performance of PGH and OTT. Copyright (c) 2013 John Wiley & Sons, Ltd.

关键词： multi-datacenter infrastructure data intensive computing task scheduling big data computing

来源：评论

学校读者我要写书评

暂无评论

Estimating the Statistical Characteristics of Remote Sensing big data in the Wavelet Transform Domain

引用

IEEE TRANSACTIONS ON EMERGING TOPICS IN computing 2014年第3期2卷 324-337页

作者： Wang, Lizhe Zhong, Hui Ranjan, Rajiv Zomaya, Albert Liu, Peng Chinese Acad Sci Inst Remote Sensing & Digital Earth Beijing 100864 Peoples R China Univ Sci & Technol China Dept Elect Engn & Informat Sci Hefei 230026 Peoples R China CSIRO Computat Informat Clayton Vic 3169 Australia Univ Sydney Sch Informat Technol Sydney NSW 2006 Australia

Since it is difficult to deal with big data using traditional models and algorithms, predicting and estimating the characteristics of big data is very important. Remote sensing big data consist of many large-scale images that are extremely complex in terms of their structural, spectral, and textual features. Based on multiresolution analysis theory, most of the natural images are sparse and have obvious clustering and persistence characters when they are transformed into another domain by a group of basic special functions. In this paper, we use a wavelet transform to represent remote sensing big data that are large scale in the space domain, correlated in the spectral domain, and continuous in the time domain We decompose the big data set into approximate multiscale detail coefficients based on a wavelet transform. In order to determine whether the density function of wavelet coefficients in a big data set are peaky at zero and have a heavy tailed shape, a two-component Gaussian mixture model (GMM) is employed. For the first time, we use the expectation-maximization likelihood method to estimate the model parameters for the remote sensing big data set in the wavelet domain. The variance of the GMM with changing of bands, time, and scale are comprehensively analyzed. The statistical characteristics of different textures are also compared. We find that the cluster characteristics of the wavelet coefficients are still obvious in the remote sensing big data set for different bands and different scales. However, it is not always precise when we model the long-term sequence data set using the GMM. We also found that the scale features of different textures for the big data set are obviously reflected in the probability density function and GMM parameters of the wavelet coefficients.

关键词： big data computing remote sensing image processing wavelet parameters estimation

来源：评论

学校读者我要写书评

暂无评论

Task-Tree Based Large-Scale Mosaicking for Massive Remote Sensed Imageries with Dynamic DAG Scheduling

引用

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS 2014年第8期25卷 2126-2137页

作者： Ma, Yan Wang, Lizhe Zomaya, Albert Y. Chen, Dan Ranjan, Rajiv Chinese Acad Sci Inst Remote Sensing & Digital Earth Beijing 100094 Peoples R China Univ Sydney Sch Informat Technol Sydney NSW 2006 Australia China Univ Geosci Sch Comp Wuhan 430074 Peoples R China CSIRO ICT Canberra ACT Australia

Remote sensed imagery mosaicking at large scale has been receiving increasing attentions in regional to global research. However, when scaling to large areas, image mosaicking becomes extremely challenging for the dependency relationships among a large collection of tasks which give rise to ordering constraint, the demand of significant processing capabilities and also the difficulties inherent in organizing these enormous tasks and RS image data. We propose a task-tree based mosaicking for remote sensed imageries at large scale with dynamic DAG scheduling. It expresses large scale mosaicking as a data-driven task tree with minimal height. And also a critical path based dynamical DAG scheduling solution with status queue named CPDS-SQ is provided to offer an optimized schedule on multi-core cluster with minimal completion time. All the individual dependent tasks are run by a core parallel mosaicking program implemented with MPI to perform mosaicking on different pairs of images. Eventually, an effective but easier approach is offered to improve the large-scale processing capability by decoupling the dependence relationships among tasks from the complex parallel processing procedure. Through experiments on large-scale mosaicking, we confirmed that our approach were efficient and scalable.

关键词： Remote sensing image processing parallel computing DAG scheduling data-intensive computing big data computing

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：