检索结果-内蒙古大学图书馆

pipsCloud: High performance cloud computing for remote sensing big data management and processing

FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE 2018年第Part1期78卷 353-368页

作者： Wang, Lizhe Ma, Yan Yan, Jining Chang, Victor Zomaya, Albert Y. China Univ Geosci Sch Comp Sci Wuhan 430074 Peoples R China Chinese Acad Sci Inst Remote Sensing & Digital Earth Beijing Peoples R China Xian Jiaotong Liverpool Univ Suzhou Peoples R China Univ Sydney Sch Informat Technol Sydney NSW Australia

Massive, large-region coverage, multi-temporal, multi-spectral remote sensing (RS) datasets are employed widely due to the increasing requirements for accurate and up-to-date information about resources and the environment for regional and global monitoring. In general, RS data processing involves a complex multi-stage processing sequence, which comprises several independent processing steps according to the type of RS application. RS data processing for regional environmental and disaster monitoring is recognized as being computationally intensive and data intensive. We propose pipsCloud to address these issues in an efficient manner, which combines recent cloud computing and HPC techniques to obtain a large-scale RS data processing system that is suitable for on-demand real-time services. Due to the ubiquity, elasticity, and high-level transparency of the cloud computing model, massive RS data management and data processing for dynamic environmental monitoring can all be performed on the cloud via Web interfaces. A Hilbert-R+-based data indexing method is employed for the optimal querying and access of RS images, RS data products, and interim data. In the core platform beneath the cloud services, we provide a parallel file system for massive high dimensional RS data, as well as interfaces for accessing irregular RS data to improve data locality and optimize the I/O performance. Moreover, we use an adaptive RS data analysis workflow management system for on-demand workflow construction and the collaborative processing of a distributed complex chain of RS data, e.g., for forest fire detection, mineral resources detection, and coastline monitoring. Our experimental analysis demonstrated the efficiency of the pipsCloud platform. (C) 2016 Elsevier B.V. All rights reserved.

关键词： Big data Cloud computing data-intensive computing High performance computing Remote sensing

来源：评论

学校读者我要写书评

暂无评论

A security framework in G-Hadoop for big data computing across distributed Cloud data centres

引用

JOURNAL OF COMPUTER AND SYSTEM SCIENCES 2014年第5期80卷 994-1007页

作者： Zhao, Jiaqi Wang, Lizhe Tao, Jie Chen, Jinjun Sun, Weiye Ranjan, Rajiv Kolodziej, Joanna Streit, Achim Georgakopoulos, Dimitrios Changchun Univ Technol Sch Basic Sci Changchun Peoples R China Chinese Acad Sci Inst Remote Sensing & Digital Earth Beijing 100864 Peoples R China Karlsruhe Inst Technol Steinbuch Ctr Comp D-76021 Karlsruhe Germany Univ Technol Sydney Fac Engn & IT Sydney NSW 2007 Australia CSIRO ICT Ctr Informat Engn Lab Marsfield NSW Australia Cracow Univ Technol Inst Comp Sci Krakow Poland

MapReduce is regarded as an adequate programming model for large-scale data-intensive applications. The Hadoop framework is a well-known MapReduce implementation that runs the MapReduce tasks on a cluster system. G-Hadoop is an extension of the Hadoop MapReduce framework with the functionality of allowing the MapReduce tasks to run on multiple clusters. However, G-Hadoop simply reuses the user authentication and job submission mechanism of Hadoop, which is designed for a single cluster. This work proposes a new security model for G-Hadoop. The security model is based on several security solutions such as public key cryptography and the SSL protocol, and is dedicatedly designed for distributed environments. This security framework simplifies the users authentication and job submission process of the current G-Hadoop implementation with a single-sign-on approach. In addition, the designed security framework provides a number of different security mechanisms to protect the G-Hadoop system from traditional attacks. (c) 2014 Elsevier Inc. All rights reserved.

关键词： Cloud computing Massive data processing data-intensive computing Security model Map Reduce

来源：评论

学校读者我要写书评

暂无评论

DECO: Joint Computation, Caching and Forwarding in data-Centric computing Networks 19

DECO: Joint Computation, Caching and Forwarding in Data-Cent...

引用

20th ACM International Symposium on Mobile Ad Hoc Networking and computing (Mobihoc)

作者： Kamran, Khashayar Yeh, Edmund Ma, Qian Northeastern Univ Boston MA 02115 USA

ISBN: (纸本)9781450367646

The emergence of IoT devices and the predicted increase in the number of data-driven and delay-sensitive applications highlight the importance of dispersed computing platforms (e.g. edge computing and fog computing) that can intelligently manage in-network computation and data placement. In this paper, we propose the DECO (data-cEntric Computation) framework for joint computation, caching, and request forwarding in data-centric computing networks. DECO utilizes a virtual control plane which operates on the demand rates for computation and data, and an actual plane which handles computation requests, data requests, data objects and computation results in the physical network. We present a throughput optimal policy within the virtual plane, and use it as a basis for adaptive and distributed computation, caching, and request forwarding in the actual plane. We demonstrate the superior performance of the DECO policy in terms of request satisfaction delay as compared with several baseline policies, through extensive numerical simulations over multiple network topologies.

关键词： Distributed computing networks fog computing mobile edge computing data-intensive computing data-centric computing caching

来源：评论

学校读者我要写书评

暂无评论

Multilevel Active Storage for Big data Applications in High Performance computing

Multilevel Active Storage for Big Data Applications in High ...

引用

IEEE International Conference on Big data (Big data)

作者： Chen, Chao Lang, Michael Chen, Yong Texas Tech Univ Dept Comp Sci Lubbock TX 79409 USA Los Alamos Natl Lab Los Alamos NM 87544 USA Texas Tech Univ Dept Comp Sci Lubbock TX 79409 USA

ISBN: (纸本)9781479912926;9781479912933

Given the growing importance of supporting data-intensive sciences and big data applications, an effective HPC I/O solution has become a key issue and has attracted intensive attention in recent years. Active storage has been shown effective in reducing data movement and network traffic as a potential new I/O solution. Existing prototypes and systems, however, are primarily designed for read-intensive applications. In addition, they generally assume that offloaded processing kernels have small computational demands, which makes this solution a poor fit for data-intensive operations that have significant computational demands, including write-intensive operations. In this research, we propose a new Multilevel Active Storage (MAS) solution. The new MAS design can support and handle both read-and write-intensive operations, as well as complex operations that have considerable computational demands. Experimental tests have been carried out and confirmed that the MAS approach is feasible and outperformed existing approaches. The new multilevel active storage design has a potential to deliver a high performance I/O solution for big data applications in HPC.

关键词： Big data active storage high performance computing data-intensive computing parallel file systems

来源：评论

学校读者我要写书评

暂无评论

Enabling Remote Fault Diagnosis through data-driven Grid computing

引用

INTERNATIONAL JOURNAL OF GRID AND DISTRIBUTED computing 2016年第4期9卷 185-199页

作者： Tang, Bing Zhang, Li Hunan Univ Sci & Technol Sch Comp Sci & Engn Xiangtan 411201 Hunan Peoples R China

Due to the complexity of modern manufacturing and mechanical equipments, it is difficult for equipment users or maintainers to accomplish fault diagnosis independently. In this paper, the development history of fault diagnosis technology is surveyed and investigated, especially remote fault diagnosis system based on Internet in detail. Then, a remote fault diagnosis system based on grid computing technology is proposed to enable collaborative resource sharing and problem solving among multiple equipment suppliers and equipment users by integrating all kinds of diagnostic resources. The architecture of this fault diagnosis system is presented, as well as the Client-Master-Worker computation model and the diagnostic workflow. Finally, a prototype system is implemented using the data-driven middleware-BitDew, in which multiple fault diagnosis grid services are integrated in a unified Web portal. The case study of data-driven fault data analysis is conducted in our prototype which has proved the effectiveness of the system.

关键词： data-driven grid computing data-intensive computing fault diagnosis

来源：评论

学校读者我要写书评

暂无评论

An ultra-high-density and energy-efficient content addressable memory design based on 3D-NAND flash

引用

Science China(Information Sciences) 2023年第4期66卷 261-271页

作者： Haozhang YANG Peng HUANG Runze HAN Xiaoyan LIU Jinfeng KANG School of Integrated Circuits Peking University

In this article, the design of a novel 3D-NAND-based content addressable memory(CAM) consisting of two flash transistors with ultra-high density and low power consumption is proposed for dataintensive computing. Hewlett simulation program with integrated circuit emphasis(HSPICE) of a 3D-NAND-based CAM array is performed to study the functionality and properties of the presented CAM design, and the results indicate that the energy consumption is 0.196 fJ/bit/search. The cell density of a 16-layer 3D-NAND flash is 157 times higher than CAMs based on conventional static random access ***, to exploit the multibit storage property of 3D-NAND flash, we also propose a multilevel CAM design, which significantly boosts the cell density and expands the functionality. As a proof-of-concept illustration, we take a 4-level CAM to successfully implement the search operation. Furthermore, the impacts of 3D-NAND layers and parasitic effects on the performance of the proposed CAM design are also discussed.

关键词： content addressable memory (CAM) 3D-NAND flash data-intensive computing in-memory computing multilevel CAM

来源：评论

学校读者我要写书评

暂无评论

Load-balanced and locality-aware scheduling for data-intensive workloads at extreme scales

引用

CONCURRENCY AND COMPUTATION-PRACTICE & EXPERIENCE 2016年第1期28卷 70-94页

作者： Wang, Ke Qiao, Kan Sadooghi, Iman Zhou, Xiaobing Li, Tonglin Lang, Michael Raicu, Ioan IIT Dept Comp Sci Chicago IL 60616 USA Google Inc Seattle WA 98103 USA Hortonworks Inc Santa Clara CA USA Los Alamos Natl Lab Los Alamos NM USA Argonne Natl Lab Lemont IL USA

data-driven programming models such as many-task computing (MTC) have been prevalent for running data-intensive scientific applications. MTC applies over-decomposition to enable distributed scheduling. To achieve extreme scalability, MTC proposes a fully distributed task scheduling architecture that employs as many schedulers as the compute nodes to make scheduling decisions. Achieving distributed load balancing and best exploiting data locality are two important goals for the best performance of distributed scheduling of data-intensive applications. Our previous research proposed a data-aware work-stealing technique to optimize both load balancing and data locality by using both dedicated and shared task ready queues in each scheduler. Tasks were organized in queues based on the input data size and location. Distributed key-value store was applied to manage task metadata. We implemented the technique in MATRIX, a distributed MTC task execution framework. In this work, we devise an analytical suboptimal upper bound of the proposed technique, compare MATRIX with other scheduling systems, and explore the scalability of the technique at extreme scales. Results show that the technique is not only scalable but can achieve performance within 15% of the suboptimal solution. Copyright (c) 2015 John Wiley & Sons, Ltd.

关键词： data-intensive computing data-aware scheduling work stealing key-value stores many-task computing

来源：评论

学校读者我要写书评

暂无评论

Accelerating big data analytics on HPC clusters using two-level storage

引用

PARALLEL computing 2017年 61卷 18-34页

作者： Xuan, Pengfei Ligon, Walter B. Srimani, Pradip K. Ge, Rong Luo, Feng Clemson Univ Sch Comp Clemson SC 29634 USA Clemson Univ Elect & Comp Engn Clemson SC 29634 USA

data-intensive applications that are inherently I/O bound have become a major workload on traditional high-performance computing (HPC) clusters. Simply employing data intensive computing storage such as HDFS or using parallel file systems available on HPC clusters to serve such applications incurs performance and scalability issues. In this paper, we present a novel two-level storage system that integrates an upper-level in-memory file system with a lower-level parallel file system. The former renders memory-speed high I/O performance and the latter renders consistent storage with large capacity. We build a two level storage system prototype with Tachyon and OrangeFS, and analyze the resulting I/O throughput for typical MapReduce operations. Theoretical modeling and experiments show that the proposed two-level storage delivers higher aggregate I/O throughput than HDFS and OrangeFS and achieves scalable performance for both read and write. We expect this two-level storage approach to provide insights on system design for big data analytics on HPC clusters. (C) 2016 Elsevier B.V. All rights reserved.

关键词： Two-level storage In-memory file system Parallel file system data-intensive computing

来源：评论

学校读者我要写书评

暂无评论

Breaking the MapReduce stage barrier

引用

CLUSTER computing-THE JOURNAL OF NETWORKS SOFTWARE TOOLS AND APPLICATIONS 2013年第1期16卷 191-206页

作者： Verma, Abhishek Cho, Brian Zea, Nicolas Gupta, Indranil Campbell, Roy H. Univ Illinois Dept Comp Sci Urbana IL 61801 USA

The MapReduce model uses a barrier between the Map and Reduce stages. This provides simplicity in both programming and implementation. However, in many situations, this barrier hurts performance because it is overly restrictive. Hence, we develop a method to break the barrier in MapReduce in a way that improves efficiency. Careful design of our barrier-less MapReduce framework results in equivalent generality and retains ease of programming. We motivate our case with, and experimentally study our barrier-less techniques in, a wide variety of MapReduce applications divided into seven classes. Our experiments show that our approach can achieve better job completion times than a traditional MapReduce framework. This is due primarily to the interleaving of I/O and computation, and forgoing disk-intensive work. We achieve a reduction in job completion times that is 25% on average and 87% in the best case.

关键词： MapReduce data-intensive computing

来源：评论

学校读者我要写书评

暂无评论

A Parallel File System with Application-Aware data Layout Policies for Massive Remote Sensing Image Processing in Digital Earth

引用

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS 2015年第6期26卷 1497-1508页

作者： Wang, Lizhe Ma, Yan Zomaya, Albert Y. Ranjan, Rajiv Chen, Dan Chinese Acad Sci Inst Remote Sensing & Digital Earth Beijing 100094 Peoples R China Univ Sydney Sch Informat Technol Sydney NSW 2006 Australia ICT CSIRO Marsfield NSW Australia Wuhan Univ Sch Comp Wuhan 430072 Hubei Peoples R China

Remote sensing applications in Digital Earth are overwhelmed with vast quantities of remote sensing (RS) image data. The intolerable I/O burden introduced by the massive amounts of RS data and the irregular RS data access patterns has made the traditional cluster based parallel I/O systems no longer applicable. We propose a RS data object-based parallel file system for remote sensing applications and implement it with the OrangeFS file system. It provides application-aware data layout policies, together with RS data object based data I/O interfaces, for efficient support of various data access patterns of RS applications from the server side. With the prior knowledge of the desired RS data access patterns, HPGFS could offer relevant space-filling curves to organize the sliced 3-D data bricks and distribute them over I/O servers. In this way, data layouts consistent with expected data access patterns could be created to explore data locality and achieve performance improvement. Moreover, the multi-band RS data with complex structured geographical metadata could be accessed and managed as a single data object. Through experiments on remote sensing applications with different access patterns, we have achieved performance improvement of about 30 percent for I/O and 20 percent overall.

关键词： Parallel file system parallel I/O data-intensive computing digital earth big data remote sensing image processing

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：