检索结果-内蒙古大学图书馆

A high-bandwidth and low-cost data processing approach with heterogeneous storage architectures

PERSONAL AND UBIQUITOUS COMPUTING 2020年第2期27卷 159-176页

作者： Wei, Bing Xiao, Limin Wei, Wei Song, Yao Yan, Baicheng Huo, Zhisheng Beihang Univ State Key Lab Software Dev Environm Beijing 100191 Peoples R China Beihang Univ Sch Comp Sci & Engn Beijing 100191 Peoples R China Xian Univ Technol Sch Comp Sci & Engn Xian 710048 Peoples R China

How to efficiently process big data at a low cost is a substantial challenge. Many efficient and economical data processing approaches have been proposed in the fields including business, scientific research, and public administration. Unfortunately, seismic data processing has not achieved the same level of devolvement in the field of oil exploration. While many storage architectures, such as network-attached storage (NAS) and storage area network (SAN), have been widely used to process massive amounts of seismic data, these architectures are expensive in terms of bandwidth and capacity. In this paper, we propose a high-bandwidth and low-cost approach to fill this gap. NASStore is our data store built on NAS for processing seismic data. However, it cannot provide a high bandwidth at a low cost when it comes to data-intensive computing scenarios due to the massive bandwidth requirement and the huge volume of data to be stored. distributed file systems, such as the hadoop distributed file system (HDFS), offer an alternative approach to store data. It delivers high aggregate performance to user applications while running on inexpensive commodity hardware. In order to overcome the shortcomings of NASStore, we first present HDFSStore that is built on HDFS for processing seismic data. We then couple NASStore and HDFSStore to construct a new hybrid data store, called SeisStore, in which efficient parallel write, read, and update mechanisms are employed to improve the system performance. The experiment results show that SeisStore reduces the storage cost than NASStore by up to 23.20% and improves the access bandwidth than NASStore and HDFSStore by up to 478.84% and 16.99%, respectively.

关键词： Network-attached storage hadoop distributed file system High-bandwidth and low-cost data processing Hybrid data store Seismic data processing

来源：评论

学校读者我要写书评

暂无评论

A new replica placement strategy based on multi-objective optimisation for HDFS

引用

INTERNATIONAL JOURNAL OF BIO-INSPIRED COMPUTATION 2020年第1期16卷 13-22页

作者： Li, Yangyang Tian, Mengzhuo Wang, Yang Zhang, Qingfu Saxena, Dhish Kumar Jiao, Licheng Xidian Univ Minist Educ Key Lab Intelligent Percept & Image Understanding Xian Peoples R China City Univ Hong Kong Dept Comp Sci Hong Kong Peoples R China Indian Inst Technol Roorkee Dept Mech & Ind Engn Roorkee Uttar Pradesh India

distributed storage systems like the hadoop distributed file system (HDFS) constitute the core infrastructure of cloud platforms which are well poised to deal with big-data. An optimised HDFS is critical for effective data management in terms of reduced file service time and access latency, improved file availability and system load balancing. Recognising that the file-replication strategy is key to an optimised HDFS, this paper focuses on the file-replica placement strategy while simultaneously considering storage and network load. Firstly, the conflicting relationship between storage and network load is analysed and a bi-objective optimisation model is built, following which a multi-objective optimisation memetic algorithm based on decomposition (MOMAD) and its improved version are used. Compared to the default strategy in HDFS, the file-replica placement strategies based on multi-objective optimisation provide more diverse solutions. And competitive performance could be obtained by the proposed algorithm.

关键词： hadoop hadoop distributed file system HDFS replica placement multi-objective optimisation memetic algorithm

来源：评论

学校读者我要写书评

暂无评论

Elastic HDFS: interconnected distributed architecture for availability-scalability enhancement of large-scale cloud storages

引用

JOURNAL OF SUPERCOMPUTING 2020年第1期76卷 174-203页

作者： Maghsoudloo, M. Khoshavi, N. Golestan Univ Fac Engn Dept Comp Engn Gorgan Golestan Iran Florida Polytech Univ Dept Comp Sci Lakeland FL USA Florida Polytech Univ Dept Elect & Comp Engn Lakeland FL USA

This paper presents an interconnected distributed architecture for storing data and metadata in large-scale cloud storage systems. The primary goal of the proposed architecture is to enhance the scalability of namespace directory in large-scale file systems. Structural shift from distinguished distributed model to interconnected distributed model and conducting effective coordination among file servers for namespace management are two key solutions considered in the context of proposed architecture. To this intent, a coordination protocol is designed for communication among file servers, and maintaining user transparency in the presence of different file system actions/reactions. The experimental results, obtained via emulations under different network conditions and cloud storage sizes, show up to 43.9% availability and 37.8% connection throughput improvements with negligible storage overhead compared to the latest released version of hadoop distributed file system.

关键词： Scalability Availability Cloud storage systems hadoop distributed file system Namespace directory

来源：评论

学校读者我要写书评

暂无评论

SecDedoop: Secure Deduplication with Access Control of Big Data in the HDFS/hadoop Environment

引用

BIG DATA 2020年第2期8卷 147-163页

作者： Ramya, P. Sundar, C. Christian Coll Engn & Technol Dept Informat Technol Oddanchatram 624619 Dindigul India Christian Coll Engn & Technol Dept Comp Sci & Engn Oddanchatram Dindigul India

With the rapid growth of storage providers, data deduplication is an essential storage optimization technique that greatly minimizes data storage costs by storing a unique copy of duplicate data. Nowadays, deduplication introduces various new challenges such as security and insufficient space issue. Hence, in this article, we propose a secure data deduplication with access control of big data over HDFS (hadoop distributed file system)/hadoop environment, called SecDedoop. First, the system achieves security for data confidentiality by third party vendor using elliptic curve cryptography. There are two types of keys (public key and private key) generated for data retrieval. Second, we consider data deduplication. The user's original file is divided into a number of equal chunks. Then, each chunk (e.g., 1. txt) is tokenized into words and the weight of words is computed by using TF-IDF frequency. The SHA-3 hash computation is performed to the user's original file. If the hash value is not duplicate, then we store data in HDFS. The PSO (particle swarm optimization)-based MapReduce model is the proposed best data node selection. Initially, MapReduce process is finished for the user's original file and it results in the best set of data nodes;then, we apply PSO to compute the fitness value for best data node selection. Further, we consider MongoDB for fast indexing of the user's original files and also apply FCM (fuzzy-C-means clustering) for clustering the user's files. In this article, we consider the modified version of PSO and FCM to eliminate the open issues in conventional PSO and FCM. The performance of our proposed SecDedoop has been evaluated by using various performance metrics and also proved it outperforms better than previous approaches.

关键词： access control data deduplication elliptic curve cryptography hadoop distributed file system particle swarm optimization-based MapReduce SHA-3

来源：评论

学校读者我要写书评

暂无评论

基于大数据的数据处理与分析技术

引用

数字技术与应用 2025年第3期43卷 208-210页

作者：童晓涛中国联合网络通信有限公司云南省分公司

在数字化浪潮下,数据跃升为核心资产,大数据处理与分析技术成为焦点。本文深入剖析了其架构,数据存储涵盖hadoop distributed file system等,计算框架有Map Reduce与Apache Spark等助力。在安全隐私保护层面,本文详述了分布式拒绝服务攻... 详细信息

在数字化浪潮下,数据跃升为核心资产,大数据处理与分析技术成为焦点。本文深入剖析了其架构,数据存储涵盖hadoop distributed file system等,计算框架有Map Reduce与Apache Spark等助力。在安全隐私保护层面,本文详述了分布式拒绝服务攻击(DDoS)等威胁及对应防护技术,对称、非对称加密算法等应用,以及欧盟GDPR、美国CCPA和我国相关法规与企业实践。

关键词：数据处理 hadoop distributed file system 大数据

来源：评论

学校读者我要写书评

暂无评论

An Efficient Group-Based Replica Placement Policy for Large-Scale Geospatial 3D Raster Data on hadoop

引用

SENSORS 2021年第23期21卷 8132-8132页

作者： Liu, Zhipeng Hua, Weihua Liu, Xiuguo Liang, Dong Zhao, Yabo Shi, Manxing China Univ Geosci Sch Geog & Informat Engn Wuhan 430074 Peoples R China

Geospatial three-dimensional (3D) raster data have been widely used for simple representations and analysis, such as geological models, spatio-temporal satellite data, hyperspectral images, and climate data. With the increasing requirements of resolution and accuracy, the amount of geospatial 3D raster data has grown exponentially. In recent years, the processing of large raster data using hadoop has gained popularity. However, data uploaded to hadoop are randomly distributed onto datanodes without consideration of the spatial characteristics. As a result, the direct processing of geospatial 3D raster data produces a massive network data exchange among the datanodes and degrades the performance of the cluster. To address this problem, we propose an efficient group-based replica placement policy for large-scale geospatial 3D raster data, aiming to optimize the locations of the replicas in the cluster to reduce the network overhead. An overlapped group scheme was designed for three replicas of each file. The data in each group were placed in the same datanode, and different colocation patterns for three replicas were implemented to further reduce the communication between groups. The experimental results show that our approach significantly reduces the network overhead during data acquisition for 3D raster data in the hadoop cluster, and maintains the hadoop replica placement requirements.

关键词： 3D raster distributed GIS hadoop distributed file system replica placement

来源：评论

学校读者我要写书评

暂无评论

Research of the Strategy to Improve the hadoop distributed file system Downloading Performance

Research of the Strategy to Improve the Hadoop Distributed F...

引用

2012电子信息与电气工程国际学术会议(EIEE2012)

作者： GE Junwei Library Chongqing University of Posts and Telecommunications Chong Qing, China Jiang Renxiang College of Computer Science and Technology Chongqing University of Posts and Telecommunications Chong Qing, China

This paper introduced the process of client downloading files from the hadoop distributed file system,in the process of downloading files were studied for the user experience is poor and possible load imbalance problem , proposed a parallel download programs,and in the process of downloading data blocks from multiple backup data storage DataNodes,used an improved rate of adaptive dynamic parallel downloading mechanism. Experimental results show that the improved download strategy greatly reduces the waiting time of the client downloading files from the hadoop distributed file system. Parallel download can improve hadoop distributed file system download efficiency and make the DataNode more balanced load.

关键词： cloud computing hadoop distributed file system parallel download load balancing

来源：评论

学校读者我要写书评

暂无评论

A Big Data Lake for Multilevel Streaming Analytics 1

A Big Data Lake for Multilevel Streaming Analytics

引用

1st International Conference on Big Data Analytics and Practices (IBDAP)

作者： Liu, Ruoran Isah, Haruna Zulkernine, Farhana Queens Univ Sch Comp Kingston ON Canada

ISBN: (数字)9781728181066

ISBN: (纸本)9781728181066

Large organizations are seeking to create new architectures and scalable platforms to effectively handle data management challenges due to the explosive nature of data rarely seen in the past. These data management challenges are largely posed by the availability of streaming data at high velocity from various sources in multiple formats. The changes in data paradigm have led to the emergence of new data analytics and management architecture. This paper focuses on storing high volume, velocity and variety data in the raw formats in a data storage architecture called a data lake. First, we present our study on the limitations of traditional data warehouses in handling recent changes in data paradigms. We discuss and compare different open source and commercial platforms that can be used to develop a data lake. We then describe our end-to-end data lake design and implementation approach using the hadoop distributed file system (HDFS) on the hadoop Data Platform (HDP). Finally, we present a real-world data lake development use case for data stream ingestion, staging, and multilevel streaming analytics which combines structured and unstructured data. This study can serve as a guide for individuals or organizations planning to implement a data lake solution for their use cases.

关键词： hadoop Data Platform hadoop distributed file system NiFi streaming data unstructured data

来源：评论

学校读者我要写书评

暂无评论

MISS-D: A fast and scalable framework of medical image storage service based on distributed file system

引用

COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2020年 186卷 105189-105189页

作者： Li, Wei Feng, Chaolu Yu, Kun Zhao, Dazhe Northeastern Univ Key Lab Intelligent Comp Med Image MIIC Minist Educ Shenyang Peoples R China Northeastern Univ Key Lab Med Image Comp MIC Shenyang Liaoning Peoples R China Northeastern Univ Sch Med & Biol Informat Engn Shenyang Peoples R China Northeastern Univ Sch Comp Sci & Engn Shenyang Peoples R China

Background and Objective Processing of medical imaging big data is deeply challenging due to the size of data, computational complexity, security storage and inherent privacy issues. Traditional picture archiving and communication system, which is an imaging technology used in the healthcare industry, generally uses centralized high performance disk storage arrays in the practical solutions. The existing storage solutions are not suitable for the diverse range of medical imaging big data that needs to be stored reliably and accessed in a timely manner. The economical solution is emerging as the cloud computing which provides scalability, elasticity, performance and better managing cost. Cloud based storage architecture for medical imaging big data has attracted more and more attention in industry and academia. Methods This study presents a novel, fast and scalable framework of medical image storage service based on distributed file system. Two innovations of the framework are introduced in this paper. An integrated medical imaging content indexing file model for large-scale image sequence is designed to adapt to the high performance storage efficiency on distributed file system. A virtual file pooling technology is proposed, which uses the memory-mapped file method to achieve an efficient data reading process and provides the data swapping strategy in the pool. Result The experiments show that the framework not only has comparable performance of reading and writing files which meets requirements in real-time application domain, but also bings greater convenience for clinical system developers by multiple client accessing types. The framework supports different user client types through the unified micro-service interfaces which basically meet the needs of clinical system development especially for online applications. The experimental results demonstrate the framework can meet the needs of real-time data access as well as traditional picture archiving and communicat

关键词： hadoop distributed file system Data packing Memory mapping file Message queue Micro-service Medical imaging

来源：评论

学校读者我要写书评

暂无评论

Improvement Of Data Throughput In Data-Intensive Cloud Computing Applications 5

Improvement Of Data Throughput In Data-Intensive Cloud Compu...

引用

5th IEEE International Conference on Big Data Computing Service and Applications (IEEE BigDataService) / Workshop on Big Data in Water Resources, Environment, and Hydraulic Engineering / Workshop on Medical, Healthcare, Using Big Data Technologies

作者： Ibrahim, Ibrahim Adel Bassiouni, Mostafa Univ Cent Florida Dept Elect & Comp Engn Orlando FL 32816 USA Univ Cent Florida Dept Comp Sci Orlando FL 32816 USA Univ Technol Baghdad Baghdad Iraq

ISBN: (纸本)9781728100593

In recent years, hadoop framework is popularly known for providing cost-effective solutions to process large-scale data intensive applications in a distributed manner. One of the key issues that can significantly affect the performance of data-intensive cloud computing is computation load balancing among cluster nodes. Replica placement in HDFS plays a significant role in data availability and balanced utilization of clusters. In the current replica placement policy of hadoop distributed file system (HDFS), the replicas of data blocks cannot be evenly distributed across cluster's nodes, so the current HDFS must rely on load balancing utility for balancing replicas' distributions, which results in more time and resources consuming. In this paper, we address the load balancing problem and present an innovative replica placement policy for HDFS for data-intensive computing on massive datasets on the cloud system. It can perfectly balance the computing load among cluster's nodes in both homogeneous and heterogeneous cluster environments. Experimental results of the proposed solution confirm that, the proposed replica placement scheme gives better data nodes utilization than the default replica placement policy of hadoop.

关键词： Cloud Computing hadoop distributed file system Replica Placement policy Data Replication Load Balance

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：