检索结果-内蒙古大学图书馆

Design and implementation of big data processing system based on Hadoop

Procedia Computer Science 2025年 259卷 1115-1122页

作者： Siqi Huang Zhenqiang Xie Jiaxiang Wang Penghui Lv Wenrong Wang Guizhou Vocational Technology College of Electronics & Information Kaili 556000 China CETC Big Data Research Institute Co. Ltd. Guiyang 550022 China

With the increasing application of big data, efficient big data processing systems can improve the utilization of big data and create greater value. Cloud computing technology is a product of massive data, and this article proposes a big data processing system developed based on the Hadoop cloud computing platform. The system adopts technologies such as distributed file system HDFS, distributed parallel database MapReduce, and distributed computing framework Spark. This system can process massive amounts of data without the need for programming, greatly improving data processing efficiency and having broad application prospects. This article analyzes the design and implementation methods of a big data processing system based on Hadoop, and conducts performance tests on it. The test results show that the system has high performance and meets basic big data processing requirements.

关键词： Hadoop big data processing Cloud computing

来源：评论

学校读者我要写书评

暂无评论

Hybrid Gradient Descent Golden Eagle Optimization (HGDGEO) Algorithm-Based Efficient Heterogeneous Resource Scheduling for big data processing on Clouds

引用

WIRELESS PERSONAL COMMUNICATIONS 2023年第2期129卷 1175-1195页

作者： Jagadish Kumar, N. Balasubramanian, C. Velammal Inst Technol Dept Informat Technol Chennai 601204 Tamil Nadu India PSR Engn Coll Dept Comp Sci & Engn Sivakasi 626140 Tamil Nadu India

Resource scheduling is indispensable for enhancing the system performance during big data processing on clouds. It is highly useful for attaining significant utilization of computing resources completely concentrating towards the facilitation of resource scalability and on-demand services. The resources essential for running different applications is determined to be maximum heterogeneous in cloud computing. This heterogeneous resource demand introduces a resource gap in which some of the resource potentialities are drained on par with the other resource potentialities still available in the same server resulting in imbalanced resource utilization. This imbalanced resource allocation condition is more apparent when the computing resources are more heterogeneous. At this juncture, intelligent resource scheduling strategy becomes essential to distribute resources for big data processing by adopting a potential decision-making process that focusses on the objective of achieving necessitated tasks over time. In this paper, Hybrid Gradient Descent Golden Eagle Optimization (HGDGEO) algorithm-based efficient heterogeneous resource scheduling process is proposed for handling the challenges that are highly possible during big data processing in the Hadoop heterogenous cloud environment. This HGDGEO algorithm is proposed as an adaptive resource scheduling strategy that handles the dynamic characteristics of the resources and users' fluctuating demand during big data stream processing by mimicking the golden eagles' intelligence which alternates the speed of tuning at different spiral trajectory stages of hunting. It handles big data processing by adopting two adaptive parameters which completely concentrates on optimal resource allocation to suitable VMs in the shortest time possible depending on their requirements. The simulation results of HGDGEO algorithm confirmed its predominance in terms of makespan, load balance and throughput on par with the competitive resource sche

关键词： Gradient descent algorithm Golden eagle optimization big data processing Hadoop cloud environment Hunting spiral trajectory

来源：评论

学校读者我要写书评

暂无评论

big data processing in Smart City Application Using 6G Driven IoT Framework

引用

WIRELESS PERSONAL COMMUNICATIONS 2024年 1-15页

作者： Sun, Maojin Sun, Minghui CEI Cloud Data Storage Technol Beijing Co Ltd 15 Countyard Kechuang Nine Rd Econ Technol Dev Are Beijing 101111 Peoples R China

Smart city has emerged as a strong technological usage for variety of data consumption and processing. The enormous applications like smart health care, traffic monitoring, climate assessment, security applications etc. in smart city generates huge data. This data handling requires next generation technologies like 6G driven IoT for real time big data processing. In this research study, medical big data processing using 6G networks using Internet of medical things (IoMT) has delve to prove efficiency of ultra-reliable and low latency networks. Efficient data transfer for medical application become challenging in 6G due to resource unavailability and causes network traffic. A State-of-art-works provides solution for efficient data transmission and processing in 6G, but still massive amount of sensor data become challenging for cellular networks. These challenges are overcome by utilizing unused network resources in existing network by utilizing Edge-Cloud devices. The deep learning based data processing for 6G enabled (IoMT) with less human intervention improves the prediction accuracy in earlier. In this research we provide diseases data prediction model with transfer learning method for 6G based medical application in first phase. secondly, novel deep learning model is used for classification. In order to facilitate online diagnosis, we intend to link the integrated deep learning algorithm with 6G-enabled IoMT to share and deliver medical records and monitoring data.

关键词： big data processing Smart city application 6G IoT framework Technological development Edge computing

来源：评论

学校读者我要写书评

暂无评论

big data processing With Minimal Delay and Guaranteed data Resolution in Disaster Areas

引用

IEEE TRANSACTIONS ON VEHICULAR TECHNOLOGY 2019年第4期68卷 3833-3842页

作者： Wang, Junbo Sato, Koichi Guo, Song Chen, Wuhui Wu, Jie Univ Aizu Sch Comp Sci & Engn Aizu Wakamatsu Fukushima 9658580 Japan Univ Aizu Aizu Wakamatsu Fukushima 9658580 Japan Hong Kong Polytech Univ Dept Comp Hong Kong Peoples R China Sun Yat Sen Univ Gunagzhou 510275 Peoples R China Temple Univ Dept Comp & Informat Sci Philadelphia PA 19122 USA

big data analysis is very important to support rescue activities when natural disaster happens, through understanding various situations, such as power/water outage regions. The traditional way to process big data is based on high-performance computation/storage resources in a cloud center. However, this is hard to be guaranteed in a disaster scenario due to destruction of communication infrastructure. Meanwhile, high latency between local sensing devices and cloud center sets a big obstacle enabling a near real-time big data analysis. On the other hand, movable base station, such as vehicle-based movable & deployable ICT resource unit (MDRU) developed by NTT, is a possible solution to reconstruct an emergency communication network and process data at the edge sites with reduced data transmission time. In this paper, we study the optimal overall delay in a fog/edge-computing platform constructed by vehicle-based MDRUs with guaranteed data resolution. We formalize the problem as a mixed-integer nonlinear program, which is a well-known NP-hard problem, and then relax the original problem to an mixed integer linear programing (MILP). Finally, we propose a two-stage heuristic algorithm to solve it in a time-efficient manner. Through evaluation, the effectiveness of the proposed heuristic approach has been validated in terms of minimizing overall delay with sufficient given data resolutions.

关键词： big data processing fog/edge computing data resolution disaster scenarios

来源：评论

学校读者我要写书评

暂无评论

big data processing and analysis platform based on deep neural network model

引用

SYSTEMS AND SOFT COMPUTING 2024年 6卷

作者： Huang, Sheng Shanghai Inst Tourism Equipment Dept Shanghai 201418 Peoples R China

Users are increasingly turning to big data processing systems to extract valuable information from massive datasets as the field of big data grows. data analytics platforms are used by e-commerce enterprises to improve product suggestions and model business processes. In order to meet the needs of large-scale data center operation and maintenance management, Internet companies often use Flink to process log data. This paper takes the big data processing and analysis platforms built by Internet financial companies and large banks as examples, and implants a stock prediction model based on Deep Neural Network (DNN). In this context, this paper completes the following work: 1) The research status of big data processing and analysis platforms at home and abroad is introduced. 2) Drawing on the modular design idea, the commercial bank big data platform is designed and the functions of each sub-module are introduced. Then the basic principle and structure of Convolutional Neural Networks (CNN) are expounded. 3) The optimal parameters of Convolutional Neural Networks are selected through experiments, and then the trained model is used for experiments. It can be seen that the stock prediction model proposed in this article has a higher prediction accuracy compared to existing models, which also verifies the validity of the proposed model. Input the data and compare the obtained results with the actual results, and finally show that the model in this paper has a good performance on stock prediction.

关键词： big data processing Analytics platform Deep neural network Stock prediction

来源：评论

学校读者我要写书评

暂无评论

big data processing of School Shooting Archives 16

Big Data Processing of School Shooting Archives

引用

16th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL)

作者： Farag, Mohamed Nakate, Pranav Fox, Edward A. Virginia Tech Blacksburg VA 24061 USA

ISBN: (纸本)9781450342292

Web archives about school shootings consist of webpages that may or may not be relevant to the events of interest. There are 3 main goals of this work;first is to clean the webpages, which involves getting rid of the stop words and non-relevant parts of a webpage. The second goal is to select just webpages relevant to the events of interest. The third goal is to upload the cleaned and relevant webpages to Apache Solr so that they are easily accessible. We show the details of all the steps required to achieve these goals. The results show that representative Web archives are noisy, with 2% - 40% relevant content. By cleaning the archives, we aid researchers to focus on relevant content for their analysis.

关键词： Web Archives big data processing Classification Digital Libraries

来源：评论

学校读者我要写书评

暂无评论

big data processing at the Edge with data Skew Aware Resource Allocation

Big Data Processing at the Edge with Data Skew Aware Resourc...

引用

IEEE Conference on Network Function Virtualization and Software Defined Networks (IEEE NFV-SDN)

作者： Ahmadvand, Hossein Dargahi, Tooska Foroutan, Fouzhan Okorie, Princewill Esposito, Flavio Sharif Univ Technol Tehran Iran Univ Salford Salford Lancs England St Louis Univ St Louis MO USA

ISBN: (纸本)9781665439831

With the increasing number of connected devices and the generation of a large amount of data, efficient methods are required to deal with the complexities of big data processing, especially within edge computing infrastructures. processing batch data on the cloud is often too costly. However, edge servers can be used as alternative resources. Various factors influence Cloud-Edge resource provisioning cause a challenging issue, one of which, data skewness is the focus of this paper. Some data blocks require more processing resources. For example, when processing a log file, the number of repetitions of a target URL varies in different parts of the file which impacts the required processing resources. Existing resource allocation methods ignore the data skew feature in big data processing and apply the same policies to different parts of the input data. In this paper, we propose a data skew aware approach for allocating each data block to the edge or the cloud server for processing. We reduce the cost of data processing on the cloud by categorizing the data blocks based on their significance (i.e., their impact on the processing output). Then, we assign the blocks that are less significant and require less processing power to the Edge and the rest to the cloud. Our experimental results confirm that our provisioning approach improves the processing cost by up to 35% compared to the state-of-the-art.

关键词： data Skew big data processing Resource Allocation Cloud Computing Edge Computing

来源：评论

学校读者我要写书评

暂无评论

Edge computing for big data processing in underwater applications

引用

WIRELESS NETWORKS 2022年第5期28卷 2255-2271页

作者： Periola, A. A. Alonge, A. A. Ogudo, K. A. Univ Johannesburg Dept Elect & Elect Engn Technol Johannesburg South Africa

Underwater data acquisition entities acquire big data that are processed aboard terrestrial data centres. However, processing the big data aboard terrestrial computing entities involves high latency data transfer. In addition, the processing of data in a terrestrial environment is challenging when there is inadequate edge node capacity. These challenges are addressed here. The paper proposes the heterogeneous edge computing paradigm to realize low latency transfer of increasing underwater big data. This is realized via the use of underwater computing entities instead of terrestrial computing entities for processing acquired big data. The proposed heterogeneous edge computing paradigm presents the multi-mode automated teller machine (ATM) as low cost terrestrial edge network entity. The multi-mode ATM is suitable when edge nodes have inadequate computing capacity. Performance evaluation shows that the use of underwater computing entities instead of terrestrial computing entities (existing work) enhances network performance and related capital costs. The number of hops, computing entity access latency and required autonomous underwater vehicle acquisition costs by an average of (5.3-88.4)%, 63.5% and (31.8-95.4)%, respectively. Evaluation shows that the use of the multi-mode ATM in the context of terrestrial cloud computing reduces the number of hops and latency by 44.4% and 37.3% on average, respectively.

关键词： Underwater big data Edge computing big data processing Future computing

来源：评论

学校读者我要写书评

暂无评论

A multi-factor monitoring fault tolerance model based on a GPU cluster for big data processing

引用

INFORMATION SCIENCES 2019年 496卷 300-316页

作者： Fang, Yuling Chen, Qingkui Xiong, Naixue Univ Shanghai Sci & Technol Shanghai 200093 Peoples R China Northeastern State Univ Dept Math & Comp Sci Tahlequah OK 74464 USA

High-performance computing clusters are widely used in large-scale data mining applications, and have higher requirements for persistence, stability and real-time use and sre therefore computationally intensive. To support large-scale data processing, we design a multi-factor real-time monitoring fault tolerance (MRMFT) model based on a GPU cluster. However, the higher clock frequency of GPU chips results in excessively high energy consumption in computing systems. Moreover, the ability to support a long-lasting high temperature operation varies greatly between different GPUs owing to the individual differences between the chips. In this paper, we design a GPU cluster energy consumption monitoring system based on wireless sensor networks (WSNs) and propose an energy consumption aware checkpointing (ECAC) for high energy consumption problems with the following two advantages: the system sets checkpoints according to actual energy consumption and the device temperature to improve the utilization of checkpoints and reduce time cost;and it exploits the parallel computing features of CPU and GPU to hide the CPU detection overhead in GPU parallel computation, and further reduce the time and energy consumption overhead in the fault tolerance phase. Using ECAC as the constraint and aiming for a persistent and reliable operation, the dynamic task migration mechanism is designed, and the reliability of the cluster is greatly improved. The theoretical analysis and experiment results show that the model improves the persistence and stability of the computing system while reducing checkpoint overhead. (C) 2018 Elsevier Inc. All rights reserved.

关键词： big data processing GPU cluster Persistence computing Energy consumption Fault tolerance Energy consumption aware checkpointing Task migration

来源：评论

学校读者我要写书评

暂无评论

FvRS: Efficiently identifying performance-critical data for improving performance of big data processing

引用

FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE 2019年 91卷 157-166页

作者： Xu, Gaoxiang Tan, Zhipeng Feng, Dan Yang, Laurence T. Zhou, Wei Zhang, Xinyan Zhang, Yang Xu, Jie Huazhong Univ Sci & Technol Wuhan Natl Lab Optoelect Key Lab Informat Storage Syst Sch Comp Sci & TechnolMinist Educ China Wuhan Hubei Peoples R China St Francis Xavier Univ Dept Comp Sci Antigonish NS Canada

Hybrid storage is widely implemented in big data processing by providing large storage capacity and high access speed in an economical manner. Performance-critical data are usually stored in SSD to obtain the most performance benefits with the least storage cost. Conventional scheme identifies performance critical data based on data's access hotness. But it does not consider data's I/O cost and may store low-cost data in SSD, resulting in the waste of SSD. The recently proposed scheme determines performance-critical data based on both access hotness and I/O cost. However, it fails to accurately evaluate I/O cost, thus still distributes many low-cost data on SSD. In this paper, we propose a sequentiality-aware identification scheme for performance-critical data, called FvRS, which boots the accuracy of I/O cost evaluation by exploiting data's access sequentiality. The key idea is to evaluate data's I/O cost based on both request size and access sequentiality. By properly identifying high-cost hot data, FvRS maximizes the utilization of SSD to improve system performance. In addition, FvRS maintains performance-critical data in a real-time table to reduce the identification overhead. We have implemented FvRS in a hybrid storage system in Linux. Extensive evaluations using three real-workload traces and a famous benchmark Postmark demonstrate the accuracy and efficiency of FvRS. Compared with the state-of-the-art schemes, such as hotness-based identification and cost-based identification, FvRS reduces I/O response time by 10.3%similar to 45.6% and 16.3%similar to 25.1%, respectively. (C) 2018 Elsevier B.V. All rights reserved.

关键词： big data processing Performance-critical data I/O cost evaluation Access sequentiality Hybrid storage

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：