检索结果-内蒙古大学图书馆

A two-level formal model for big data processing programs

SCIENCE OF COMPUTER PROGRAMMING 2022年 215卷 102764-102764页

作者： de Souza Neto, Joao Batista Moreira, Anamaria Martins Vargas-Solar, Genoveva Musicante, Martin A. Univ Fed Rio Grande do Norte Dept Informat & Appl Math DIMAp Natal RN Brazil Fed Ctr Technol Educ Minas Gerais Dept Informat Management & Design DIGD DV Divinopolis Brazil Univ Fed Rio de Janeiro Inst Comp IC Rio De Janeiro Brazil LIRIS French Council Sci Res CNRS Lyon France

This paper proposes a model for specifying data flow-based parallel data processing programs agnostic of target big data processing frameworks. The paper focuses on the formal abstract specification of non-iterative and iterative programs, generalizing the strategies adopted by data flow big data processing frameworks. The proposed model relies on Monoid Algebra and Petri Nets to abstract big data processing programs in two levels: a higher level representing the program data flow and a lower level representing data transformation operations (e.g., filtering, aggregation, join). We extend the model for data processing programs proposed in [1], for modeling iterative data processing programs. The general specification of these programs implemented by data flow-based parallel programming models is essential given the democratization of iterative and greedy big data analytics algorithms. Indeed, these algorithms call for revisiting parallel programming models to express iterations. The paper gives a comparative analysis of the iteration strategies proposed by Apache Spark, DryadLINQ, Apache Beam, and Apache Flink. It discusses how the model achieves to generalize these strategies. (c) 2021 Elsevier B.V. All rights reserved.

关键词： big data processing data flow programming models Petri nets Monoid algebra

来源：评论

学校读者我要写书评

暂无评论

Research on Improved Shallow Neural Network big data processing Model based on Gaussian Function

引用

INTERNATIONAL JOURNAL OF ADVANCED COMPUTER SCIENCE AND APPLICATIONS 2022年第10期13卷 566-573页

作者： Fu, Lifang Henan Univ Anim Husb & Econ Fac Sci Zhengzhou 450044 Peoples R China

The application of the current new generation communication technology is gradually diversified, and the global Internet users are increasing, leading some large enterprises to increasingly rely on faster and more efficient big data processing technology. In order to solve the shortcomings of the current big data processing algorithms, such as slow computing speed, computing accuracy to be improved, and poor online real-time learning ability, this research combines incremental learning and sliding window ideas to design two improved radial basis function (RBF) neural network algorithms with Gaussian function as the kernel function. The Duffing equation example and the data of "Top 100 single products for Taobao search glasses sales" were used to verify the performance of the design algorithm. The experimental results of Duffing equation example show that when the total sample is 100000, the mean square errors of IOL, SWOL, SVM and ResNet50 algorithms are 1.86e-07, 1.59e-07, 3.37e-07 and 2.67e-07 respectively. The experimental results of the data set of "Top 100 SKUs for Taobao Search Glasses Sales" show that when the number of samples in the test set is 800, the root mean square errors of IOL, SWOL, SVM and ResNet50 algorithms are 0.0060, 0.0056, 0.0069 and 0.0073 respectively. This shows that the RBF online learning algorithm designed in this study, which integrates sliding windows, has a stronger comprehensive ability to process big data, and has certain application value for improving the accuracy of online data based commodity recommendation in e -commerce and other industries.

关键词： Gaussian function RBF big data processing incremental learning sliding window

来源：评论

学校读者我要写书评

暂无评论

Fast and Efficient In-Memory big data processing

引用

INTERNATIONAL JOURNAL OF ADVANCED COMPUTER SCIENCE AND APPLICATIONS 2019年第5期10卷 517-524页

作者： Malik, Babur Hayat Maryam, Maliha Khalid, Myda Khlaid, Javaria Rehman, Naj Am Ur Sajjad, Syeda Iqra Islam, Tanveer Butt, Umair Ahmed Raza, Ali Nasr, M. Saad Univ Lahore Dept CS & IT Chenab Campus Gujrat Pakistan

With the passage of time, the data is growing exponentially and the mostly endured areas are social media networks, media hosting applications, and servers. They have thousands of Tera-bytes of data and the efficient systems, however, they are as yet confronting issue to oversee such volume of information and its size is growing each day. data systems retrieve information with less time of In-memory. Instead of each factor data systems are required to define good usage of cache and fast memory access with help of optimization. The proposed technique to solve this problem can be the optimal indexing technique with better and efficient utilization of Cache and having less overhead of DRAM with the goal that energy can also be saved for the high-end servers.

关键词： big data processing indexing techniques R-tree B-tree X-tree hashing inverted index graph query tree

来源：评论

学校读者我要写书评

暂无评论

Performance Analysis of IoT-Based Sensor, big data processing, and Machine Learning Model for Real-Time Monitoring System in Automotive Manufacturing

引用

SENSORS 2018年第9期18卷 2946-2946页

作者： Syafrudin, Muhammad Alfian, Ganjar Fitriyani, Norma Latif Rhee, Jongtae Dongguk Univ Dept Ind & Syst Engn Seoul 100715 South Korea Dongguk Univ Nano Informat Technol Acad U SCM Res Ctr Seoul 100715 South Korea

With the increase in the amount of data captured during the manufacturing process, monitoring systems are becoming important factors in decision making for management. Current technologies such as Internet of Things (IoT)-based sensors can be considered a solution to provide efficient monitoring of the manufacturing process. In this study, a real-time monitoring system that utilizes IoT-based sensors, big data processing, and a hybrid prediction model is proposed. Firstly, an IoT-based sensor that collects temperature, humidity, accelerometer, and gyroscope data was developed. The characteristics of IoT-generated sensor data from the manufacturing process are: real-time, large amounts, and unstructured type. The proposed big data processing platform utilizes Apache Kafka as a message queue, Apache Storm as a real-time processing engine and MongoDB to store the sensor data from the manufacturing process. Secondly, for the proposed hybrid prediction model, Density-Based Spatial Clustering of Applications with Noise (DBSCAN)-based outlier detection and Random Forest classification were used to remove outlier sensor data and provide fault detection during the manufacturing process, respectively. The proposed model was evaluated and tested at an automotive manufacturing assembly line in Korea. The results showed that IoT-based sensors and the proposed big data processing system are sufficiently efficient to monitor the manufacturing process. Furthermore, the proposed hybrid prediction model has better fault prediction accuracy than other models given the sensor data as input. The proposed system is expected to support management by improving decision-making and will help prevent unexpected losses caused by faults during the manufacturing process.

关键词： monitoring system IoT-based sensor big data processing fault detection DBSCAN Random Forest

来源：评论

学校读者我要写书评

暂无评论

A flexible I/O arbitration framework for netCDF-based big data processing workflows on high-end supercomputers

引用

CONCURRENCY AND COMPUTATION-PRACTICE & EXPERIENCE 2017年第15期29卷 1-12页

作者： Liao, Jianwei Gerofi, Balazs Lien, Guo-Yuan Miyoshi, Takemasa Nishizawa, Seiya Tomita, Hirofumi Liao, Wei-Keng Choudhary, Alok Ishikawa, Yutaka Southwest Univ China Coll Comp & Informat Sci Tianshen Rd 2 Chongqing Peoples R China Nanjing Univ State Key Lab Novel Software Technol Nanjing Jiangsu Peoples R China RIKEN Adv Inst Computat Sci Kobe Hyogo Japan Northwestern Univ Dept Elect Engn & Comp Sci Evanston IL USA

On the verge of the convergence between high-performance computing and big data processing, it has become increasingly prevalent to deploy large-scale data analytics workloads on high-end supercomputers. Such applications often come in the form of complex workflows with various different components, assimilating data from scientific simulations as well as from measurements streamed from sensor networks, such as radars and satellites. For example, as part of the Flagship 2020 (post-K) supercomputer project of Japan, RIKEN is investigating the feasibility of a highly accurate weather forecasting system that would provide a real-time outlook for severe guerrilla rainstorms. One of the main performance bottlenecks of this application is the lack of efficient communication among workflow components, which currently takes place over the parallel file *** this paper, we present an initial study of a direct communication framework designed for complex workflows that eliminates unnecessary file I/O among components. Specifically, we propose an I/O arbitration layer that provides direct parallel data transfer (both synchronous and asynchronous) among job components that rely on the netCDF interface for performing I/O operations. Our solution requires only minimal modifications to application code. Moreover, we propose a configuration file-based approach that allows users to specify the desired data transfer pattern among workflow components, offering a general solution for different application contexts. We present a preliminary evaluation of the proposed framework on the K Computer (running on up to 4800 compute nodes) using RIKEN's experimental weather forecasting workflow as a case study.

关键词： asynchronous transfer big data processing customizability netCDF parallel direct data transfer real time

来源：评论

学校读者我要写书评

暂无评论

Research on the Access Control and Communications Security Research Based on Hadoop big data processing

Research on the Access Control and Communications Security R...

引用

2016 4th International Conference on Machinery,Materials and Computing Technology(ICMMCT 2016)

作者： Yanhua Hu Department of Electrical and Computer Engineering Lushan College of Guangxi University of Science and Technology

ISBN: (纸本)9781510821934

Hadoop is an open cloud computing platform. Cloud computing is a new computing model. It derives from distributed computing, grid computing etc, integrating available valid resources and providing computing resources, storage resource for user as service. The object faced by cloud computing could be enterprise, government, personal and so on, but Cloud computing is transparent to user who could not know the detail operation of the step in it, which can give user a feeling of high performance computing and large storage space and it begin to blend in human life. However, Cloud security issues have been blocking the development of cloud computing. To the majority of users, security is more important than computing power.

关键词： Hadoop big data processing Access Control Communications Security

来源：评论

学校读者我要写书评

暂无评论

Research on Key Technologies of Medical Cloud for Medical big data processing

Research on Key Technologies of Medical Cloud for Medical Bi...

引用

作者： Jimin Gao Department of Artificial Intelligence Shenzhen Polytechnic

Facing the demands of medical and health services and the opportunities and challenges of big data, traditional medical informationization cannot effectively cope with them. This paper designs a three-tier model of medical cloud for medical data processing, which integrates "cloud architecture" with the existing hospital information system perfectly by using private cloud of hospital, and establishes a new medical resource projection mode of "cloud service" hospital information system as service by using medical community cloud. And the medical data processing system can effectively use "cloud computing" to process and utilize medical data. With cloud computing as the brain, big data as the soul, hospital informatization as the carrier, based on cloud computing and big data core technology and concept, the application of medical cloud and big data is realized in stages under this model.

关键词： Key Technologies Medical Cloud big data processing

来源：评论

学校读者我要写书评

暂无评论

Function of big data processing on E-commerce Development under Cloud Computing Environment

Function of Big Data Processing on E-commerce Development un...

引用

2015 2nd International Conference on Education,Management and Information Technology(ICEMIT 2015)

作者： Xianwei Feng Jiangsu Institute of Commerce

E-commerce highlights overall tendency of virtualization and shows transparency. With the help of efficiency advantage, it rapidly extends data processing and data transmission under network path, and occupies a main market. Cloud computing is classified into a novel mode and shows leap-frog development by depending network. Seeing from currants status, information loaded by network progressively increases, and numerical treatment under traditional path is difficult to comply with requirements. In cloud computing environment, it is required to analyze the influence of big data analysis and processing on e-commerce development and propose application strategies.

关键词： Cloud computing environment big data processing E-Commerce development Specific function

来源：评论

学校读者我要写书评

暂无评论

Analysis of the effect of weather determinants on lodging demands using big data processing

Analysis of the effect of weather determinants on lodging de...

引用

IEEE/ACS International Conference on Computer Systems and Applications

作者： Seungwoo Jeon Bonghee Hong Hyeongsoon Im Dept. of Electrical and Computer Engineering Pusan National University Busan South Korea

ISBN: (纸本)9781509004799

Weather conditions determine the variation in the floating population at tourist sites and, consequently, the number of persons who stay overnight;therefore, it is necessary to select those determinants that are most influenced by the weather. Our aim is to construct a multiple linear regression model and system based on big data processing tools to process large volumes of meteorological and population data with the ultimate goal of supporting predicted accommodation congestion with high prediction accuracy. First, we transform the floating population data into accommodation congestion by estimating the resident population during periods of high demand. Second, we conduct a multiple linear regression analysis with variable removal steps and a residual analysis. Moreover, to verify our regression model, we use two prediction accuracy measures: the mean absolute percentage error and the R-squared value. The final part of the paper describes the construction of a big data processing system that was built to compute the series of prediction operations.

关键词： Accommodation congestion big data processing Floating population data Multiple linear regression model Weather determinants

来源：评论

学校读者我要写书评

暂无评论

Research and Application of Spark Platform on big data processing in Intelligent Agriculture of Jilin Province

Research and Application of Spark Platform on Big Data Proce...

引用

The International Conference on Intelligent Agriculture 2017(ICIA2017)

作者： Siwei Fu Guifen Chen Shan Zhao Enze Xiao Jilin Agricultural University

Aiming at the demand of real-time massive data processing of Intelligent Agriculture in Jilin Province, this paper studies the big data processing of Intelligent Agriculture in Jilin Province based on Spark platform by acquiring real-time data through monitoring platform. This study first conducted the performance comparison experiment of Hadoop and Spark data processing platform, then used the Spark distributed cluster computing platform, real-time processing the big data of monitoring area. The experimental results show that the Spark platform speeds up 11.4 times faster than the Hadoop platform in the case of 100 million data sizes;and based on the Spark platform for real-time processing of big data intelligent agricultural monitoring network, not only provides memory calculations to reduce IO overhead, but also the results are faster and more accurate. The research results provide strong support for the implementation of precision agriculture technology in intelligent agriculture.

关键词： Spark big data processing MapReduce Intelligent Agriculture in Jilin Province

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：