检索结果-内蒙古大学图书馆

Proceedings of the 2025 International Conference on big data and Informatization Education

作者： Naiyu Liu The People's Bank of China Jilin Province Branch Changchun Jilin China

ISBN: (纸本)9798400714405

A new optimization framework is put forward, which is based on compliance and the quantitative index of legal responsibility. As for the method, the algorithm makes dynamic adjustment to the process by embedding the weight function of the compliance factor, so as to reach the equilibrium of law compliance and data processing efficiency. In the experiment section, we use a common dataset that includes many kinds of data to validate, and sets up a number of key performance metrics, such as compliance, processing efficiency, and computation cost. The simulation results show that the proposed algorithm can increase the compliance rate by 17% and the processing efficiency by 23%, and the computational cost is reduced by 15%. The results of the experiment show that the algorithm is stable and reliable. The results indicate that the new algorithm can not only meet the requirements of legal responsibility and compliance, but also improve the performance of large scale data processing.

关键词： big data processing

来源：评论

学校读者我要写书评

暂无评论

Low Latency big data processing without Prior Information

引用

IEEE TRANSACTIONS ON CLOUD COMPUTING 2021年第4期9卷 1521-1534页

作者： Hu, Zhiming Li, Baochun Qin, Zheng Goh, Rick Siow Mong Univ Toronto Dept Elect & Comp Engn Toronto ON M5S 2E4 Canada ASTAR Inst High Performance Comp Singapore 138632 Singapore

Job scheduling plays an important role in improving the overall system performance in big data processing frameworks. Simple job scheduling policies, such as Fair and FIFO scheduling, do not consider job sizes and may degrade the performance when jobs of varying sizes arrive. More elaborate job scheduling policies make the convenient assumption that jobs are recurring, and complete information about their sizes is available from their prior runs. In this paper, we design and implement an efficient and practical job scheduler for big data processing systems to achieve better performance even without prior information about job sizes. The superior performance of our job scheduler originates from the design of multiple level priority queues, where jobs are demoted to lower priority queues if the amount of service consumed so far reaches a certain threshold. In this case, jobs in need of a small amount of service can finish in the topmost several levels of queues, while jobs that need a large amount of service to complete are moved to lower priority queues to avoid head-of-line blocking. Our new job scheduler can effectively mimic the shortest job first scheduling policy without knowing the job sizes in advance. To demonstrate its performance, we have implemented our new job scheduler in YARN, a popular resource manager used by Hadoop/Spark, and validated its performance with experiments in both real testbeds including Amazon EC2 and large-scale trace-driven simulations. Our experimental and simulation results have strongly confirmed the effectiveness of our design: our new job scheduler can reduce the average job response time of the Fair scheduler by up to 45 percent and achieve better fairness at the same time.

关键词： Job scheduling big data processing multilevel feedback queue

来源：评论

学校读者我要写书评

暂无评论

Analysis of Effective E-Commerce Coordination big data processing Strategies under Infinite Deep Neural Network Topology

Analysis of Effective E-Commerce Coordination Big Data Proce...

引用

Asia-Pacific Conference on Communications Technology and Computer Science (ACCTCS)

作者： Zhang, Ailing Yao, Fuxiao Shandong Vocat & Tech Univ Int Studies Rizhao 276800 Shandong Peoples R China

ISBN: (纸本)9781728176499

This paper researches and analyses the effective e-commerce coordination big data processing strategy through the infinite-depth neural network topology. This paper firstly proposes a neural network training model Neural Network-Storm (NN-S) based on Storm streaming distributed architecture, which decomposes the neural network training task into multiple computing units by data-parallel method, the parameters are updated synchronously after the training of a single batch of data is completed. In the Storm architecture, a Zookeeper network is used for multi-server distributed deployment. The training results show that the NN-S model can significantly improve the training speed of neural networks. At the same time, the NN-S architecture can quickly recover from node failures and network resource scheduling abnormalities with strong robustness. In this paper, we investigate the streaming-based distributed neural network training and design a Storm-based distributed neural network training model and optimized training algorithms, which are of reference significance for distributed neural network training.

关键词： infinite depth neural network topology effective e-conunerce coordination big data processing strategy analysis

来源：评论

学校读者我要写书评

暂无评论

Cellular automata-based MapReduce design: Migrating a big data processing model from Industry 4.0 to Industry 5.0

引用

e-Prime - Advances in Electrical Engineering, Electronics and Energy 2024年 8卷

作者： Mitra, Arnab Department of Information Technology Techno International New Town West Bengal Kolkata 700156 India

A successful deployment of Industry 5.0 is significantly dependent on the synergetic integration of several advanced technologies such as big data processing, Artificial Intelligence (AI) integration, and several effective digitization techniques that emphasize the uses of Robotics, Internet of Things (IoTs), Cloud Computing, etc. with active participations from human workers. Several researchers have explored the importance of big data processing in view of Industry 4.0 as it facilitated enhanced production at any smart manufacturing line by ensuring efficient process management, typically involving big data processing. Researchers presented several MapReduce-based data processing models at smart manufacturing lines to facilitate big data processing. Among several others, the Elementary Cellular Automata (ECAs)-based MapReduce model was introduced as an energy-efficient low-cost model for big data processing in view of the Industry 4.0 scenario. In the present research, an investigation is further proposed to explore the true potential (if any) for the ECAs-based MapReduce model with reference to available several MapReduce models, to migrate an existing big data processing model from Industry 4.0 into the future i.e., Industry 5.0. Investigation results as achieved from the comparison among several MapReduce models and further examinations about the inherent quality of shuffle in those MapReduce blocks, explore the inherent advantages of ECAs-based MapReduce model towards its choice for big data processing in Industry 5.0 scenario. © 2024 The Author(s)

关键词： big data processing Cyber-physical systems (CPS) Elementary cellular automata (ECAs) Green computing Industry 4.0 Industry 5.0 MapReduce model Sustainability

来源：评论

学校读者我要写书评

暂无评论

Argus: Efficient Job Scheduling in RDMA-assisted big data processing 35

Argus: Efficient Job Scheduling in RDMA-assisted Big Data Pr...

引用

35th IEEE International Parallel and Distributed processing Symposium (IPDPS)

作者： Wu, Sijie Chen, Hanhua Wang, Yonghui Jin, Hai Huazhong Univ Sci & Technol Natl Engn Res Ctr Big Data Technol & Syst Cluster & Grid Comp Lab Serv Comp Technol & Syst LabSch Comp Sci & Techn Wuhan 430074 Peoples R China

ISBN: (纸本)9781665440660

Efficient job scheduling is an important and challenging issue in big data processing systems. Traditional designs commonly give priority to data locality during scheduling and follow a network-optimized principle to avoid costly data moving across the network. The emergence of the high-performance Remote Direct Memory Access (RDMA) network brings new opportunities for big data processing systems. However, the existing RDMA-assisted designs ignore the dependency among stages during scheduling and this can result in unsatisfied system efficiency. In this work, we propose Argus, a novel RDMA-assisted job scheduler which achieves high resource utilization by fully exploiting the structure feature of stage dependency. Argus prioritizes the stages whose completion can enable more schedulable stages. We implement Argus on top of RDMA-Spark, and conduct comprehensive experiments to evaluate the performance using large-scale traces collected from real-world systems. Results show that compared to state-of-the-art designs, Argus reduces the job completion time and makespan by 38% and 31%, respectively.

关键词： Remote direct memory access big data processing job scheduling

来源：评论

学校读者我要写书评

暂无评论

A two-level formal model for big data processing programs

引用

SCIENCE OF COMPUTER PROGRAMMING 2022年 215卷 102764-102764页

作者： de Souza Neto, Joao Batista Moreira, Anamaria Martins Vargas-Solar, Genoveva Musicante, Martin A. Univ Fed Rio Grande do Norte Dept Informat & Appl Math DIMAp Natal RN Brazil Fed Ctr Technol Educ Minas Gerais Dept Informat Management & Design DIGD DV Divinopolis Brazil Univ Fed Rio de Janeiro Inst Comp IC Rio De Janeiro Brazil LIRIS French Council Sci Res CNRS Lyon France

This paper proposes a model for specifying data flow-based parallel data processing programs agnostic of target big data processing frameworks. The paper focuses on the formal abstract specification of non-iterative and iterative programs, generalizing the strategies adopted by data flow big data processing frameworks. The proposed model relies on Monoid Algebra and Petri Nets to abstract big data processing programs in two levels: a higher level representing the program data flow and a lower level representing data transformation operations (e.g., filtering, aggregation, join). We extend the model for data processing programs proposed in [1], for modeling iterative data processing programs. The general specification of these programs implemented by data flow-based parallel programming models is essential given the democratization of iterative and greedy big data analytics algorithms. Indeed, these algorithms call for revisiting parallel programming models to express iterations. The paper gives a comparative analysis of the iteration strategies proposed by Apache Spark, DryadLINQ, Apache Beam, and Apache Flink. It discusses how the model achieves to generalize these strategies. (c) 2021 Elsevier B.V. All rights reserved.

关键词： big data processing data flow programming models Petri nets Monoid algebra

来源：评论

学校读者我要写书评

暂无评论

Research on Improved Shallow Neural Network big data processing Model based on Gaussian Function

引用

INTERNATIONAL JOURNAL OF ADVANCED COMPUTER SCIENCE AND APPLICATIONS 2022年第10期13卷 566-573页

作者： Fu, Lifang Henan Univ Anim Husb & Econ Fac Sci Zhengzhou 450044 Peoples R China

The application of the current new generation communication technology is gradually diversified, and the global Internet users are increasing, leading some large enterprises to increasingly rely on faster and more efficient big data processing technology. In order to solve the shortcomings of the current big data processing algorithms, such as slow computing speed, computing accuracy to be improved, and poor online real-time learning ability, this research combines incremental learning and sliding window ideas to design two improved radial basis function (RBF) neural network algorithms with Gaussian function as the kernel function. The Duffing equation example and the data of "Top 100 single products for Taobao search glasses sales" were used to verify the performance of the design algorithm. The experimental results of Duffing equation example show that when the total sample is 100000, the mean square errors of IOL, SWOL, SVM and ResNet50 algorithms are 1.86e-07, 1.59e-07, 3.37e-07 and 2.67e-07 respectively. The experimental results of the data set of "Top 100 SKUs for Taobao Search Glasses Sales" show that when the number of samples in the test set is 800, the root mean square errors of IOL, SWOL, SVM and ResNet50 algorithms are 0.0060, 0.0056, 0.0069 and 0.0073 respectively. This shows that the RBF online learning algorithm designed in this study, which integrates sliding windows, has a stronger comprehensive ability to process big data, and has certain application value for improving the accuracy of online data based commodity recommendation in e -commerce and other industries.

关键词： Gaussian function RBF big data processing incremental learning sliding window

来源：评论

学校读者我要写书评

暂无评论

Enhancement of Infrared Images Using Super Resolution Techniques Based on big data processing

引用

MULTIMEDIA TOOLS AND APPLICATIONS 2020年第9-10期79卷 5671-5692页

作者： Abd El-Samie, Fathi E. Ashiba, Huda I. Shendy, H. Mansour, Hala M. Ahmed, Hossameldin M. Taha, Taha E. Dessouky, Moawad I. Elkordy, Mohamed F. Abd-Elnaby, Mohammed El-Fishawy, Adel S. Menoufia Univ Dept Elect & Elect Commun Engn Fac Elect Engn Menoufia 32952 Egypt Bilbis Higher Inst Engn Dept Elect & Elect Commun Bilbis Sharqia Egypt Benha Univ Shoubra Fac Engn Dept Elect & Elect Commun Banha Egypt Taif Univ Dept Comp Engn Coll Comp & Informat Technol Al Hawiya 21974 Saudi Arabia

This paper presents a super-resolution (SR) technique for enhancement of infrared (IR) images. The suggested technique relies on the image acquisition model, which benefits from the sparse representations of low-resolution (LR) and high-resolution (HR) patches of the IR images. It uses bicubic interpolation and minimum mean square error (MMSE) estimation in the prediction of the HR image with a scheme that can be interpreted as a feed-forward neural network. The suggested algorithm to overcome the problem of having only LR images due to hardware limitations is represented with a big data processing model. The performance of the suggested technique is compared with that of the standard regularized image interpolation technique as well as an adaptive block-by-block least-squares (LS) interpolation technique from the peak signal-to-noise ratio (PSNR) perspective. Numerical results reveal the superiority of the proposed SR technique.

关键词： Single-image super-resolution Image interpolation IR images Neural networks big data processing

来源：评论

学校读者我要写书评

暂无评论

big data processing With Minimal Delay and Guaranteed data Resolution in Disaster Areas

引用

IEEE TRANSACTIONS ON VEHICULAR TECHNOLOGY 2019年第4期68卷 3833-3842页

作者： Wang, Junbo Sato, Koichi Guo, Song Chen, Wuhui Wu, Jie Univ Aizu Sch Comp Sci & Engn Aizu Wakamatsu Fukushima 9658580 Japan Univ Aizu Aizu Wakamatsu Fukushima 9658580 Japan Hong Kong Polytech Univ Dept Comp Hong Kong Peoples R China Sun Yat Sen Univ Gunagzhou 510275 Peoples R China Temple Univ Dept Comp & Informat Sci Philadelphia PA 19122 USA

big data analysis is very important to support rescue activities when natural disaster happens, through understanding various situations, such as power/water outage regions. The traditional way to process big data is based on high-performance computation/storage resources in a cloud center. However, this is hard to be guaranteed in a disaster scenario due to destruction of communication infrastructure. Meanwhile, high latency between local sensing devices and cloud center sets a big obstacle enabling a near real-time big data analysis. On the other hand, movable base station, such as vehicle-based movable & deployable ICT resource unit (MDRU) developed by NTT, is a possible solution to reconstruct an emergency communication network and process data at the edge sites with reduced data transmission time. In this paper, we study the optimal overall delay in a fog/edge-computing platform constructed by vehicle-based MDRUs with guaranteed data resolution. We formalize the problem as a mixed-integer nonlinear program, which is a well-known NP-hard problem, and then relax the original problem to an mixed integer linear programing (MILP). Finally, we propose a two-stage heuristic algorithm to solve it in a time-efficient manner. Through evaluation, the effectiveness of the proposed heuristic approach has been validated in terms of minimizing overall delay with sufficient given data resolutions.

关键词： big data processing fog/edge computing data resolution disaster scenarios

来源：评论

学校读者我要写书评

暂无评论

Modeling big data processing Programs 23rd

Modeling Big Data Processing Programs

引用

23rd Brazilian Symposium on Formal Methods (SBMF)

作者： de Souza Neto, Joao Batista Moreira, Anamaria Martins Vargas-Solar, Genoveva Musicante, Martin A. Univ Fed Rio Grande do Norte Dept Informat & Appl Math DIMAp Natal RN Brazil Univ Fed Rio de Janeiro Comp Sci Dept DCC Rio De Janeiro Brazil Univ Grenoble Alpes LIG LAFMIA Grenoble INP CNRS Grenoble France

ISBN: (纸本)9783030638818;9783030638825

We propose a new model for data processing programs. Our model generalizes the data flow programming style implemented by systems such as Apache Spark, DryadLINQ, Apache Beam and Apache Flink. The model uses directed acyclic graphs (DAGs) to represent the main aspects of data flow-based systems, namely, operations over data (filtering, aggregation, join) and program execution, defined by data dependence between operations. We use Monoid Algebra to model operations over distributed, partitioned datasets and Petri Nets to represent the data flow. This approach allows the data processing program specification to be agnostic of the target big data processing system. As a first application of the model, we used it to formalize mutation operators for the application of mutation testing in big data processing programs. The testing tool TRANSMUT-Spark implement these operators.

关键词： big data processing data flow programming models Petri Nets Monoid Algebra

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：