检索结果-内蒙古大学图书馆

SDAC: A model for analysis of the execution semantics of data processing framework in cloud

COMPUTER LANGUAGES SYSTEMS & STRUCTURES 2018年 54卷 406-426页

作者： Zhou, Wenbo Liu, Lei Zhang, Peng Lu, Shuai Li, Jingyao Jilin Univ Coll Comp Sci & Technol Changchun 130012 Jilin Peoples R China Jilin Univ Key Lab Symbol Computat & Knowledge Engn Minist Educ Changchun 130012 Jilin Peoples R China Jilin Univ Coll Software Changchun 130012 Jilin Peoples R China Jilin Univ Coll Math Changchun 130012 Jilin Peoples R China

data processing framework is at the core of cloud computing. Firstly, to further specify the execution flow of data processing framework in cloud environments, this paper proposes SDAC (semantic distributed abstract configuration), a formal model introduced from the view of model definition and execution semantics. Then, to demonstrate the reasonability, the paper compares the proposed approach with the Aeolus model since they both concern configurations in cloud environments. And to evaluate the model, a MapReduce case and some applications of SDAC (i.e. the interpretation of fault tolerance and performance optimization) are presented. Finally, the future work about using SDAC to analyse other mechanisms in cloud are discussed. (C) 2018 Elsevier Ltd. All rights reserved.

关键词： Cloud computing data processing framework Execution semantics Fault tolerance Performance optimization

来源：评论

学校读者我要写书评

暂无评论

A Modular framework for data processing at the Edge: Design and Implementation

引用

SENSORS 2023年第17期23卷 7662-7662页

作者： Urblik, Lubomir Kajati, Erik Papcun, Peter Zolotova, Iveta Tech Univ Kosice Fac EE & Informat Dept Cybernet & Artificial Intelligence Kosice 04200 Slovakia

There is a rapid increase in the number of edge devices in IoT solutions, generating vast amounts of data that need to be processed and analyzed efficiently. Traditional cloud-based architectures can face latency, bandwidth, and privacy challenges when dealing with this data flood. There is currently no unified approach to the creation of edge computing solutions. This work addresses this problem by exploring containerization for data processing solutions at the network's edge. The current approach involves creating a specialized application compatible with the device used. Another approach involves using containerization for deployment and monitoring. The heterogeneity of edge environments would greatly benefit from a universal modular platform. Our proposed edge computing-based framework implements a streaming extract, transform, and load pipeline for data processing and analysis using ZeroMQ as the communication backbone and containerization for scalable deployment. Results demonstrate the effectiveness of the proposed framework, making it suitable for time-sensitive IoT applications.

关键词： containerization edge computing data processing framework Kubernetes Docker

来源：评论

学校读者我要写书评

暂无评论

A General-Purpose Middleware System for Edge-side data processing 9

A General-Purpose Middleware System for Edge-side Data Proce...

引用

9th IEEE International Conference on Fog and Mobile Edge Computing (FMEC)

作者： Satoh, Ichiro Natl Inst Informat 2-1-2 HitotsubashiChiyoda Ku Tokyo 1018430 Japan

ISBN: (纸本)9798350366495;9798350366488

Edge computing, including fog computing, processes data measured by their sensing systems. Such data often contain noise, losses, and non-canonical representations, necessitating preliminary processing to enhance data quality and convert it into certain formats. While data processing tasks have been centralized in data centers so far, edge computing, including fog computing, decentralizes these tasks to edge nodes. This paper presents a data processing framework for edge computing. The contribution of the framework is to adapt existing popular data processing approaches, originally designed for use in data centers consisting of high-performance servers connected through wide-band networks, for use in edge computing. The framework hides network configurations and node failures, enabling developers to focus solely on data processing tasks without worrying about underlying infrastructural complexities. This paper details the framework's architecture, discusses its integration with edge and fog computing environments, and evaluates its performance, demonstrating its superior efficiency and effectiveness in handling edge-based data processing challenges.

关键词： data processing framework MapReduce edge computing

来源：评论

学校读者我要写书评

暂无评论

Tree Roots Reconstruction framework for Accurate Positioning in Heterogeneous Soil

引用

IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING 2022年 15卷 1912-1925页

作者： Luo, Wenhao Lee, Yee Hui Sun, Hai-Han Ow, Genevieve Yusof, Mohamed Lokman Mohd Yucel, Abdulkadir C. Nanyang Technol Univ Sch Elect & Elect Engn Singapore 639798 Singapore Natl Pk Board Singapore 259569 Singapore

Ground-penetrating radar has recently found wide application in the underground imaging of tree roots. However, ignoring the random and complex nature of the heterogeneous soil and assuming the soil's relative permittivity constant throughout the survey region may yield an inaccurate tree root positioning. Meanwhile, the incompatible relative soil permittivity results in low image quality of the roots reconstruction. Furthermore, the soil's spatial heterogeneity introduces unwanted environmental clutter in the mapping of the tree root. A data processing framework is proposed to address these issues for retrieving the tree roots in heterogeneous soil environments. The proposed framework combines four techniques to be applied consecutively: First, a hyperbola extraction method based on a column-connection clustering algorithm is used to extract individual hyperbolae in B-scans, eliminate mutual influence in the process, and suppress noise. Second, an improved Hough transform technique is adopted to estimate the equivalent permittivity of each root's surrounding soil environment for each extracted hyperbola. Third, individual root restoration is done by transferring each hyperbola to a spot using its corresponding soil equivalent permittivity. Finally, individually restored features are combined in the final image. The images obtained via the proposed framework show a well reconstructed two-dimensional tree roots scenario. The applicability and the effectiveness of the proposed framework have been demonstrated through numerical simulations and fieldmeasurements.

关键词： Soil Permittivity Feature extraction Image reconstruction Vegetation Clustering algorithms Reflection data processing framework ground-penetrating radar (GPR) heterogeneous soil two-dimensional (2-D) tree roots reconstruction

来源：评论

学校读者我要写书评

暂无评论

Redis-Based Messaging Queue and Cache-Enabled Parallel processing Social Media Analytics framework

引用

COMPUTER JOURNAL 2022年第4期65卷 843-857页

作者： Singh, Ravindra Kumar Verma, Harsh Kumar Dr BR Arnbedkar Natl Inst Technol Dept Comp Sci & Engn GT Rd Jalandhar 144011 Punjab India

The extensive usage of social media polarity analysis claims the need for real-time analytics and runtime outcomes on dashboards. In data analytics, only 30% of the time is consumed in modeling and evaluation stages and 70% is consumed in data engineering tasks. There are lots of machine learning algorithms to achieve a desirable outcome in prediction points of view, but they lack in handling data and their transformation so-called data engineering tasks, and reducing its time remained still challenging. The contribution of this research paper is to encounter the mentioned challenges by presenting a parallelly, scalable, effective, responsive and fault-tolerant framework to perform end-to-end data analytics tasks in real-time and batch-processing manner. An experimental analysis on Twitter posts supported the claims and signifies the benefits of parallelism of data processing units. This research has highlighted the importance of processing mentioned URLs and embedded images along with post content to boost the prediction efficiency. Furthermore, this research additionally provided a comparison of naive Bayes, support vector machines, extreme gradient boosting and long short-term memory (LSTM) machine learning techniques for sentiment analysis on Twitter posts and concluded LSTM as the most effective technique in this regard.

关键词： social media analytics real time analytics message broker parallel processing data processing framework Elasticsearch Kibana

来源：评论

学校读者我要写书评

暂无评论

SwiftFrame: Developing Low-latency Near Real-time Response framework 15

SwiftFrame: Developing Low-latency Near Real-time Response F...

引用

15th International Conference on Computing Communication and Networking Technologies, ICCCNT 2024

作者： Saipranith, Swargam Singh, Anand Kumar Agrawal, Neha Chilumula, Sruthi Iiit Sri City Computer Science and Engineering AP Chittoor517646 India

ISBN: (纸本)9798350370249

In today's era, handling of big data poses significant challenges due to its massive volume and diverse formats. A rapid processing of such diverse data formats, spanning vast volumes, is essential for delivering swift and effective real-time responses. This paper proposes a near real-time response framework named as SwiftFrame for low-latency, high-speed applications. Leveraging Docker, the proposed architecture seamlessly deploys and manages Apache Kafka, Apache Zookeeper, and PostgreSQL. Apache Flink efficiently processes data streams from Kafka, orchestrates tasks, and stores results in PostgreSQL. Grafana enables real-time visualization, offering a comprehensive monitoring interface. The proposed architecture, SwiftFrame, is designed to handle modern data processing requirements, ensures low latency and high throughput, suitable for diverse high-speed applications. In the experimental setup, four different scenarios have been considered and each scenario has repeated 5 times. The experimental results show that the proposed architecture outperforms in terms of latency and throughput achieving an average latency of 0.13 sec and an average throughput of 8.7 records per second. The future research directions of this paper are also highlighted. © 2024 IEEE.

关键词： Big-data data processing framework Docker Low-latency framework Near realtime Stream data processing

来源：评论

学校读者我要写书评

暂无评论

Application Integration framework for Large Language Models 5

Application Integration Framework for Large Language Models

引用

5th International Conference on Artificial Intelligence and data Sciences, AiDAS 2024

作者： Ho, Joe Ee Ooi, Boon Yaik Westner, Markus Universiti Tunku Abdul Rahman Department of Computer Science Perak Kampar Malaysia Oth Regensburg Faculty of Computer Science & Mathematics Regensburg Germany

ISBN: (纸本)9798331528553

Large Language Models (LLMs) have unlocked new opportunities in processing non-structured information. However, integrating LLM into conventional applications poses challenges due to their non-deterministic nature. This paper introduces a framework designed to effectively integrate LLM into intermediate modules by ensuring more consistent and reliable outputs. The framework includes three key components: the Sieve, which captures and retries processing of incorrect outputs;the Circuit Breaker, which stops processing persistently incorrect outputs;and the Optimizer, which enhances processing efficiency by combining inputs into single prompts. Experimental results employing structured methodology demonstrate the framework's effectiveness, achieving significant improvements a 71.05% reduction in processing time and an 82.97% reduction in token usage while maintaining high accuracy. The proposed framework, agnostic to specific LLM implementations, aids the integration of LLMs into diverse applications, enhancing automation and efficiency in fields such as finance, healthcare, and education. © 2024 IEEE.

关键词： AI integration automation data pipeline optimization data processing framework error handling token efficiency

来源：评论

学校读者我要写书评

暂无评论

A data processing framework for IoT based Online Monitoring System

A Data Processing Framework for IoT based Online Monitoring ...

引用

IEEE 17th International Conference on Computer Supported Cooperative Work in Design (CSCWD)

作者： Zhou, Zhuo Liu, Min Zhang, Feng Bai, Li Shen, Weiming School of Electronic and Information Engineering Tongji University Shanghai Lixin University of Commerce

ISBN: (纸本)9781467360852;9781467360845

Online monitoring system for continuous casting equipment is established based on IOT (Internet of things) sensing technology and communication technology. As the system contains a variety of sensor types and data transmission protocols, it will lead to a large amount of heterogeneous data and the data is difficult to integrate with applications in upper layer. A data processing framework is introduced into the system to deal with such problems. The framework focuses on protocol conversion, data processing methods and integration with applications in upper layer. Finally, application in online monitoring system proved the validity of the framework.

关键词： Online monitoring system IOT data processing framework

来源：评论

学校读者我要写书评

暂无评论

Effective Parallel processing Social Media Analytics framework

引用

JOURNAL OF KING SAUD UNIVERSITY-COMPUTER AND INFORMATION SCIENCES 2022年第6期34卷 2860-2870页

作者： Singh, Ravindra Kumar Verma, Harsh Kumar Dr BR Ambedkar Natl Inst Technol Dept Comp Sci & Engn Jalandhar Punjab India

The widespread adoption of opinion mining and sentiment analysis in higher cognitive processes encourages the need for real-time processing of social media data to capture insights about user's sentiment polarity, user's opinion, and current trends of the domain. In recent years lots of research were conducted and various machine learning algorithms were developed around the processing of data to achieve higher accuracy while reducing the processing time is still challenging. Big data technologies came unraveled these challenges but they have their own set of complexities along with having hardware deadweight on the system. The contribution of this research paper is to touch upon the mentioned challenges by presenting a climbable, instantaneous and fault-tolerant framework to process real-time data to extract hidden insights within it while not bearing any additional overhead of big data technologies. This framework is versatile enough to support batch processing along with real-time data streams in parallel and distributed environments. Experimental results concluded that 4-threaded parallel architecture of the framework performs at 2X speed compared to single-threaded architecture and shared URLs, embedded Images and Author's meta info it boosting the tweets prediction. Moreover, this research additionally provides a comparison of Support Vector Machines (SVM), Light GBM (LGBM) and Long Short Term Memory (LSTM) supervised machine learning techniques for sentiment analysis and concluded LGBM is the most effective model. (C) 2020 The Authors. Published by Elsevier B.V. on behalf of King Saud University.

关键词： Social media analytics Real time analytics MongoDB Redis Python-dash Visualization Message broker Cache management Pre-processing block Parallel processing data processing framework

来源：评论

学校读者我要写书评

暂无评论

Are Novel Deep Learning Methods Effective for Fault Diagnosis?

引用

IEEE TRANSACTIONS ON RELIABILITY 2024年

作者： Jiang, Dongnian He, Chenxian Chen, Zeyang Zhao, Jinjiang Lanzhou Univ Technol Sch Elect & Informat Engn Lanzhou 730050 Gansu Peoples R China Jiangnan Univ Sch Internet Things Wuxi 214122 Jiangsu Peoples R China

In recent years, deep learning has become the standard approach for fault diagnosis in mechanical equipment, with models becoming increasingly complex and large in scale. Through a thorough review and analysis of existing literature, we found that while many studies report performance improvements with the latest models, there has been no comparison of state-of-the-art (SOTA) methods. This gap is primarily due to two factors: first, the wide variation in the quality and nature of fault diagnosis data leads to significant performance fluctuations across different datasets;second, the diverse preprocessing methods employed make it challenging to compare models objectively. For instance, while deep learning has demonstrated high accuracy in bearing fault diagnosis, variations in vibration signal processing methods often skew the evaluation of model performance. To address these issues and evaluate the true performance of the latest deep learning models for fault diagnosis, this article establishes a unified data processing framework that ensures fair performance comparisons across models. Using this framework, we reproduce eight SOTA deep learning models and assess their effectiveness on three publicly available bearing datasets. Additionally, we design three benchmark models to quantify performance differences. The experimental results highlight that current deep learning-based fault diagnosis methods still face significant challenges in real-world applications. Finally, the future research directions in this domain are given.

关键词： Fault diagnosis Deep learning Mathematical models Vibrations Adaptation models Feature extraction Convolution Computational modeling Attention mechanisms Training Baseline models comparison of models data processing framework mechanical fault diagnosis performance comparison state-of-the-art (SOTA) model

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：