检索结果-内蒙古大学图书馆

real-time data processing for serial crystallography experiments

IUCRJ 2025年第1期12卷 97-108页

作者： White, Thomas Schoof, Tim Yakubov, Sergey Tolstikova, Aleksandra Middendorf, Philipp Karnevskiy, Mikhail Mariani, Valerio Henkel, Alessandra Klopprogge, Bjarne Hannappel, Juergen Oberthuer, Dominik Aquino, Ivan De Gennaro Egorov, Dmitry Munke, Anna Sprenger, Janina Pompidor, Guillaume Taberman, Helena Gruzinov, Andrey Meyer, Jan Hakanpaeae, Johanna Gasthuber, Martin Deutsch Elektronen Synchrotron DESY Hamburg Germany Deutsch Elektronen Synchrotron DESY Ctr Data & Comp Nat Sci CDCS Hamburg Germany Deutsch Elektronen Synchrotron DESY Ctr Free Electron Laser Sci CFEL Hamburg Germany SLAC Natl Accelerator Lab Linac Coherent Light Source Menlo Pk CA USA Hamburg Ctr Ultrafast Imaging Hamburg Germany Oak Ridge Natl Lab ORNL Oak Ridge TN USA

We report the use of streaming data interfaces to perform fully online data processing for serial crystallography experiments, without storing intermediate data on disk. The system produces Bragg reflection intensity measurements suitable for scaling and merging, with a latency of less than 1 s per frame. Our system uses the CrystFEL software in combination with the ASAP::O data framework. In a series of user experiments at PETRA III, frames from a 16 megapixel Dectris EIGER2 X detector were searched for peaks, indexed and integrated at the maximum full-frame readout speed of 133 frames per second. The computational resources required depend on various factors, most significantly the fraction of non-blank frames ('hits'). The average single-thread processing time per frame was 242 ms for blank frames and 455 ms for hits, meaning that a single 96-core computing node was sufficient to keep up with the data, with ample headroom for unexpected throughput reductions. Further significant improvements are expected, for example by binning pixel intensities together to reduce the pixel count. We discuss the implications of real-time data processing on the 'data deluge' problem from recent and future photon-science experiments, in particular on calibration requirements, computing access patterns and the need for the preservation of raw data.

关键词： macromolecular crystallography serial crystallography X-ray crystallography real-time data processing

来源：评论

学校读者我要写书评

暂无评论

real-time data processing Pipeline for Trigger Readout Board-Based data Acquisition Systems

引用

IEEE TRANSACTIONS ON NUCLEAR SCIENCE 2022年第7期69卷 1765-1772页

作者： Malige, A. Korcyl, G. Firlej, M. Fiutowski, T. Idzik, M. Korzeniak, B. Lalik, R. Misiak, A. Molenda, A. Moron, J. Rathod, N. Salabura, P. Smyrski, J. Swientek, K. Wintz, P. Jagiellonian Univ Fac Phys Astron & Appl Comp Sci Marian Smoluchowski Inst Nucl Phys PL-30348 Krakow Poland Akad Gorniczo Hutniczej Wydzial Fizyki & Informatyki Stosowanej PL-30059 Krakow Poland Forschungszentrum Nucl Phys Inst D-52425 Julich Germany

Large-scale physics experiments running at high interaction rates place a high demand on the data acquisition system (DAQ) responsible for transporting the data from the detector to the storage. The antiProton ANihilation at DArmstadt (PANDA) at the facility for anti-proton and ion research (FAIR) is one such experiment of the future that will not use fixed hardware triggers;instead, the event selection is based on real-time feature extraction, filtering, and high-level correlations. A firmware framework for such real-time data processing has been developed and tested with hardware setup for a PANDA Forward Tracker (FT) prototype. The solution is applicable for other detector subsystems based on the so-called Trigger Readout Board (TRB) data read-out system.

关键词： Detectors Pipelines Feature extraction Hardware data acquisition Anodes real-time systems data acquisition systems (DAQs) real-time data processing tracking detectors

来源：评论

学校读者我要写书评

暂无评论

The real-time data processing framework for blockchain and edge computing

引用

ALEXANDRIA ENGINEERING JOURNAL 2025年 120卷 50-61页

作者： Gao, Zhaolong Yan, Wei Shandong Technol & Business Univ Sch Comp Sci & Technol Yantai 264005 Peoples R China Shandong Technol & Business Univ Coll Innovat & Entrepreneurship Coll Blockchain Applicat Technol Yantai 264005 Peoples R China

The rapid growth of IoT has increased the demand for large-scale data processing. However, traditional centralized methods struggle with real-time requirements and data security. This paper introduces VCDTSNet, a novel real-time IoT data processing framework that combines blockchain and edge computing. By integrating deep learning models like VGG, ConvLSTM, and DNN, VCD-TSNet effectively performs spatial feature extraction, temporal modeling, and decision-making, while using blockchain to ensure data integrity and privacy. Experimental results demonstrate that VCD-TSNet outperforms baseline models in classification accuracy, prediction precision, and real-time performance. For instance, on the BoT-IoT dataset, the classification accuracy reaches 97.5%, throughput increases to 920 TPS, and response time stays below 85 ms. This study validates the model's effectiveness and highlights its potential in large-scale IoT environments, offering efficient, secure solutions for real-time data processing. It also provides insights for future improvements in frameworks that combine edge computing with blockchain.

关键词： Blockchain real-time data processing IoT Edge computing Deep learning

来源：评论

学校读者我要写书评

暂无评论

Map Matching Algorithm for real-time data processing of Non-route GPS data in Seoul

引用

KSCE JOURNAL OF CIVIL ENGINEERING 2021年第9期25卷 3511-3522页

作者： Han, Yohee Kim, Youngchan Ku, Jisun Jung, Yeonghun Roh, Jeongrae Univ Seoul Dept Transportat Engn Seoul 02504 South Korea Seoul Metropolitan Govt Transportat Informat Div Seoul 03153 South Korea Zetalux Syst Co Ltd Solut Div Seoul 05713 South Korea

Global Positioning System (GPS) data are satellite-based longitude and latitude coordinates for which map matching with road links, which are electronic maps, may be used in traffic operation management. Various studies have been conducted to improve the accuracy of map matching;however, research on the application of map matching algorithms for real-time data processing systems has not been widely conducted. Therefore, this study investigated a map matching algorithm that can shorten the processing time required for application to systems that process large volumes of GPS data collected for a wide network. To reduce the processing time, a candidate link selection method and a grid-type geospatial indexing process were designed. The processing time was analyzed based on an emulator before applying the map matching algorithm to the Transport Operation and Information Service Center, which generates the real-time travel speed data in Seoul City. The results obtained from comparing the prior and new algorithms showed a 96% decrease in the processing time. We believe that the findings of this study will benefit the real-time processing of GPS data, which are expected to be used in various ways in traffic operations and management in the future.

关键词： Map matching real-time data processing GPS big data processing time Geospatial indexing

来源：评论

学校读者我要写书评

暂无评论

Publish/Subscribe Method for real-time data processing in Massive IoT Leveraging Blockchain for Secured Storage

引用

SENSORS 2023年第24期23卷 9692-9692页

作者： Ataei, Mohammadhossein Eghmazi, Ali Shakerian, Ali Landry Jr, Rene Chevrette, Guy Ecole Technol Super Dept Elect Engn Montreal PQ H3C 1K3 Canada Corp Off iMETRIK Global Inc Montreal PQ J4P 2K7 Canada

In the Internet of Things (IoT) era, the surge in Machine-Type Devices (MTDs) has introduced Massive IoT (MIoT), opening new horizons in the world of connected devices. However, such proliferation presents challenges, especially in storing and analyzing massive, heterogeneous data streams in real time. In order to manage Massive IoT data streams, we utilize analytical database software such as Apache Druid version 28.0.0 that excels in real-time data processing. Our approach relies on a publish/subscribe mechanism, where device-generated data are relayed to a dedicated broker, effectively functioning as a separate server. This broker enables any application to subscribe to the dataset, promoting a dynamic and responsive data ecosystem. At the core of our data transmission infrastructure lies Apache Kafka version 3.6.1, renowned for its exceptional data flow management performance. Kafka efficiently bridges the gap between MIoT sensors and brokers, enabling parallel clusters of brokers that lead to more scalability. In our pursuit of uninterrupted connectivity, we incorporate a fail-safe mechanism with two Software-Defined Radios (SDR) called Nutaq PicoLTE Release 1.5 within our model. This strategic redundancy enhances data transmission availability, safeguarding against connectivity disruptions. Furthermore, to enhance the data repository security, we utilize blockchain technology, specifically Hyperledger Fabric, known for its high-performance attributes, ensuring data integrity, immutability, and security. Our latency results demonstrate that our platform effectively reduces latency for 100,000 devices, qualifying as an MIoT, to less than 25 milliseconds. Furthermore, our findings on blockchain performance underscore our model as a secure platform, achieving over 800 Transactions Per Second in a dataset comprising 14,000 transactions, thereby demonstrating its high efficiency.

关键词： Massive Internet of Things (MIoT) publish/subscribe methodology real-time data processing Apache Kafka Apache Druid latency Hyperledger Fabric storage security

来源：评论

学校读者我要写书评

暂无评论

Forecast Model Update Based on a real-time data processing Lambda Architecture for Estimating Partial Discharges in Hydrogenerator

引用

SENSORS 2020年第24期20卷 7242页

作者： Pereira, Fabio Henrique Bezerra, Francisco Elanio Oliva, Diego Martha de Souza, Gilberto Francisco Chabu, Ivan Eduardo Santos, Josemir Coelho Nagao Junior, Shigueru Nabeta, Silvio Ikuyo Nove de Julho Univ UNINOVE Informat & Knowledge Management Grad Program BR-01525000 Sao Paulo Brazil Nove de Julho Univ UNINOVE Ind Engn Grad Program BR-01525000 Sao Paulo Brazil Univ Sao Paulo EPUSP Polytech Sch BR-05508010 Sao Paulo Brazil

The prediction of partial discharges in hydrogenerators depends on data collected by sensors and prediction models based on artificial intelligence. However, forecasting models are trained with a set of historical data that is not automatically updated due to the high cost to collect sensors' data and insufficient real-time data analysis. This article proposes a method to update the forecasting model, aiming to improve its accuracy. The method is based on a distributed data platform with the lambda architecture, which combines real-time and batch processing techniques. The results show that the proposed system enables real-time updates to be made to the forecasting model, allowing partial discharge forecasts to be improved with each update with increasing accuracy.

关键词： autoregressive forecasting model lambda architecture partial discharges power hydrogenerators real-time data processing

来源：评论

学校读者我要写书评

暂无评论

Smart Futures Based Resource Trading and Coalition Formation for real-time Mobile data processing

引用

IEEE TRANSACTIONS ON SERVICES COMPUTING 2022年第5期15卷 3047-3060页

作者： Chen, Ruitao Wang, Xianbin Liu, Xue Western Univ Dept Elect & Comp Engn London ON N6A 5B9 Canada McGill Univ Sch Comp Sci Montreal PQ H3A 0E9 Canada

Collaboration among mobile devices (MDs) is becoming more important, as it could augment computing capacity at the network edge through peer-to-peer service provisioning, and directly enhance real-time computational performance in smart Internet-of-Things applications. As an important aspect of collaboration mechanism, conventional resource trading (RT) among MDs relies on an onsite interaction process, i.e., price negotiation between service providers and requesters, which, however, inevitably incurs excessive latency and degrades RT efficiency. To overcome this challenge, this article adopts the concept of futures contract (FC) used in financial market, and proposes a smart futures for low latency RT. This new technique enables MDs to form trading coalitions and negotiate multilateral forward contracts applied to a collaboration term in the future. To maximize the benefits of self-interested MDs, the negotiation process of FC is modelled as a coalition formation game comprised of three components executed in an iterative manner, i.e., futures resource allocation, revenue sharing and payment allocation, and distributed decision-making of individual MD. Additionally, a FC enforcement scheme is implemented to efficiently manage the onsite resource sharing via recording resource balances of different task-types and MDs. Simulation results prove the superiority of smart futures in RT latency reduction and trading fairness provisioning.

关键词： Task analysis Resource management real-time systems Internet of Things Collaboration Decision making Privacy Smart futures coalition formation game resource trading task offloading real-time data processing

来源：评论

学校读者我要写书评

暂无评论

real-time data processing in supply chain management: revealing the uncertainty dilemma

引用

INTERNATIONAL JOURNAL OF PHYSICAL DISTRIBUTION & LOGISTICS MANAGEMENT 2019年第10期49卷 1003-1019页

作者： Lechler, Sabrina Canzaniello, Angelo Rossmann, Bernhard von der Gracht, Heiko A. Hartmann, Evi Friedrich Alexander Univ Erlangen Nuremberg Chair Supply Chain Management Nurnberg Germany Steinbeis Univ Berlin Sch Int Business & Entrepreneurship Futures Studi Herrenberg Germany Friedrich Alexander Univ Erlangen Nuremberg Nurnberg Germany

Purpose Particularly in volatile, uncertain, complex and ambiguous (VUCA) business conditions, staff in supply chain management (SCM) look to real-time (RT) data processing to reduce uncertainties. However, based on the premise that data processing can be perfectly mastered, such expectations do not reflect reality. The purpose of this paper is to investigate whether RT data processing reduces SCM uncertainties under real-world conditions. Design/methodology/approach Aiming to facilitate communication on the research question, a Delphi expert survey was conducted to identify challenges of RT data processing in SCM operations and to assess whether it does influence the reduction of SCM uncertainty. In total, 14 prospective statements concerning RT data processing in SCM operations were developed and evaluated by 68 SCM and data-science experts. Findings RT data processing was found to have an ambivalent influence on the reduction of SCM complexity and associated uncertainty. Analysis of the data collected from the study participants revealed a new type of uncertainty related to SCM data itself. Originality/value This paper discusses the challenges of gathering relevant, timely and accurate data sets in VUCA environments and creates awareness of the relationship between data-related uncertainty and SCM uncertainty. Thus, it provides valuable insights for practitioners and the basis for further research on this subject.

关键词： Uncertainty Supply chain management VUCA Delphi survey real-time data processing

来源：评论

学校读者我要写书评

暂无评论

real-time data processing With Lambda Architecture

Real-Time Data Processing With Lambda Architecture

引用

作者： Omkar Ashok Malusare San Jose State University

学位级别：硕士

data has evolved immensely in recent years, in type, volume and velocity. There are several frameworks to handle the big data applications. The project focuses on the Lambda Architecture proposed by Marz and its application to obtain real-time data processing. The architecture is a solution that unites the benefits of the batch and stream processing techniques. data can be historically processed with high precision and involved algorithms without loss of short-term information, alerts and insights. Lambda Architecture has an ability to serve a wide range of use cases and workloads that withstands hardware and human mistakes. The layered architecture enhances loose coupling and flexibility in the system. This a huge benefit that allows understanding the trade-offs and application of various tools and technologies across the layers. There has been an advancement in the approach of building the LA due to improvements in the underlying tools. The project demonstrates a simplified architecture for the LA that is maintainable.

关键词： Lambda Architecture (LA) Batch processing Stream processing real-time data processing Systems Architecture

来源：评论

学校读者我要写书评

暂无评论

A Survey of real-time Big data processing Algorithms 1st

A Survey of Real-Time Big Data Processing Algorithms

引用

1st International and 4th National Conference on Reliability and Safety Engineering, INCRS 2018

作者： Lal, Devesh Kumar Suman, Ugrasen School of Computer Science & IT Devi Ahilya University Indore India

ISBN: (纸本)9789811537455

data collection and processing in real time is one of the most challenging domains for big data. The sustainable proliferation of unbounded streaming data has become arduous for data collection, data pre-process, data optimization, etc. real-time streaming for data collection can effectively be performed by windowing mechanism. In this communication, we have discussed various windowing mechanisms such as sliding window, tumbling window, landmark window, index-based window, adaptive size tumbling window, and partitioned-based window. The reliability measure, which depends upon selection of appropriate windowing mechanism, has also been discussed. These window-based algorithms have been compared on the basis of CPU utilization, memory consumption, time efficiency, and operation compatibility. In this paper, we have surveyed various aggregation algorithms such as reactive aggregator, flatFAT, flatFIT, B-Int, DABA, and two stacks aggregator and compared them based on time complexity. Remarkably, a hybrid window mechanism has been introduced in this study which can handle the most recent data stream and variable rate of data stream by sliding window and tumbling window, respectively. © Springer Nature Singapore Pte Ltd. 2020.

关键词： Aggregation algorithms Big data real-time data processing Streaming data Window algorithms

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：