检索结果-内蒙古大学图书馆

The Essence of online data processing

PROCEEDINGS OF THE ACM ON PROGRAMMING LANGUAGES-PACMPL 2022年第OOPSLA期6卷 899–928页

作者： Dexter, Philip Liu, Yu David Chiu, Kenneth State Univ New York SUNY 4400 Vestal Pkwy East Binghamton NY 13902 USA

data processing systems are a fundamental component of the modern computing stack. These systems are routinely deployed online: they continuously receive the requests of data processing operations, and continuously return the results to end users or client applications. online data processing systems have unique features beyond conventional data processing, and the optimizations designed for them are complex, especially when data themselves are structured and dynamic. This paper describes DON Calculus, the first rigorous foundation for online data processing. It captures the essential behavior of both the backend data processing engine and the frontend application, with the focus on two design dimensions essential yet unique to online data processing systems: incremental operation processing (IOP) and temporal locality optimization (TLO). A novel design insight is that the operations continuously applied to the data can be defined as an operation stream flowing through the data structure, and this abstraction unifies diverse designs of IOP and TLO in one calculus. DON Calculus is endowed with a mechanized metatheory centering around a key observable equivalence property: despite the significant non-deterministic executions introduced by IOP and TLO, the observable result of DON Calculus data processing is identical to that of conventional data processing without IOP and TLO. Broadly, DON Calculus is a novel instance in the active pursuit of providing rigorous guarantees to the software system stack. The specification and mechanization of DON Calculus provide a sound base for the designers of future data processing systems to build upon, helping them embrace rigorous semantic engineering without the need of developing from scratch.

关键词： Formal Reasoning online data processing Incremental Evaluation online data Optimization

来源：评论

学校读者我要写书评

暂无评论

A horizontally scalable online processing system for trigger-less data acquisition

引用

NUCLEAR INSTRUMENTS & METHODS IN PHYSICS RESEARCH SECTION A-ACCELERATORS SPECTROMETERS DETECTORS AND ASSOCIATED EQUIPMENT 2022年 1036卷

作者： Migliorini, Matteo Pazzini, Jacopo Triossi, Andrea Zanetti, Marco Zucchetta, Alberto Padova Univ Dept Phys & Astron Galileo Galilei Via Marzolo 8 I-35131 Padua Italy Natl Inst Nucl Phys Padova Div Via Marzolo 8 I-35131 Padua Italy

The vast majority of high energy physics experiments rely on data acquisition and hardware-based trigger systems performing a number of stringent selections before storing data for offline analysis. The online reconstruction and selection performed at the trigger level are bound to the synchronous nature of the data acquisition system, resulting in a trade-off between the amount of data collected and the complexity of the online reconstruction performed. Exotic physics processes, such as long-lived and slow-moving particles, are rarely targeted by online triggers as they require complex and nonstandard online reconstruction, usually incompatible with the time constraints of most data acquisition systems. The online trigger selection can thus impact as one of the main limiting factors to the experimental reach for exotic signatures. Alternative data acquisition solutions based on the continuous and asynchronous processing of the stream of data from the detectors are therefore foreseeable as a way to extend the experimental physics reach. Trigger-less data readout systems, paired with efficient streaming data processing solutions, can provide a viable alternative. In this document, an end-to-end implementation of a fully trigger-less data acquisition and online data processing system is discussed. An easily scalable and deployable implementation of such an architecture is proposed, based on open-source distributed computing frameworks capable of performing asynchronous online processing of streaming data. The proposed schema can be suitable for deployment as a fully integrated data acquisition system for small-scale experimental apparatus, or to complement the trigger-based data acquisition systems of larger experiments. A muon telescope setup consisting of a set of gaseous detectors is used as the experimental development testbed in this work, and a fully integrated online processing pipeline deployed on cloud computing resources is implemented and described.

关键词： data acquisition Trigger online data processing

来源：评论

学校读者我要写书评

暂无评论

online dynamic flat-field correction for MHz microscopy data at European XFEL

引用

JOURNAL OF SYNCHROTRON RADIATION 2023年第6期30卷 1030-1037页

作者： Birnsteinova, Sarlota de Lima, Danilo E. Ferreira Sobolev, Egor Kirkwood, Henry J. Bellucci, Valerio Bean, Richard J. Kim, Chan Koliyadu, Jayanath C. P. Sato, Tokushi Dall'Antonia, Fabio Asimakopoulou, Eleni Myrto Yao, Zisheng Buakor, Khachiwan Zhang, Yuhe Meents, Alke Chapman, Henry N. Mancuso, Adrian P. Villanueva-Perez, Pablo Vagovic, Patrik European XFEL GmbH Schenefeld Germany Lund Univ Synchrotron Radiat Res & NanoLund Lund Sweden DESY Ctr Free Electron Laser Sci CFEL Hamburg Germany Univ Hamburg Hamburg Germany Diamond Light Source Ltd Harwell Sci & Innovat Campus Harwell OX11 0DE Berks England La Trobe Univ La Trobe Inst Mol Sci Dept Chem & Phys Melbourne Vic Australia

The high pulse intensity and repetition rate of the European X-ray Free-Electron Laser (EuXFEL) provide superior temporal resolution compared with other X-ray sources. In combination with MHz X-ray microscopy techniques, it offers a unique opportunity to achieve superior contrast and spatial resolution in applications demanding high temporal resolution. In both live visualization and offline data analysis for microscopy experiments, baseline normalization is essential for further processing steps such as phase retrieval and modal decomposition. In addition, access to normalized projections during data acquisition can play an important role in decision-making and improve the quality of the data. However, the stochastic nature of X-ray free-electron laser sources hinders the use of standard flat-field normalization methods during MHz X-ray microscopy experiments. Here, an online (i.e. near real-time) dynamic flat-field correction method based on principal component analysis of dynamically evolving flat-field images is presented. The method is used for the normalization of individual X-ray projections and has been implemented as a near real-time analysis tool at the Single Particles, Clusters, and Biomolecules and Serial Femtosecond Crystallography (SPB/SFX) instrument of EuXFEL.

关键词： MHz X-ray microscopy flat-field correction online data processing X-ray free-electron laser

来源：评论

学校读者我要写书评

暂无评论

RDMA data transfer and GPU acceleration methods for high-throughput online processing of serial crystallography images

引用

JOURNAL OF SYNCHROTRON RADIATION 2020年第5期27卷 1297-1306页

作者： Ponsard, Raphael Janvier, Nicolas Kieffer, Jerome Houzet, Dominique Fristot, Vincent ESRF European Synchrotron 71 Ave Martyrs F-38000 Grenoble France Grenoble Alpes Univ GIPSALAB 11 Rue Math F-38400 St Martin Dheres France

The continual evolution of photon sources and high-performance detectors drives cutting-edge experiments that can produce very high throughput data streams and generate large data volumes that are challenging to manage and store. In these cases, efficient data transfer and processing architectures that allow online image correction, data reduction or compression become fundamental. This work investigates different technical options and methods for data placement from the detector head to the processing computing infrastructure, taking into account the particularities of modern modular high-performance detectors. In order to compare realistic figures, the future ESRF beamline dedicated to macromolecular X-ray crystallography, EBSL8, is taken as an example, which will use a PSI JUNGFRAU 4M detector generating up to 16 GB of data per second, operating continuously during several minutes. Although such an experiment seems possible at the target speed with the 100 Gb s(-1) network cards that are currently available, the simulations generated highlight some potential bottlenecks when using a traditional software stack. An evaluation of solutions is presented that implements remote direct memory access (RDMA) over converged ethernet techniques. A synchronization mechanism is proposed between a RDMA network interface card (RNIC) and a graphics processing unit (GPU) accelerator in charge of the online data processing. The placement of the detector images onto the GPU is made to overlap with the computation carried out, potentially hiding the transfer latencies. As a proof of concept, a detector simulator and a backend GPU receiver with a rejection and compression algorithm suitable for a synchrotron serial crystallography (SSX) experiment are developed. It is concluded that the available transfer throughput from the RNIC to the GPU accelerator is at present the major bottleneck in online processing for SSX experiments.

关键词： online data processing RDMA RoCEv2 GPU SSX online data analysis

来源：评论

学校读者我要写书评

暂无评论

Achieving Low-Latency, High-Throughput online Partial Particle Identification for the NA62 Experiment Using FPGAs and Machine Learning

引用

ELECTRONICS 2025年第9期14卷 1892-1892页

作者： Perticaroli, Pierpaolo Ammendola, Roberto Biagioni, Andrea Chiarini, Carlotta Ciardiello, Andrea Cretaro, Paolo Frezza, Ottorino Lo Cicero, Francesca Martinelli, Michele Piandani, Roberto Pontisso, Luca Raggi, Mauro Rossi, Cristian Simula, Francesco Turisini, Matteo Vicini, Piero Lonardo, Alessandro Ist Nazl Fis Nucl Sez Roma Ple Aldo Moro 2 I-00185 Rome Italy Ist Nazl Fis Nucl Sez Roma Tor Vergata Via Ric Sci 1 I-00133 Rome Italy Univ Sapienza Roma Dipartimento Fis Ple Aldo Moro 2 I-00185 Rome Italy Univ Autonoma San Luis Potosi Inst Fis San Luis Potosi 78000 Mexico

FPGA-RICH is an FPGA-based online partial particle identification system for the NA62 experiment employing AI techniques. Integrated between the readout of the Ring Imaging Cherenkov detector (RICH) and the low-level trigger processor (L0TP+), FPGA-RICH implements a fast pipeline to process in real-time the RICH raw hit data stream, producing trigger primitives containing elaborate physics information-e.g., the number of charged particles in a physics event-that L0TP+ can use to improve trigger decision efficiency. Deployed on a single FPGA, the system combines classical online processing with a compact Neural Network algorithm to achieve efficient event classification while managing the challenging similar to 10 MHz throughput requirement of NA62. The streaming pipeline ensures similar to 1 mu s latency, comparable to that of the NA62 detectors, allowing its seamless integration in the existing TDAQ setup as an additional detector. Development leverages High-Level Synthesis (HLS) and the open-source hls4ml package software-hardware codesign workflow, enabling fast and flexible reprogramming, debugging, and performance optimization. We describe the implementation of the full processing pipeline, the Neural Network classifier, their functional validation, performance metrics and the system's current status and outlook.

关键词： FPGA-based systems real-time particle identification neural network acceleration high-level synthesis Cherenkov detectors low-latency trigger systems high-energy physics experiments online data processing quantized neural networks high-throughput computing

来源：评论

学校读者我要写书评

暂无评论

Design and perspectives of the CMS Level-1 trigger data Scouting system

引用

NUCLEAR INSTRUMENTS & METHODS IN PHYSICS RESEARCH SECTION A-ACCELERATORS SPECTROMETERS DETECTORS AND ASSOCIATED EQUIPMENT 2024年 1067卷

作者： Ardino, Rocco Deldicque, C. Dobson, M. Gigi, D. Giorgetti, S. James, T. Miotto, G. Lazzari Meschi, E. Migliorini, M. Petrucciani, G. Rabady, D. Racz, A. Sakulin, H. Zejdl, P. Univ Padua Via VIII Febbraio 2 I-35131 Padua Italy INFN Padova Via Francesco Marzolo 8 I-35131 Padua Italy CERN Esplanade Particules 1 CH-1211 Meyrin Switzerland

The CMS detector will undergo a significant upgrade to cope with the HL-LHC instantaneous luminosity and average number of proton-proton collisions per bunch crossing (BX). The Phase-2 CMS detector will be equipped with a new Level-1 (L1) trigger system that will have access to an unprecedented level of information. Advanced reconstruction algorithms will be deployed directly on the L1 FPGA-based processors, producing reconstructed physics primitives of quasi-offline quality. The latter will be collected and processed by the Level- 1 trigger data Scouting (L1DS) system at the full bunch crossing rate. Besides providing vast amounts of data for L1 and detector monitoring, the L1DS will perform quasi-online analysis in a heterogeneous computing farm. It is expected that the study of signatures too common to fit within the L1 acceptance budget, or orthogonal to the standard physics trigger selection strategies will greatly benefit from this approach. An L1DS prototype system has been set up to operate in the current LHC Run-3, with the main goal of demonstrating the basic principle and shape the development of the Phase-2 system. The Run-3 L1DS receives trigger primitives from the Global Muon and Calorimeter Trigger, the Global Trigger decision bits and the muon segments from the Barrel Muon Track Finder. FPGA boards acquire and aggregate the synchronous trigger data streams and perform basic data reduction, before sending the trigger primitives to a set of computing nodes through 100 Gbps Ethernet connections running a simplified firmware version of the TCP/IP protocol. An Intel TBB-based DAQ software receives the TCP/IP streams and applies further processing before the ingestion of the data into a cluster of servers running the CMS reconstruction framework. The output of the computing farm are data sets in the standard CMS data analysis format. This contribution presents the Run-3 L1DS demonstrator architecture and recent physics results extracted from the collected

关键词： data acquisition Trigger online data processing

来源：评论

学校读者我要写书评

暂无评论

A 40 MHz Level-1 trigger scouting system for the CMS Phase-2 upgrade

引用

NUCLEAR INSTRUMENTS & METHODS IN PHYSICS RESEARCH SECTION A-ACCELERATORS SPECTROMETERS DETECTORS AND ASSOCIATED EQUIPMENT 2023年 1047卷

作者： Ardino, Rocco Deldicque, Christian Dobson, Marc Giorgetti, Sabrina Grosso, Gaia James, Thomas Meschi, Emilio Rabady, Dinyar Racz, Attila Sakulin, Hannes Zejdl, Petr CERN Esplanade Particules 1 CH-1211 Meyrin Switzerland Univ Padua Via VIII Febbraio 2 I-35122 Padua Italy Imperial Coll Exhibit Rd London SW7 2BX England

The CMS Phase-2 upgrade for the HL-LHC aims at preserving and expanding the current physics capability of the experiment under extreme pileup conditions. A new tracking system incorporates a track finder processor, providing tracks to the Level-1 (L1) trigger. A new high-granularity calorimeter provides fine-grained energy deposition information in the endcap region. New front-end and back-end electronics feed the L1 trigger with high-resolution information from the barrel calorimeter and the muon systems. The upgraded L1 will be based primarily on the Xilinx Ultrascale Plus series of FPGAs, capable of sophisticated feature searches with resolution often similar to the offline reconstruction. The L1 data Scouting system (L1DS) will capture L1 intermediate data produced by the trigger processors at the beam-crossing rate of 40 MHz, and carry out online analyses based on these limited-resolution data. The L1DS will provide fast and virtually unlimited statistics for detector diagnostics, alternative luminosity measurements, and, in some cases, calibrations. It also has the potential to enable the study of otherwise inaccessible signatures, either too common to fit in the L1 trigger accept budget or with requirements that are orthogonal to "mainstream"physics. The requirements and architecture of the L1DS system are presented, as well as some of the potential physics opportunities under study. The first results from the assembly and commissioning of a demonstrator currently being installed for LHC Run-3 are also presented. The demonstrator collects data from the Global Muon Trigger, the Layer-2 Calorimeter Trigger, the Barrel Muon Track Finder, and the Global Trigger systems of the current CMS L1. This demonstrator, as a data acquisition (DAQ) system operating at the LHC bunch-crossing rate, faces many of the challenges of the Phase-2 system, albeit with scaled-down connectivity, reduced data throughput and physics capabilities, providing a testing ground for new techn

关键词： data acquisition Trigger online data processing

来源：评论

学校读者我要写书评

暂无评论

Towards Real-Time Analysis of Gas-Liquid Pipe Flow: A Wire-Mesh Sensor for Industrial Applications

引用

SENSORS 2023年第8期23卷 4067页

作者： Wiedemann, Philipp Dias, Felipe de Assis Trepte, Manuel Schleicher, Eckhard Hampel, Uwe Helmholtz Zent Dresden Rossendorf Inst Fluid Dynam Bautzner Landstr 400 D-01328 Dresden Germany Teletron Rossendorf GmbH Bautzener Landstr 45 D-01454 Radeberg Germany Tech Univ Dresden Chair Imaging Tech Energy & Proc Engn D-01062 Dresden Germany

Real-time monitoring of gas-liquid pipe flow is highly demanded in industrial processes in the chemical and power engineering sectors. Therefore, the present contribution describes the novel design of a robust wire-mesh sensor with an integrated data processing unit. The developed device features a sensor body for industrial conditions of up to 400 degrees C and 135 bar as well as real-time processing of measured data, including phase fraction calculation, temperature compensation and flow pattern identification. Furthermore, user interfaces are included via a display and 4 horizontal ellipsis 20 mA connectivity for the integration into industrial process control systems. In the second part of the contribution, we describe the experimental verification of the main functionalities of the developed system. Firstly, the calculation of cross-sectionally averaged phase fractions along with temperature compensation was tested. Considering temperature drifts of up to 55 K, an average deviation of 3.9% across the full range of the phase fraction was found by comparison against image references from camera recordings. Secondly, the automatic flow pattern identification was tested in an air-water two-phase flow loop. The results reveal reasonable agreement with well-established flow pattern maps for both horizontal and vertical pipe orientations. The present results indicate that all prerequisites for an application in industrial environments in the near future are fulfilled.

关键词： wire-mesh sensor two-phase flow flow pattern identification online data processing

来源：评论

学校读者我要写书评

暂无评论

Toward Big data processing in IoT: Path Planning and Resource Management of UAV Base Stations in Mobile-Edge Computing System

引用

IEEE INTERNET OF THINGS JOURNAL 2020年第7期7卷 5995-6009页

作者： Wan, Shuo Lu, Jiaxun Fan, Pingyi Letaief, Khaled B. Tsinghua Univ Beijing Natl Res Ctr Informat Sci & Technol Dept Elect Engn Beijing 100084 Peoples R China Hong Kong Univ Sci & Technol Dept Elect Engn Hong Kong Peoples R China

Heavy data load and wide cover range have always been crucial problems for big data processing in Internet of Things (IoT). Recently, mobile-edge computing (MEC) and unmanned aerial vehicle base stations (UAV-BSs) have emerged as promising techniques in IoT. In this article, we propose a three-layer online data processing network based on the MEC technique. On the bottom layer, raw data are generated by distributed sensors with local information. Upon them, UAV-BSs are deployed as moving MEC servers, which collect data and conduct initial steps of data processing. On top of them, a center cloud receives processed results and conducts further evaluation. For online processing requirements, the edge nodes should stabilize delay to ensure data freshness. Furthermore, limited onboard energy poses constraints to edge processing capability. In this article, we propose an online edge processing scheduling algorithm based on Lyapunov optimization. In cases of low data rate, it tends to reduce edge processor frequency for saving energy. In the presence of a high data rate, it will smartly allocate bandwidth for edge data offloading. Meanwhile, hovering UAV-BSs bring a large and flexible service coverage, which results in a path planning issue. In this article, we also consider this problem and apply deep reinforcement learning to develop an online path planning algorithm. Taking observations of around environment as an input, a CNN network is trained to predict action rewards. By simulations, we validate its effectiveness in enhancing service coverage. The result will contribute to big data processing in future IoT.

关键词： Cloud computing Internet of Things Base stations Distributed databases Sensors Big data Big data deep reinforcement learning edge computing Internet of Things (IoT) online data processing

来源：评论

学校读者我要写书评

暂无评论

data fusion using Bayesian theory and reinforcement learning method

引用

Science China(Information Sciences) 2020年第7期63卷 122-124页

作者： Tongle ZHOU Mou CHEN Chenguang YANG Zhiqiang NIE College of Automation Engineering Nanjing University of Aeronautics and Astronautics Science and Technology on Electro-optic Control Laboratory School of Automation Science and Engineering South China University of Technology

Dear editor,In recent years, the theory and method of mission planning technology have been widely applied to bio-robotic systems. Increasing demands can be addressed by the application of innovative and advanced tech... 详细信息

关键词： data fusion Reinforcement learning Bayesian theory Information entropy online data processing

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：