The high energy physics (HEP) community relies upon a global network of computing and data centers to analyze data produced by multiple experiments at the Large Hadron Collider (LHC). However, this global network does...
详细信息
ISBN:
(纸本)9781467365987
The high energy physics (HEP) community relies upon a global network of computing and data centers to analyze data produced by multiple experiments at the Large Hadron Collider (LHC). However, this global network does not satisfy all research needs. Ambitious researchers often wish to harness computing resources that are not integrated into the global network, including private clusters, commercial clouds, and other production grids. To enable these use cases, we have constructed Lobster, a system for deploying data intensive high throughput applications on non-dedicated clusters. This requires solving multiple problems related to non-dedicated resources, including work decomposition, software delivery, concurrency management, data access, data merging, and performance troubleshooting. With these techniques, we demonstrate Lobster running effectively on 10k cores, producing throughput at a level comparable with some of the largest dedicated clusters in the LHC infrastructure.
This paper presents SCALANYTICS, a declarative platform that supports high-performance application layer analysis of network traffic. SCALANYTICS uses (1) stateful network packet processing techniques for extracting a...
详细信息
ISBN:
(纸本)9781450319102
This paper presents SCALANYTICS, a declarative platform that supports high-performance application layer analysis of network traffic. SCALANYTICS uses (1) stateful network packet processing techniques for extracting application-layer data from network packets, (2) a declarative rule-based language called ANALOG for compactly specifying analysis pipelines from reusable modules, and (3) a task-stealing architecture for processing network packets at high throughput within these pipelines, by leveraging multi-core processing capabilities in a load-balanced manner without the need for explicit performance profiling. We have developed a prototype of SCALANYTICS that enhances a declarative networking engine with support for ANALOG and various stateful components, integrated with a parallel task-stealing execution model. We evaluate our SCALANYTICS prototype on a wide range of pipelines for analyzing SMTP and SIP traffic, and for detecting malicious traffic flows. Our evaluation on a 16-core machine demonstrate that SCALANYTICS achieves up to 11.4× improvement in throughput compared with the best uniprocessor implementation. Moreover, SCALANYTICS outperforms the Bro intrusion detection system by an order of magnitude when used for analyzing SMTP traffic.
暂无评论