distributedsystems have become ubiquitous in recent years, with those based on distributed ledger technology (DLT), such as blockchains, gaining more and more weight. Indeed, DLT ensures strong data integrity thanks ...
详细信息
The next generation of radio telescopes, such as the Square Kilometer Array (SKA), will need to process an incredible amount of data in real-time. In addition, the sensitivity of SKA will require a new generation of c...
详细信息
ISBN:
(数字)9781665485241
ISBN:
(纸本)9781665485241
The next generation of radio telescopes, such as the Square Kilometer Array (SKA), will need to process an incredible amount of data in real-time. In addition, the sensitivity of SKA will require a new generation of calibration and imaging software to exploit its full potential. The wide-field direction-dependent spectral deconvolution framework, called DDFacet, has been successfully used in several existing SKA pathfinders and precursors like MeerKAT and LOFAR. This imager allows a multi-core execution based on facets parallelization and a multinode execution based on observations parallelization. However, because of the amount of data to be processed, the data on a single observation will have to be distributed on several nodes. This paper proposes the first two-level parallelization of DDFacet in the case of a single observation. A multi-core parallelization based on facets and a multi-node parallelization based on frequency distribution grouped in Measurement Sets. We show that this multi-core multi-node parallelization has successfully reduced the total execution time by a factor of 5:7 on a LOFAR dataset.
Within scientific infrastructuscientists execute millions of computational jobs daily, resulting in the movement of petabytes of data over the heterogeneous infrastructure. Monitoring the computing and user activities...
详细信息
Within scientific infrastructuscientists execute millions of computational jobs daily, resulting in the movement of petabytes of data over the heterogeneous infrastructure. Monitoring the computing and user activities over such a complex infrastructure is incredibly demanding. Whereas present solutions are traditionally based on a Relational Database Management System (RDBMS) for data storage and processing, recent developments evaluate the Lambda Architecture (LA). In particular these studies have evaluated data storage and batch processing for processing large-scale monitoring datasets using Hadoop and its MapReduce framework. Although LA performed better than the RDBMS following evaluation, it was fairly complex to implement and maintain. This paper presents an Optimised Lambda Architecture (OLA) using the Apache Spark ecosystem, which involves modelling an efficient way of joining batch computation and real-time computation transparently without the need to add complexity. A few models were explored: pure streaming, pure batch computation, and the combination of both batch and streaming. An evaluation of the OLA on the CERN IT on-premises Hadoop cluster and the public Amazon cloud infrastructure for the monitoring WLCG Data acTivities (WDT) use case are both presented, demonstrating how the new architecture can offer benefits by combining both batch and real-time processing to compensate for batch-processing latency.
Due to the complexity of the resistance spot welding process, it is still a challenge to accurately know the operating status of the welding robot under the current parameter settings and to assess the welding quality...
详细信息
Due to the complexity of the resistance spot welding process, it is still a challenge to accurately know the operating status of the welding robot under the current parameter settings and to assess the welding quality of electrode caps under different types of plates in realtime with large data sizes. To solve this problem, this paper classifies the overall data set and proposes a parallel strategy method for predicting the quality of weld joints using machine learning for subsets of the data with different distribution patterns. Firstly, the PCA dimensionality reduction model was used to set the number of principal components to reduce the dimensionality of the welding process feature value dataset and reduce the difficulty of classifying the data subgroups, and the elbow method was used to set the number of clustering centers to complete the classification of the sub-datasets by applying the k-means model on the basis of the dimensionality reduction data. Finally, the feature parameters of each sub-dataset are used as input for machine learning, and a parallel prediction strategy for weld joint quality is developed based on the data distribution characteristics of each sub-dataset. The test results show that the model in this paper outperforms the static BP neural network in predicting the quality of all types of welded joints, the machine learning parallel strategy tailored to the characteristics of the data population works well with more complexly distributed welded big data. This paper provides accurate and effective estimation of body resistance welding condition, which can provide some guidance for online inspection of body resistance spot welding quality in automotive production lines.
Edge computing responds to users' requests with low latency by storing the relevant files at the network edge. Various data deduplication technologies are currently employed at edge to eliminate redundant data chu...
详细信息
Edge computing responds to users' requests with low latency by storing the relevant files at the network edge. Various data deduplication technologies are currently employed at edge to eliminate redundant data chunks for space saving. However, the lookup for the global huge-volume fingerprint indexes imposed by detecting redundancies can significantly degrade the data processing performance. Besides, we envision a novel file storage strategy that realizes the following rationales simultaneously: 1) space efficiency, 2) access efficiency, and 3) load balance, while the existing methods fail to achieve them at one shot. To this end, we report LOFS, a Lightweight Online File Storage strategy, which aims at eliminating redundancies through maximizing the probability of successful data deduplication, while realizing the three design rationales simultaneously. LOFS leverages a lightweight three-layer hash mapping scheme to solve this problem with constant-time complexity. To be specific, LOFS employs the Bloom filter to generate a sketch for each file, and thereafter feeds the sketches to the Locality Sensitivity hash (LSH) such that similar files are likely to be projected nearby in LSH tablespace. At last, LOFS assigns the files to real-world edge servers with the joint consideration of the LSH load distribution and the edge server capacity. Trace-driven experiments show that LOFS closely tracks the global deduplication ratio and generates a relatively low load std compared with the comparison methods.
The proceedings contain 6 papers. The special focus in this conference is on distributed Computing for Emerging Smart Networks. The topics include: Leveraging Machine Learning for WBANs;blockchain and Cooperative...
ISBN:
(纸本)9783030990039
The proceedings contain 6 papers. The special focus in this conference is on distributed Computing for Emerging Smart Networks. The topics include: Leveraging Machine Learning for WBANs;blockchain and Cooperative Intelligent Transport systems: Challenges and Opportunities;blockchain-Based realtime Healthcare Emergency;A Survey of Machine Learning Methods for DDoS Threats Detection Against SDN.
The management of distributed energy resources has gained special attention in the field of research with the aim of exploiting the capacity of these resources to the maximum, where many expect from the correct manage...
详细信息
Mobile Edge Computing (MEC) has been proposed as an efficient solution for Mobile crowdsensing (MCS). It allows the parallel collection and processing of data in realtime in response to a requested task. A sensing ta...
详细信息
Mobile Edge Computing (MEC) has been proposed as an efficient solution for Mobile crowdsensing (MCS). It allows the parallel collection and processing of data in realtime in response to a requested task. A sensing task can be one-time or continuous, with multiple readings collected over time. Integrating MEC and continuous sensing in MCS is challenging due to many factors, including workers' mobility, edge node placement, task location, Reputation, and data quality. In addition, guarantying cooperative communication in the presence of Anomalous data while maintaining a high quality of service (QoS) is a fundamental issue in continuous sensing. A stability-based edge node selection and anomaly detection-based decision-making framework for worker recruitment in continuous sensing is proposed to address these challenges. It can a) Select the most stable edge nodes in the area of interest (AoI), b) Dynamically cluster the workers according to their movement in the AoI, c) Locally detect and eliminate anomalies within the sensing data, and d) Adopt a feedback mechanism that ensures the cooperation between the edge nodes to eliminate untrustworthy workers in the whole sensing period and future tasks. A real-life dataset is used to evaluate the efficiency of the proposed framework. Results show that the framework outperforms the baselines by achieving higher QoS while introducing lower delay, energy consumption, and less resource consumption.
Existing software development methodologies mostly assume that an application runs on a single device without concern about the non-functional requirements of an embedded system such as latency and resource consumptio...
详细信息
Existing software development methodologies mostly assume that an application runs on a single device without concern about the non-functional requirements of an embedded system such as latency and resource consumption. Besides, embedded software is usually developed after the hardware platform is determined, since a non-negligible portion of the code depends on the hardware platform. In this article, we present a novel model-based software synthesis framework for parallel and distributed embedded systems. An application is specified as a set of tasks with the given rules for execution and communication. Having such rules enables us to perform static analysis to check some software errors at compile-time to reduce the verification difficulty. Platform-specific programs are synthesized automatically after the mapping of tasks onto processing elements is determined. The proposed framework is expandable to support new hardware platforms easily. The proposed communication code synthesis method is extensible and flexible to support various communication methods between devices. In addition, the fault-tolerant feature can be added by modifying the task graph automatically according to the selected fault-tolerance configurations by the user. The viability of the proposed software development methodology is evaluated with a real-life surveillance application that runs on six processing elements.
Blockchains are the first distributed Ledger Technologies (DLT), technologies based on decentralized networks of devices which share data on a distributed ledger. They possess qualities such as resilience, reliability...
详细信息
ISBN:
(纸本)9781665410939
Blockchains are the first distributed Ledger Technologies (DLT), technologies based on decentralized networks of devices which share data on a distributed ledger. They possess qualities such as resilience, reliability, and traceability of sensitive data. The low scalability and frequent inefficiency of Blockchain result in ineffective integration with Internet of Things (IoT) systems. Other DLTs such as DAGs (Directed Acyclic Graph DLTs) possess benefits comparable to those of Blockchains without presenting most of the limitations that prevent their application in the IoT domain. In particular, the objective of this work was to design a framework for IoT systems based on DLT with focus on security, reliability, traceability, and decentralization. The benefit obtained in terms of security, scalability, traceability, and resilience is bound to the architecture utilized. This makes our solution usable for most IoT applications that do not need to process external data in real-time. Nevertheless, a hybrid edge-DLT architecture would be sufficient to extend the system capabilities to any IoT application.
暂无评论