This paper considers a distributed computing framework, where the task of T matrix-vector products is distributed among n worker machines. External adversaries have access to a subset L (the cardinality of which is |L...
详细信息
ISBN:
(纸本)9781665421607;9781665421591
This paper considers a distributed computing framework, where the task of T matrix-vector products is distributed among n worker machines. External adversaries have access to a subset L (the cardinality of which is |L|) of these machines, and can maliciously perturb the result of each of their computations with probability alpha. To correctly recover each matrix-vector product, the master has to identify a set (of a fixed cardinality) of 'unattacked' worker machines. Towards this end, this work proposes four schemes that aim at performing such an identification. These schemes are analyzed and compared under different regimes of (|L|;alpha) for the two cases when |L| is (1) known or (2) unknown at the master.
Hydrological models (HMs) are essential for understanding the complexities of the water cycle and runoff dynamics. Sensitivity analysis (SA), an essential component of HMs, plays a key role in identifying the paramete...
详细信息
Hydrological models (HMs) are essential for understanding the complexities of the water cycle and runoff dynamics. Sensitivity analysis (SA), an essential component of HMs, plays a key role in identifying the parameters that have the greatest impact on model outcomes. It helps to simplify the complexity of hydrological systems and provides a comprehensive understanding of the underlying physical processes. However, the complexity of HMs and the diversity of SA methods pose significant challenges for researchers, making the SA configuration process intricate and requiring substantial computational resources. To address these challenges, we propose a comprehensive strategy that integrates knowledge-driven configuration services with distributed online model services. First, we establish a rule-based knowledge repository and a case-based knowledge repository. These repositories provide general configuration guidance and similar SA case recommendations, respectively, to support decision-making in critical SA steps. This ensures that the configuration of SA is accurate and reliable. Secondly, we encapsulate HMs as web services and leverage distributed computing resources to optimize execution efficiency. Then, we integrate the HM services with the SA modules to achieve a complete SA experiment. Based on this strategy, we finally developed a prototype system that offers a userfriendly tool for conducting SA with enhanced computational performance and streamlined workflow. The watershed-scale HM, SWAT, was used to test the effectiveness of the prototype system. The results demonstrate that this strategy enables more comprehensive analysis and improves decision-making through configuration guidance, and holds promise for enhancing the reliability and efficiency of SA in hydrological modeling.
Coded computing has proven its efficiency in tolerating stragglers in distributed computing. Workers return the sub-computation results to the master after computing, and the master recovers the final computation resu...
详细信息
ISBN:
(纸本)9789819708581;9789819708598
Coded computing has proven its efficiency in tolerating stragglers in distributed computing. Workers return the sub-computation results to the master after computing, and the master recovers the final computation result by decoding. However, the workers may provide incorrect results, which leads to wrong final result. Therefore, it is meaningful to improve the resilience of coded computing against errors. Most existing verification schemes only use the workers' fully correct computations to recover the final result, and the defective computations are not considered for decoding. In this paper, we focus on matrix multiplication and design a general Test-and-Decode (TD) scheme to recover the final result efficiently. Furthermore, we divide each sub-computation result into multiple parts and fully use the correct parts for partial recovery, which can improve the tolerance for errors in computations. Decoding is performed only when the verification result satisfies the permission, which avoids repetitive decoding. We conduct extensive simulation experiments to evaluate the probability of successful recovery of the results and the computation time of the TD scheme. We also compare the TD scheme with other verification schemes and the results show that it outperforms the current schemes in terms of efficiency in verifying and recovering computational results.
For more than three decades, distributed systems have been described and analyzed using topological tools, primarily using two techniques: protocol complexes and directed algebraic topology. In both cases, the conside...
详细信息
In this paper, we first show the condition under which the transient faults are observable and the safety requirements are satisfied for a broadcast algorithm. Then we propose the first safe fault-containing and self-...
详细信息
ISBN:
(纸本)9781665497794
In this paper, we first show the condition under which the transient faults are observable and the safety requirements are satisfied for a broadcast algorithm. Then we propose the first safe fault-containing and self-healing broadcast algorithm for locally observable faults in tree networks. A locally observable transient fault refers to a transient fault that perturbs the state of a process such that a faulty state of a process could be distinguished from a non-faulty state by all neighbours of the faulty process, and locally non-observable transient faults, otherwise. Our proposed algorithm contains and self-heals an unlimited number of transient faults in at most O(3) rounds provided that any two faulty processes are separated by two non-faulty processes.
Data exchange has become increasingly important in modern business and research. Consequently, many initiatives are being developed worldwide to facilitate open data exchange in secure distributed marketplaces. Ideall...
详细信息
ISBN:
(纸本)9798350366266;9798350366259
Data exchange has become increasingly important in modern business and research. Consequently, many initiatives are being developed worldwide to facilitate open data exchange in secure distributed marketplaces. Ideally, each party maintains control over their data and implements access through legal contracts, in the form of programmable policy. Such policy would express where the data exchange takes place, and who has access to the data. Inspired by how traditional Operating Systems abstract the complexities of computer architectures into standardized core functions, this research focuses on abstracting different data exchange patterns into a unified set of core data exchange microservices that adhere to agreed-upon data exchange policies. DYNAMOS implements a distributed data exchange platform and recreates real-life data exchange use cases. It is designed to be self-adaptive, utilizing extendable algorithms to generate dynamic microservice compositions and dynamically choose archetype patterns, influenced by policy, user input, or system events. In our study, we highlight key insights from our experience with a dynamic microservice platform. Employing sidecars for communication abstraction, protocol buffers for strict interface definition, and ephemeral single-use jobs for improved security emerged as pivotal strategies. However, these approaches do introduce a trade-off between operational speed and especially system complexity.
This study introduces a method to forecast data popularity in high energy physics (HEP) experiments, focusing on unbalanced and irregular time-series data. The goal is to predict the popularity of specific datasets ac...
详细信息
This study introduces a method to forecast data popularity in high energy physics (HEP) experiments, focusing on unbalanced and irregular time-series data. The goal is to predict the popularity of specific datasets accurately over time, which is crucial for optimizing data replication and placement strategies and enhancing distributed computing efficiency in HEP experiments. The methodology utilizes advanced machine learning techniques and time-series analysis to tackle the challenges posed by the unbalanced nature of the data. The paper outlines the key components of the methodology, including data preprocessing and balancing techniques, filtration, and model selection. To evaluate the effectiveness of the presented approach, the authors conduct experiments on real-world HEP datasets, comparing their predictions against actual data. The findings of this study have important implications for resource management and decision-making in distributed computing of various large-scale scientific projects. By providing forecasts of data popularity, researchers and administrators can efficiently allocate resources, optimize data storage and retrieval mechanisms, and improve overall data processing efficiency.
The rapid emergence of massive datasets in various fields poses a serious challenge to tra-ditional statistical ***,it provides opportunities for researchers to develop novel *** by the idea of divide-and-conquer,vari...
详细信息
The rapid emergence of massive datasets in various fields poses a serious challenge to tra-ditional statistical ***,it provides opportunities for researchers to develop novel *** by the idea of divide-and-conquer,various distributed frameworks for statistical estimation and inference have been *** were developed to deal with large-scale statistical optimization *** paper aims to provide a comprehensive review for related *** includes parametric models,nonparametric models,and other frequently used *** key ideas and theoretical properties are *** trade-off between communication cost and estimate precision together with other concerns is discussed.
Message oriented and robotics middleware play an important role in facilitating robot control, abstracting complex functionality, and unifying communication patterns between sensors and devices. However, using multipl...
详细信息
ISBN:
(数字)9798400703225
ISBN:
(纸本)9798400703225
Message oriented and robotics middleware play an important role in facilitating robot control, abstracting complex functionality, and unifying communication patterns between sensors and devices. However, using multiple middleware frameworks presents a challenge in integrating diferent robots within a single system. To address this challenge, we present Wrapyf, a Python wrapper supporting multiple message oriented and robotics middleware, including ZeroMQ, YARP, ROS, and ROS 2. Wrapyf also provides plugins for exchanging deep learning framework data, without additional encoding or preprocessing steps. Using Wrapyf eases the development of scripts that run on multiple machines, thereby enabling cross-platform communication and workload distribution. We fnally present the three communication schemes that form the cornerstone of Wrapyf's communication model, along with examples that demonstrate their applicability.
Management and information systems are considered two major disciplines that have to be combined to benefit from knowledge for rational decision-making which means strict procedures utilizing objective knowledge and l...
详细信息
暂无评论