Many distributedsystems require coordination between the components involved. With the steady growth of such systems, the probability of failures increases, which necessitates scalable fault-tolerant agreement protoc...
详细信息
ISBN:
(纸本)9781450346993
Many distributedsystems require coordination between the components involved. With the steady growth of such systems, the probability of failures increases, which necessitates scalable fault-tolerant agreement protocols. The most common practical agreement protocol, for such scenarios, is leader-based atomic broadcast. In this work, we propose ALLCONCUR, a distributed system that provides agreement through a leaderless concurrent atomic broadcast algorithm, thus, not suffering from the bottleneck of a central coordinator. In ALLCONCUR, all components exchange messages concurrently through a logical overlay network that employs early termination to minimize the agreement latency. Our implementation of ALLCONCUR supports standard sockets-based TCP as well as high-performance InfiniBand Verbs communications. ALLCONCUR can handle up to 135 million requests per second and achieves 17x higher throughput than today's standard leader-based protocols, such as Libpaxos. Thus, ALLCONCUR is highly competitive with regard to existing solutions and, due to its decentralized approach, enables hitherto unattainable system designs in a variety of fields.
Elastic distributed storage systems have been increasingly studied in recent years because power consumption has become a major problem in data centers. Much progress has been made in improving the agility of resizing...
详细信息
Elastic distributed storage systems have been increasingly studied in recent years because power consumption has become a major problem in data centers. Much progress has been made in improving the agility of resizing small- and large-scale distributed storage systems. However, most of these studies focus on metadata based distributed storage systems. On the other hand, emerging consistent hashing based distributed storage systems are considered to allow better scalability and are highly attractive. We identify challenges in achieving elasticity in consistent hashing based distributed storage. These challenges cannot be easily solved by techniques used in current studies. In this paper, we propose an elastic consistent hashing based distributed storage to solve two problems. First, in order to allow a distributed storage to resize quickly, we modify the data placement algorithm using a primary server design and achieve an equal-work data layout. second, we propose a selective data re-integration technique to reduce the performance impact when resizing a cluster. Our experimental and trace analysis results confirm that our proposed elastic consistent hashing works effectively and allows significantly better elasticity.
Dangling pointer error is pervasive in C/C++ programs and it is very hard to detect. This paper introduces an efficient detector to detect dangling pointer error in C/C++ programs. By selectively leave some memory acc...
详细信息
The Pipe-and-Filter style represents a well-known family of component-based architectures. By executing each filter on a dedicated processing unit, it is also possible to leverage contemporary distributedsystems and ...
详细信息
ISBN:
(纸本)9781509025695
The Pipe-and-Filter style represents a well-known family of component-based architectures. By executing each filter on a dedicated processing unit, it is also possible to leverage contemporary distributedsystems and multi-core systems for a high throughput. However, this simple parallelization approach is not very effective when (1) the workload is uneven distributed over all filters and when (2) the number of available processing units exceeds the number of filters. In the first case, parallelizing all filters can lead to a waste of resources since only the slowest filter is responsible for the overall throughput. In the second case, some processing units remain unused. In this paper, we present an automatic parallelization approach providing high throughput and utilizing the available processing units. Our main idea is to provide a composite filter that is wrapped around an existing filter to increase its throughput. We call this composite filter the Task Farm Filter since it implements the Task Farm parallelization pattern. It creates and executes multiple instances of the underlying filter in parallel. Moreover, we present a modular, self-adaptive mechanism that automatically adapts the number of instances at runtime to achieve the highest possible throughput. Finally, we present an extensive experimental evaluation of our self-adaptive task farm filter by employing a CPU-intensive, an I/O-intensive, and a hybrid scenario. The evaluation shows that our task farm automatically parallelize the underlying filter and thus increases the overall throughput. Furthermore, the evaluation shows that our task farm scales well with the workload of the executed Pipe-and-Filter architecture.
Big graphs are finding increasing applications in many science and engineering domains, such as computational biology, cybermanufacturing and social media. Graphs provide a very flexible mathematical abstraction for d...
详细信息
ISBN:
(纸本)9781467388450
Big graphs are finding increasing applications in many science and engineering domains, such as computational biology, cybermanufacturing and social media. Graphs provide a very flexible mathematical abstraction for describing relationships between entities in complex systems. Real world graphs are characterized by high connectivity and high irregularity. Such non-uniform characteristics increase the mismatch between the vertex centric parallel computation model and the computer hardware resources. Another problem with the vertex-centric computation model is that it treats vertices symmetrically and this uniform assumption breaks when graphs exhibit high irregularity and graph algorithms reveal non-uniform workloads. In this keynote, I will advocate a fundamental revisit of graph computation models and promotes a methodical framework for support high performance graph parallel abstractions that are resource aware, composable and programmable. I will discuss a suite of graph optimization techniques that explore workload characteristics of graph algorithms and irregularity hidden in graph structures. I will conclude the talk by presenting some interesting research problems and unique opportunities for big graph analytics.
Data driven science is becoming increasingly more common, complex, and is placing tremendous stresses on visualization and analysis frameworks. Data sources producing 10GB per second (and more) are becoming increasing...
详细信息
ISBN:
(纸本)9781509036837
Data driven science is becoming increasingly more common, complex, and is placing tremendous stresses on visualization and analysis frameworks. Data sources producing 10GB per second (and more) are becoming increasingly commonplace in both simulation, sensor and experimental sciences. These data sources, which are often distributed around the world, must be analyzed by teams of scientists that are also distributed. Enabling scientists to view, query and interact with such large volumes of data in near-real-time requires a rich fusion of visualization and analysis techniques, middleware and workflow systems. This paper discusses initial research into visualization and analysis of distributed data workflows that enables scientists to make near-real-time decisions of large volumes of time varying data.
We report on our development of a new course: Introduction to Performance Measurement, Modeling and Analysis (IPMMA). The first offering was in Fall 2014, with a second offering in Winter 2015 in finals week at this w...
详细信息
ISBN:
(纸本)9781467376846
We report on our development of a new course: Introduction to Performance Measurement, Modeling and Analysis (IPMMA). The first offering was in Fall 2014, with a second offering in Winter 2015 in finals week at this writing. The course focuses on the fundamentals of measuring, analyzing, and modeling computer performance. As we cover the basics we will move through a set of case studies, allowing us to apply the techniques to increasingly complex problems. Case studies used in Fall 2014 include: multithreaded code;web servers;MPI code;virtualized servers;and Map/Reduce - Hadoop. In Winter 2015 we added a case study for single server queue simulation. These case studies include hands-on programming exercises both during class time and as take-home exercises. We use a variety of performance tools through the course to learn the state of the art for performance techniques and practices. We have successfully included a number of PDC topics into this course, with relatively small background needed for the students to succeed in hands-on exercises. However, we have found a total lack of textbook support for the inclusion of PDC topics into an introductory performance course.
In this paper, we present a new parallel method named SDFEM that enables frequent pattern mining (FPM) on cluster with multiple multi-core compute nodes to provide high performance. SDFEM is distinguished from previou...
详细信息
We report on our development of a new course: Introduction to Performance Measurement, Modeling and Analysis (IPMMA). The first offering was in Fall 2014, with a second offering in Winter 2015 in finals week at this w...
详细信息
ISBN:
(纸本)9781467376853
We report on our development of a new course: Introduction to Performance Measurement, Modeling and Analysis (IPMMA). The first offering was in Fall 2014, with a second offering in Winter 2015 in finals week at this writing. The course focuses on the fundamentals of measuring, analyzing, and modeling computer performance. As we cover the basics we will move through a set of case studies, allowing us to apply the techniques to increasingly complex problems. Case studies used in Fall 2014 include: multithreaded code; web servers; MPI code; virtualized servers; and Map/Reduce - Hadoop. In Winter 2015 we added a case study for single server queue simulation. These case studies include hands-on programming exercises both during class time and as take-home exercises. We use a variety of performance tools through the course to learn the state of the art for performance techniques and practices. We have successfully included a number of PDC topics into this course, with relatively small background needed for the students to succeed in hands-on exercises. However, we have found a total lack of textbook support for the inclusion of PDC topics into an introductory performance course.
暂无评论