The value of graph-based big data can be unlocked by exploring the topology and metrics of the networks they represent, and the computational approaches to this exploration take on many forms. For the use-case of perf...
详细信息
The value of graph-based big data can be unlocked by exploring the topology and metrics of the networks they represent, and the computational approaches to this exploration take on many forms. For the use-case of performing global computations over a graph, it is first ingested into a graph processing system from one of many digital representations. Extracting information from graphs involves processing all their elements globally, which can be done with single-machine systems (with varying approaches to hardware usage), distributed systems (either homogeneous or heterogeneous groups of machines) and systems dedicated to high-performance computing (HPC). For these systems focused on processing the bulk of graph elements, common use-cases consist in executing for example algorithms for vertex ranking or community detection, which produce insights on graph structure and relevance of their elements. Many distributed systems (such as Flink, Spark) and libraries (e.g. Gelly, GraphX) have been built to enable these tasks and improve performance. This is achieved with techniques ranging from classic load balancing (often geared to reduce communication overhead) to exploring trade-offs between delaying computation and relaxing accuracy. In this survey we firstly familiarize the reader with common graph datasets and applications in the world of today. We provide an overview of different aspects of the graph processing landscape and describe classes of systems based on a set of dimensions we describe. The dimensions we detail encompass paradigms to express graph processing, different types of systems to use, coordination and communication models in distributed graph processing, partitioning techniques and different definitions related to the potential for a graph to be updated. This survey is aimed at both the experienced software engineer or researcher as well as the graduate student looking for an understanding of the landscape of solutions (and their limitations) for graph
The Affinity Platform is an innovative Software as a Service orchestration system designed for building Service-Oriented software solutions. It challenges the modularity of regular software frameworks and microservice...
详细信息
ISBN:
(纸本)9781450380768
The Affinity Platform is an innovative Software as a Service orchestration system designed for building Service-Oriented software solutions. It challenges the modularity of regular software frameworks and microservice architectures by abstracting the connectivity layer between two services. The main benefit of utilizing this approach is that components designed to communicate through a specific interface can be easily routed to handle this communication over a different medium. A service could extract health data from a smartwatch, use Bluetooth connectivity to send it to a smartphone for pre-processing, and the result can then be transmitted over HTTP to a server for centralization (Figure 1). All this would be possible without requiring services to agree on a specific communication protocol or data format. The capabilities of the platform were enhanced to assist the development of a multi-modal solution for monitoring patients in a constantly changing environment.
With the slowing down of Moore's law, major cloud service providers---such as Amazon Web Services, Microsoft Azure, and Alibaba Cloud---all started deploying FPGAs in their cloud platforms to improve the performan...
详细信息
ISBN:
(纸本)9781450356145
With the slowing down of Moore's law, major cloud service providers---such as Amazon Web Services, Microsoft Azure, and Alibaba Cloud---all started deploying FPGAs in their cloud platforms to improve the performance and energy-efficiency. From the perspective of performance per unit cost in the cloud, it is essential to efficiently utilize all available CPU and FPGA resources within a requested computing instance. However, most prior studies overlook the CPU-FPGA co-optimization or require a considerable amount of manual efforts to achieve it. In this poster, we present a framework called K-Flow, which enables easy FPGA accelerator integration and efficient CPU-FPGA co-scheduling for big data applications. K-Flow abstracts an application as a widely used directed acyclic graph (DAG), and dynamically schedules a number of CPU threads and/or FPGA accelerator processing elements (PEs) to execute the dataflow tasks on each DAG node. Moreover, K-Flow provides user-friendly interfaces to program each DAG node and automates the tedious process of FPGA accelerator integration and CPU-FPGA co-optimization using the genomic read alignment application BWA-MEM as a case study. Experimental results show that K-Flow achieves a throughput that is on average 94.5% of the theoretical upper bound and 1.4x better than a straightforward FPGA integration.
Interaction design research typically differentiates processes involving hardware and software tools as being led by tinkering and play, versus engineering and conceptualisation. Increasingly however, embedded maker t...
详细信息
ISBN:
(纸本)9781450372978
Interaction design research typically differentiates processes involving hardware and software tools as being led by tinkering and play, versus engineering and conceptualisation. Increasingly however, embedded maker tools and platforms require hybridisation of these processes. In the domain of digital musical instrument (DMI) design, we were motivated to explore the tensions of such a hybrid process. We designed a workshop where groups of DMI designers were given the same partly-finished instrument consisting of four microphones exciting four vibrating string models. Their task was to refine this simple instrument to their liking for one hour using Pure Data software. All groups sought to use the microphone signals to control the instrument's behaviour in rich and complex ways, but found even apparently simple mappings difficult to realise within the time constraint. We describe the difficulties they encountered and discuss emergent issues with tinkering in and with software. We conclude with further questions and suggestions for designers and technologists regarding embedded DMI design processes and tools.
Recently, efforts have been made to bring together the areas of high-performance computing (HPC) and massive data processing (Big Data). Traditional HPC frameworks, like COMPSs, are mostly task-based, while popular bi...
详细信息
ISBN:
(纸本)9781728141947
Recently, efforts have been made to bring together the areas of high-performance computing (HPC) and massive data processing (Big Data). Traditional HPC frameworks, like COMPSs, are mostly task-based, while popular big-data environments, like Spark, are based on functional programming principles. The earlier are know for their good performance for regular, matrix-based computations;on the other hand, for fine-grained, data-parallel workloads, the later has often been considered more successful. In this paper we present our experience with the integration of some dataflow techniques into COMPSs, a task-based framework, in an effort to bring together the best aspects of both worlds. We present our API, called DDF, which provides a new data abstraction that addresses the challenges of integrating Big Data application scenarios into COMPSs. DDF has a functional-based interface, similar to many Data Science tools, that allows us to use dynamic evaluation to adapt the task execution in runtime. Besides the performance optimization it provides, the API facilitates the development of applications by experts in the application domain. In this paper we evaluate DDF's effectiveness by comparing the resulting programs to their original versions in COMPSs and Spark. The results show that DDF can improve COMPSs execution time and even outperform Spark in many use cases.
Compute-intensive applications have gradually changed focus from massively parallel supercomputers to capacity as a resource obtained on-demand. This is particularly true for the large-scale adoption of cloud computin...
详细信息
Compute-intensive applications have gradually changed focus from massively parallel supercomputers to capacity as a resource obtained on-demand. This is particularly true for the large-scale adoption of cloud computing and MapReduce in industry, while it has been difficult for traditional high-performance computing (HPC) usage in scientific and engineering computing to exploit this type of resources. However, with the strong trend of increasing parallelism rather than faster processors, a growing number of applications target parallelism already on the algorithm level with loosely coupled approaches based on sampling and ensembles. While these cannot trivially be formulated as MapReduce, they are highly amenable to throughput computing. There are many general and powerful frameworks, but in particular for sampling-based algorithms in scientific computing there are some clear advantages from having a platform and scheduler that are highly aware of the underlying physical problem. Here, we present how these challenges are addressed with combinations of dataflow programming, peer-to-peer techniques and peer-to-peer networks in the Copernicus platform. This allows automation of sampling-focused workflows, task generation, dependency tracking, and not least distributing these to a diverse set of compute resources ranging from supercomputers to clouds and distributed computing (across firewalls and fragile networks). Workflows are defined from modules using existing programs, which makes them reusable without programming requirements. The system achieves resiliency by handling node failures transparently with minimal loss of computing time due to checkpointing, and a single server can manage hundreds of thousands of cores e.g. for computational chemistry applications. (C) 2016 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY license
In this paper, we propose a new dataflow runtime that dynamically manages several dataflow applications. We propose an implementation of such a runtime called CalMAR built on top of the RVC-Cal environment, and valida...
详细信息
ISBN:
(纸本)9781450351867
In this paper, we propose a new dataflow runtime that dynamically manages several dataflow applications. We propose an implementation of such a runtime called CalMAR built on top of the RVC-Cal environment, and validate its efficiency compared to the RVC-CAL traditional approach.
In this paper, we present a compiler extension for applications targeting high performance embedded systems. It analyzes the graph of a dataflow application in order to adapt its parallelism degree. Our approach consi...
详细信息
In this paper, we present a compiler extension for applications targeting high performance embedded systems. It analyzes the graph of a dataflow application in order to adapt its parallelism degree. Our approach consists in the detection and the substitution of built-in patterns in the dataflow. Modifications applied on the graph do not alter the semantic of the application. A parallelism reduction engine is also described to perform an exhaustive search of the best reduction. Our proposition has been implemented within an industry-grade compiler for the Sigma-C dataflow language. It shows that for dataflow applications, the parallelism reduction extension helps the user focus on the algorithm by hiding all parallelism tuning considerations. Experimentations demonstrate the accuracy and the performance of the reduction engine for both synthetic and real applications.
dataflow languages provide natural support for specifying constraints between objects in dynamic applications, where programs need to react efficiently to changes in their environment. In this article, we show that on...
详细信息
dataflow languages provide natural support for specifying constraints between objects in dynamic applications, where programs need to react efficiently to changes in their environment. In this article, we show that one-way dataflow constraints, largely explored in the context of interactive applications, can be seamlessly integrated in any imperative language and can be used as a general paradigm for writing performance-critical reactive applications that require efficient incremental computations. In our framework, programmers can define ordinary statements of the imperative host language that enforce constraints between objects stored in special memory locations designated as "reactive." Reactive objects can be of any legal type in the host language, including primitive data types, pointers, arrays, and structures. Statements defining constraints are automatically re-executed every time their input memory locations change, letting a program behave like a spreadsheet where the values of some variables depend on the values of other variables. The constraintsolving mechanism is handled transparently by altering the semantics of elementary operations of the host language for reading and modifying objects. We provide a formal semantics and describe a concrete embodiment of our technique into C/C++, showing how to implement it efficiently in conventional platforms using off-the-shelf compilers. We discuss common coding idioms and relevant applications to reactive scenarios, including incremental computation, observer design pattern, data structure repair, and software visualization. The performance of our implementation is compared to problem-specific change propagation algorithms, as well as to language-centric approaches such as self-adjusting computation and subject/observer communication mechanisms, showing that the proposed approach is efficient in practice.
暂无评论