The growth of data has increased exponentially in recent years. In this context, data-center scale computer systems are built to meet the high storage and processing demands of these applications. Such systems are com...
详细信息
ISBN:
(纸本)9781467362405
The growth of data has increased exponentially in recent years. In this context, data-center scale computer systems are built to meet the high storage and processing demands of these applications. Such systems are composed of hundreds, thousands, or even millions of commodity computers connected through a LAN housed in a data center. It has a much larger scale than a traditional computer cluster. Hadoop enables the distributed processing of large data sets across clusters of commodity servers. It is designed to scale up from a single server to thousands of machines, with a very high degree of fault tolerance. Strength of Hadoop is in its ability to detect and handle failures. The original Hadoop native task scheduler implicitly assumes that cluster nodes are homogeneous. This assumption is used to identify a slow task and re-execute it. However, this assumption does not hold where the cluster nodes are heterogeneous, since speculatively identifying a slow task will give rise to erroneous conclusions. In MapReduce, the sub-task is transferred to a node for execution. The input to the subtask, if not present in the node, must be transferred from another node in the network. Transferring data takes time and delays execution. In this paper, we have proposed a methodology for improving Hadoop in-terms of Heterogeneity and data Locality. The performance of improved version can be measured using these metrics, i) execution time, ii) response time, iii) tasks submitted, iv) time related to jobs i. e. arrival, start and completion, v) completed task, vi) fairness, vii) locality and viii) mean completion time.
We describe Javaflow and Paraflow, the client and server parts of a digital library, providing high-performance data-retrieval and data-mining services, with emphasis on user interface as well as computing efficiency....
详细信息
We describe Javaflow and Paraflow, the client and server parts of a digital library, providing high-performance data-retrieval and data-mining services, with emphasis on user interface as well as computing efficiency. Paraflow is a component model for high-performance computing, implemented as a thin layer on Message Passing Interface (MPI);it controls a heterogeneous metacomputer, allowing groups of processes (services) to work collectively and communicate with each other by parallel messaging links (channels). Javaflow is a straightforward, intuitive, loosely coupled component model for Java applets, allowing them to be combined into a GUI that can run on a thin client from a standing start. Javaflow controls Paraflow through a well-exposed text interface which is carried by authenticated telnet, while the output of the computation can be web-pages, files, or high-speed graphics. As an example, we discuss the SARA remote-sensing library, which provides public retrieval of multichannel images of the Earth, as well as private, authenticated access to a variety of image-processing services running on parallel supercomputers. (C) 1998 Elsevier Science B.V. All rights reserved.
暂无评论