检索结果-内蒙古大学图书馆

35th IEEE International Conference on Application-Specific Systems, Architectures and Processors (ASAP)

作者： Miura, Shinya Chang, Qiong Miyazaki, Jun Tokyo Inst Technol Sch Comp Tokyo Japan

ISBN: (纸本)9798350349641;9798350349634

We propose a high-performance k-way in-place merging algorithm by CPU-GPU cooperative processing. Current merging algorithms are either k-way in-place with CPUs only or in-place for data smaller than the GPU memory size. To solve this problem, we extend these merging algorithms to be more adaptable to a wider range of applications by improving the segmentation, reordering, and inter-segment merging methods used in the current merging algorithms by using both CPUs and GPUs cooperatively. We conducted experiments to evaluate the performance of our proposed algorithm and analyzed its detailed behavior by changing parameters and data distributions. We also applied the proposed merging algorithm to the merge sort algorithm to verify its performance.

关键词： GPU CPU-GPU cooperative processing large-scale data processing Merging Sorting

来源：评论

学校读者我要写书评

暂无评论

The Family of MapReduce and large-scale data processing Systems

引用

ACM COMPUTING SURVEYS 2013年第1期46卷 11-11页

作者： Sakr, Sherif Liu, Anna Fayoumi, Ayman G. NICTA Dept Comp Sci & Engn Sydney NSW Australia Univ New S Wales Sydney NSW Australia King Abdulaziz Univ Jeddah 21413 Saudi Arabia

In the last two decades, the continuous increase of computational power has produced an overwhelming flow of data which has called for a paradigm shift in the computing architecture and large-scale data processing mechanisms. MapReduce is a simple and powerful programming model that enables easy development of scalable parallel applications to process vast amounts of data on large clusters of commodity machines. It isolates the application from the details of running a distributed program such as issues on data distribution, scheduling, and fault tolerance. However, the original implementation of the MapReduce framework had some limitations that have been tackled by many research efforts in several followup works after its introduction. This article provides a comprehensive survey for a family of approaches and mechanisms of large-scale data processing mechanisms that have been implemented based on the original idea of the MapReduce framework and are currently gaining a lot of momentum in both research and industrial communities. We also cover a set of introduced systems that have been implemented to provide declarative programming interfaces on top of the MapReduce framework. In addition, we review several large-scale data processing systems that resemble some of the ideas of the MapReduce framework for different purposes and application scenarios. Finally, we discuss some of the future research directions for implementing the next generation of MapReduce-like solutions.

关键词： Design Algorithms Performance MapReduce big data large-scale data processing

来源：评论

学校读者我要写书评

暂无评论

Autonomic Resource Management for Program Orchestration in large-scale data Analysis 31

Autonomic Resource Management for Program Orchestration in L...

引用

31st IEEE International Parallel and Distributed processing Symposium (IPDPS)

作者： Tanaka, Masahiro Taurat, Kenjiro Torisawa, Kentaro Natl Inst Informat & Commun Technol NICT Data Driven Intelligent Syst Res Ctr DIRECT Universal Commun Res Inst 3-5 HikaridaiSeika Cho Kyoto 6190289 Japan Univ Tokyo Grad Sch Informat Sci & Technol Dept Informat & Commun Engn Bunkyo Ku 7-3-1 Hongo Tokyo 1130033 Japan

ISBN: (纸本)9781538639146

large-scale data analysis applications are becoming more and more prevalent in a wide variety of areas. These applications are composed of many currently available programs called analysis components. Thousands of analysis component processes are orchestrated on many compute nodes. This paper proposes a novel self-tuning framework for optimizing an application's throughput in large-scale data analysis. One challenge is developing efficient orchestration that takes into account the diversity of analysis components and the varying performances of compute nodes. In our previous work, we achieved such an orchestration to a certain degree by introducing our own middleware, which wraps each analysis component as a remote procedure call (RPC) service. The middleware also pools the processes to reduce startup overhead, which is a serious obstacle to achieving high throughput. This work tackles the remaining task of tuning the size of the analysis components' process pools to maximize the application's throughput. This is challenging because analysis components differ drastically in turnaround times and memory footprints. The size of the process pool for each type of analysis component should be set by giving consideration to these properties as well as the constraints on both the memory capacity and the processor core counts. In this work, we formulate this task as a linear programming problem and obtain the optimal pool sizes by solving it. Compared to our previous work, we significantly improved the scalability of our framework by reformulating the performance model to work on hundreds of heterogeneous nodes. We also extended the service allocation mechanism to manage the computational load on each compute node and reduce communication overhead. The experimental results show that our approach is scalable to thousands of analysis component processes running on 200 compute nodes across three clusters. Moreover, our approach significantly reduces memory footprint.

关键词： large-scale data processing program composition service composition self-tuning resource management

来源：评论

学校读者我要写书评

暂无评论

Low Latency and Resource-aware Program Composition for large-scale data Analysis 16

Low Latency and Resource-aware Program Composition for Large...

引用

16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid)

作者： Tanaka, Masahiro Taura, Kenjiro Torisawa, Kentaro Natl Inst Informat & Commun Technol NICT Universal Commun Res Inst 3-5 Hikaridai Seika Kyoto 6190289 Japan Univ Tokyo Grad Sch Informat Sci & Technol Dept Informat & Commun Engn 7-3-1 Hongo Bunkyo Ku Tokyo 1130033 Japan

ISBN: (纸本)9781509024537

The importance of large-scale data analysis has shown a recent increase in a wide variety of areas, such as natural language processing, sensor data analysis, and scientific computing. Such an analysis application typically reuses existing programs as components and is often required to continuously process new data with low latency while processing large-scale data on distributed computation nodes. However, existing frameworks for combining programs into a parallel data analysis pipeline (e.g., workflow) are plagued by the following issues: (1) Most frameworks are oriented toward high-throughput batch processing, which leads to high latency. (2) A specific language is often imposed for the composition and/or such a specific structure as a simple unidirectional dataflow among constituting tasks. (3) A program used as a component often takes a long time to start up due to the heavy load at initialization, which is referred to as the startup overhead. Our solution to these problems is a remote procedure call (RPC)-based composition, which is achieved by our middleware Rapid Service Connector (RaSC). RaSC can easily wrap an ordinary program and make it accessible as an RPC service, called a RaSC service. Using such component programs as RaSC services enables us to integrate them into one program with low latency without being restricted to a specific workflow language or dataflow structure. In addition, a RaSC service masks the startup overhead of a component program by keeping the processes of the component program alive across RPC requests. We also proposed architecture that automatically manages the number of processes to maximize the throughput. The experimental results showed that our approach excels in overall throughput as well as latency, despite its RPC overhead. We also showed that our approach can adapt to runtime changes in the throughput requirements.

关键词： large-scale data processing program composition service composition

来源：评论

学校读者我要写书评

暂无评论

RANDOM PROJECTIONS THROUGH MULTIPLE OPTICAL SCATTERING: APPROXIMATING KERNELS AT THE SPEED OF LIGHT 41

RANDOM PROJECTIONS THROUGH MULTIPLE OPTICAL SCATTERING: APPR...

引用

41st IEEE International Conference on Acoustics, Speech and Signal processing (ICASSP)

作者： Saade, A. Caltagirone, F. Carron, I. Daudet, L. Dremeau, A. Gigan, S. Krzakala, F. CNRS UMR 8550 Lab Phys Stat Paris France Ecole Normale Super F-75005 Paris France ESPCI Inst Langevin F-75005 Paris France CNRS UMR 7587 F-75005 Paris France Paris Diderot Univ Sorbonne Paris Cite F-75013 Paris France ENSTA Bretagne F-29806 Brest France Lab STICC UMR 6285 F-29806 Brest France CNRS UMR 8552 Lab Kastler Brossel F-75005 Paris France Univ Pierre & Marie Curie Paris 06 Sorbonne Univ F-75005 Paris France PSL Res Univ F-75005 Paris France

ISBN: (纸本)9781479999880

Random projections have proven extremely useful in many signal processing and machine learning applications. However, they often require either to store a very large random matrix, or to use a different, structured matrix to reduce the computational and memory costs. Here, we overcome this difficulty by proposing an analog, optical device, that performs the random projections literally at the speed of light without having to store any matrix in memory. This is achieved using the physical properties of multiple coherent scattering of coherent light in random media. We use this device on a simple task of classification with a kernel machine, and we show that, on the MNIST database, the experimental results closely match the theoretical performance of the corresponding kernel. This framework can help make kernel methods practical for applications that have large training sets and/or require real-time prediction. We discuss possible extensions of the method in terms of a class of kernels, speed, memory consumption and different problems.

关键词： Kernel methods random projections machine learning large-scale data processing optical computing

来源：评论

学校读者我要写书评

暂无评论

Performance impact of JobTracker failure in Hadoop

引用

INTERNATIONAL JOURNAL OF COMMUNICATION SYSTEMS 2015年第7期28卷 1265-1281页

作者： Kim, Young-Pil Hong, Cheol-Ho Yoo, Chuck Korea Univ Comp Sci & Engn Seoul 136713 South Korea

In this paper, we analyze the performance impact of JobTracker failure in Hadoop. A JobTracker failure is a serious problem that affects the overall job processing performance. We describe the cause of failure and the system behaviors because of failed job processing in the Hadoop. On the basis of the analysis, we build a job completion time model that reflects failure effects. Our model is based on a stochastic process with a node crash probability. With our model, we run simulation of performance impact with very credible failure data available from USENIX called computer failure data repository that have been collected for past 9years. The results show that the performance impact is very severe in that the job completion time increases about four times typically, and in a worst case, it increases up to 68 times. Copyright (c) 2014 John Wiley & Sons, Ltd.

关键词： Hadoop large-scale data processing failure analysis JobTracker

来源：评论

学校读者我要写书评

暂无评论

RANDOM PROJECTIONS THROUGH MULTIPLE OPTICAL SCATTERING: APPROXIMATING KERNELS AT THE SPEED OF LIGHT

RANDOM PROJECTIONS THROUGH MULTIPLE OPTICAL SCATTERING: APPR...

引用

IEEE International Conference on Acoustics, Speech and Signal processing

作者： A. Saade F. Caltagirone I. Carron L. Daudet A. Drémeau S. Gigan F. Krzakala Laboratoire de Physique Statistique CNRS UMR 8550 & école Normale Supérieure Paris France. Institut Langevin ESPCI and CNRS UMR 7587 Paris F-75005 France ENSTA Bretagne and Lab-STICC UMR 6285 F-29806 Brest France Laboratoire Kastler Brossel CNRS UMR 8552 & Ecole Normale Supérieure75005 Paris France.

ISBN: (纸本)9781479999897

关键词： Kernel methods random projections machine learning large-scale data processing optical computing Optical computing Kernel kernel method Light Seeds Artificial Intelligence Machine Learning Memory Memory PROJECTION coherent light Light velocity Random media THROUGH

来源：评论

学校读者我要写书评

暂无评论

The role of text pre-processing in opinion mining on a social media language dataset 3

The role of text pre-processing in opinion mining on a socia...

引用

3rd Brazilian Conference on Intelligent Systems (BRACIS)

作者： dos Santos, Fernando Leandro Ladeira, Marcelo Univ Brasilia CIC UnB Brasilia DF Brazil

ISBN: (纸本)9781479956180

This work describes an opinion mining application over a dataset extracted from the web and composed of reviews with several Internet slangs, abbreviations and typo errors. Opinion mining is a study field that tries to identify and classify subjectivity, such as opinions, emotions or sentiments in natural language. In this research, 759.176 Portuguese reviews were extracted from the app store Google Play. Due to the large amount of reviews, large-scale processing techniques were needed, involving powerful frameworks such as Hadoop and Mahout. Based on tests conducted it was concluded that pre-processing has an insignificant role in opinion mining task for the specific domain of reviews of mobile apps. The work also contributed to the creation of a corpus consisting of 759 thousand reviews and a dictionary of slangs and abbreviations commonly used in the Internet.

关键词： opinion mining text mininig sentiment analysis text pre-processing large-scale data processing

来源：评论

学校读者我要写书评

暂无评论

MapReduce with communication overlap (MaRCO)

引用

JOURNAL OF PARALLEL AND DISTRIBUTED COMPUTING 2013年第5期73卷 608-620页

作者： Ahmad, Faraz Lee, Seyong Thottethodi, Mithuna Vijaykumar, T. N. Purdue Univ W Lafayette IN 47906 USA Oak Ridge Natl Lab Oak Ridge TN 37831 USA

MapReduce is a programming model from Google for cluster-based computing in domains such as search engines, machine learning, and data mining. MapReduce provides automatic data management and fault tolerance to improve programmability of clusters. MapReduce's execution model includes an all-map-to-all-reduce communication, called the shuffle, across the network bisection. Some MapReductions move large amounts of data (e.g., as much as the input data), stressing the bisection bandwidth and introducing significant runtime overhead. Optimizing such shuffle-heavy MapReduction's is important because (1) they include key applications (e.g., inverted indexing for search engines and data clustering for machine learning) and (2) they run longer than shuffle-light MapReductions (e.g., 5x longer). In MapReduce, the asynchronous nature of the shuffle results in some overlap between the shuffle and map. Unfortunately, this overlap is insufficient in shuffle-heavy MapReductions. We propose MapReduce with communication overlap (MaRCO) to achieve nearly full overlap via the novel idea of including reduce in the overlap. While MapReduce lazily performs reduce computation only after receiving all the map data, MaRCO employs eager reduce to process partial data from some map tasks while overlapping with other map tasks' communication. MaRCO's approach of hiding the latency of the inevitably high shuffle volume of shuffle-heavy MapReductions is fundamental for achieving performance. We implement MaRCO in Hadoop's MapReduce and show that on a 128-node Amazon EC2 cluster, MaRCO achieves 23% average speed-up over Hadoop for shuffle-heavy MapReductions. (C) 2012 Elsevier Inc. All rights reserved.

关键词： Cloud computing Parallel computing MapReduce Performance optimization Distributed processing large-scale data processing

来源：评论

学校读者我要写书评

暂无评论

Distributed Kernel Matrix Approximation and Implementation Using Message Passing Interface

Distributed Kernel Matrix Approximation and Implementation U...

引用

12th International Conference on Machine Learning and Applications (ICMLA)

作者： Dameh, Taher A. Abd-Almageed, Wael Hefeeda, Mohamed Simon Fraser Univ Sch Comp Sci Burnaby BC V5A 1S6 Canada Univ Maryland Inst Adv Comp Studies College Pk MD 20742 USA

ISBN: (纸本)9780769551449

We propose a distributed method to compute similarity (also known as kernel and Gram) matrices used in various kernel-based machine learning algorithms. Current methods for computing similarity matrices have quadratic time and space complexities, which make them not scalable to large-scale data sets. To reduce these quadratic complexities, the proposed method first partitions the data into smaller subsets using various families of locality sensitive hashing, including random project and spectral hashing. Then, the method computes the similarity values among points in the smaller subsets to result in approximated similarity matrices. We analytically show that the time and space complexities of the proposed method are subquadratic. We implemented the proposed method using the Message Passing Interface (MPI) framework and ran it on a cluster. Our results with real large-scale data sets show that the proposed method does not significantly impact the accuracy of the computed similarity matrices and it achieves substantial savings in running time and memory requirements.

关键词： large-scale data processing big data distributed clustering kernel matrix approximation kernel-based algorithms big data message passing space complexities matrices similarity matrix Seeds load capacity based algorithm evolutionary algorithm Hash based algorithms dataset hashing Quadratic memory requirements Runtime Kernel

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：