检索结果-内蒙古大学图书馆

22nd Euromicro International Conference on Parallel, Distributed, and Network-Based processing (PDP)

作者： Palazzo, Simone Spampinato, Concetto Giordano, Daniela Univ Catania Dept Elect Elect & Comp Engn I-95125 Catania Italy

ISBN: (纸本)9781479927289

Ecology is, nowadays, an interdisciplinary, collaborative and data-intensive science;therefore, discovering, integrating and analysing daily-produced data is necessary to support researchers to investigate complex questions, ranging from single particles to animals to the biosphere [1]. As a consequence, ecology-related multimedia content has been produced massively in recent years: for example, the Xeno-canto project(1) and the Pl@ntNet project(2) respectively collected 140,000 audio records of 8,700 bird species and about 60,000 thousand images covering thousand of plant species, to be used by scientists or professionals. Unfortunately, a manual analysis of such amount of generated data is impossible: automatic analysis tools combined with highperformance computing (HPC) solutions are therefore heavily demanded for making sense of such big ecological data. In this paper we present a case study of large-scale video processing on HPC facilities for underwater fish monitoring in the context of the Fish4Knowledge project(3), where a system to analyse long-term underwater camera footage has been developed. The paper is meant to report on the employed hardware/software architecture, the design and deployment of the parallel job manager, and the problems encountered during the whole process, from load balancing to job submission policies to bottlenecks.

关键词： computerised monitoring ecology geophysical image processing video signal processing Fish4Knowledge project HPC facilities automatic analysis tools ecology-related multimedia content high-performance computing job submission policies large scale data processing large-scale video processing load balancing long-term underwater camera footage long-term underwater video monitoring parallel job manager underwater fish monitoring Algorithm design and analysis Cameras databases Environmental factors Marine animals Monitoring Streaming media

来源：评论

学校读者我要写书评

暂无评论

An efficient theta-join query processing in distributed environment

引用

JOURNAL OF PARALLEL AND DISTRIBUTED COMPUTING 2018年 121卷 42-52页

作者： Liu, Wenjie Li, Zhanhuai Northwestern Polytech Univ Sch Comp Xian 710072 Shaanxi Peoples R China

Theta-join query is very useful in many data analysis tasks, but it is not efficiently processed in distributed environment, especially in large scale data. Although there is much progress in dealing theta-join with MapReduce paradigm, the methods are either complex which require fundamental changes to MapReduce framework or only consider the overheads of load balance in the network, when data scale is large, they will make much computation cost and induce OOM (Out of Memory) errors. In this work, we propose a filter method for theta-join on the purpose of reducing the computation cost and achieving the minimum execution time in distributed environment. We consider not only the load balance in the cluster, but also the memory cost in parallel framework. We also propose a keys-based join solution for multi-way theta-join to reduce the data amount for cross product, then improve the performance of join efficiency. We implement our methods in a popular general-purpose data processing framework, Spark. The experimental results demonstrate that our methods can significantly improve the performance of theta-joins comparing with the state-of-art solutions. (C) 2018 Elsevier Inc. All rights reserved.

关键词： Parallel distributed framework Theta-join algorithm Query optimization large scale data processing

来源：评论

学校读者我要写书评

暂无评论

Accelerating MapReduce framework on multi-GPU systems

引用

CLUSTER COMPUTING-THE JOURNAL OF NETWORKS SOFTWARE TOOLS AND APPLICATIONS 2014年第2期17卷 293-301页

作者： Jiang, Hai Chen, Yi Qiao, Zhi Li, Kuan-Ching Ro, WonWoo Gaudiot, Jean-Luc Arkansas State Univ Dept Comp Sci Jonesboro AR USA Providence Univ Dept Comp Sci & Informat Engn Taichung Taiwan Yonsei Univ Sch Elect & Elect Engn Seoul 120749 South Korea Univ Calif Irvine Dept Elect Engn & Comp Sci Irvine CA USA

Graphics processors evolve rapidly and promise to support power-efficient, cost, differentiated price-performance, and scalable high performance computing. MapReduce is a well-known distributed programming model to ease the development of applications for large-scale data processing on a large number of commodity CPUs. When compared to CPUs, GPUs are an order of magnitude faster in terms of computation power and memory bandwidth, but they are harder to program. Although several studies have implemented the MapReduce model on GPUs, most of them are based on the single GPU model and bounded by a GPU memory with inefficient atomic operations. This paper focuses on the development of MGMR, a standalone MapReduce system that utilizes multiple GPUs to manage large-scale data processing beyond the GPU memory limitation, and also to eliminate serial atomic operations. Experimental results have demonstrated the effectiveness of MGMR in handling large data sets.

关键词： GPU MapReduce large scale data processing Multi-GPUs

来源：评论

学校读者我要写书评

暂无评论

MRO-MPI: MapReduce overlapping using MPI and an optimized data exchange policy

引用

PARALLEL COMPUTING 2013年第12期39卷 851-866页

作者： Mohamed, Hisham Marchand-Maillet, Stephane Univ Geneva Comp Vis & Multimedia Lab Viper Grp Geneva Switzerland

MapReduce is a programming model proposed to simplify large-scale data processing. In contrast, the message passing interface (MPI) standard is extensively used for algorithmic parallelization, as it accommodates an efficient communication infrastructure. In the original implementation of MapReduce, the reduce function can only start processing following termination of the map function. If the map function is slow for any reason, this will affect the whole running time. In this paper, we propose MapReduce overlapping using MPI, which is an adapted structure of the MapReduce programming model for fast intensive data processing. Our implementation is based on running the map and the reduce functions concurrently in parallel by exchanging partial intermediate data between them in a pipeline fashion using MPI. At the same time, we maintain the usability and the simplicity of MapReduce. Experimental results based on three different applications (WordCount, Distributed Inverted Indexing and Distributed Approximate Similarity Search) show a good speedup compared to the earlier versions of MapReduce such as Hadoop and the available MPI-MapReduce implementations. (C) 2013 Elsevier B.V. All rights reserved.

关键词： Map Reduce overlapping MPI-MapReduce Parallel Map Reduce Big data large scale data processing

来源：评论

学校读者我要写书评

暂无评论

Pladipus Enables Universal Distributed Computing in Proteomics Bioinformatics

引用

JOURNAL OF PROTEOME RESEARCH 2016年第3期15卷 707-712页

作者： Verheggen, Kenneth Maddelein, Davy Hulstaert, Niels Martens, Lennart Barsnes, Harald Vandel, Marc VIB Ctr Med Biotechnol Albert Baertsoenkaai 3 B-9000 Ghent Belgium Univ Ghent Dept Biochem Albert Baertsoenkaai 3 B-9000 Ghent Belgium Univ Ghent Bioinformat Inst Ghent Albert Baertsoenkaai 3 B-9000 Ghent Belgium Univ Bergen Dept Biomed Prote Unit Postboks 7804 N-5020 Bergen Norway Univ Bergen Dept Clin Sci KG Jebsen Ctr Diabet Res Postboks 7804 N-5020 Bergen Norway

The use of proteomics bioinformatics substantially contributes to an improved understanding of proteomes, but this novel and in-depth knowledge comes at the cost of increased computational complexity. Parallelization across multiple computers, a strategy termed distributed computing, can be used to handle this increased complexity;however, setting up and maintaining a distributed computing infrastructure requires resources and skills that are not readily available to most research groups. Here we propose a free and open -source framework named Pladipus that greatly facilitates the establishment of distributed computing networks for proteomics bioinformatics tools. Pladipus is straightforward to install and operate thanks to its user-friendly graphical interface, allowing complex bioinformatics tasks to be run easily on a network instead of a single computer. As a result, any researcher can benefit from the increased computational efficiency provided by distributed computing, hence empowering them to tackle more complex bioinformatics challenges. Notably, it enables any research group to perform large-scale reprocessing of publicly available proteomics data, thus supporting the scientific community in mining these data for novel discoveries.

关键词： distributed computing large scale data processing cluster computing parallelization

来源：评论

学校读者我要写书评

暂无评论

Distributed Human Activity data processing using HASC Tool 11

Distributed Human Activity Data Processing using HASC Tool

引用

13th International Conference on Ubiquitous Computing (UbiComp)

作者： Kawaguchi, Nobuo Ogawa, Nobuhiro Iwasaki, Yohei Kaji, Katsuhiko Nagoya Univ Grad Sch Engn Nagoya Aichi Japan

ISBN: (纸本)9781450309103

To accelerate and simplify human activity recognition research, we have been developing a data processing tool named "HASC Tool." As the activity corpus becomes huge, it is not simple to handle the large number of files because it takes a lot of time to process. In this paper, we propose a distributed data processing mechanism which is implemented in the HASC Tool. By using the system, we can simply scale the local system into distributed processing. We also show the preliminary experimental result.

关键词： Activity Recognition Activity Understandings Distributed data processing large scale data processing HASC

来源：评论

学校读者我要写书评

暂无评论

data Multiverse: The Uncertainty Challenge of Future Big data Analytics 2nd

Data Multiverse: The Uncertainty Challenge of Future Big Dat...

引用

COST Action IC1302 2nd International KEYSTONE Conference (IKC)

作者： Tudoran, Radu Nicolae, Bogdan Brasche, Goetz Huawei German Res Ctr Riesstr 25 D-80992 Munich Germany

ISBN: (纸本)9783319536408;9783319536392

With the explosion of data sizes, extracting valuable insight out of big data becomes increasingly difficult. New challenges begin to emerge that complement traditional, long-standing challenges related to building scalable infrastructure and runtime systems that can deliver the desired level of performance and resource efficiency. This vision paper focuses on one such challenge, which we refer to as the analytics uncertainty: with so much data available from so many sources, it is difficult to anticipate what the data can be useful for, if at all. As a consequence, it is difficult to anticipate what data processing algorithms and methods are the most appropriate to extract value and insight. In this context, we contribute with a study on current big data analytics state-of-art, the use cases where the analytics uncertainty is emerging as a problem and future research directions to address them.

关键词： Big data analytics large scale data processing data access model data uncertainty Approximate computing

来源：评论

学校读者我要写书评

暂无评论

Flexible and efficient computation in large data centres

Flexible and efficient computation in large data centres

引用

作者： Gog, Ionel Corneliu University of Cambridge

学位级别：ET

Increasingly, online computer applications rely on large-scale data analyses to offer personalised and improved products. These large-scale analyses are performed on distributed data processing execution engines that run on thousands of networked machines housed within an individual data centre. These execution engines provide, to the programmer, the illusion of running data analysis workflows on a single machine, and offer programming interfaces that shield developers from the intricacies of implementing parallel, fault-tolerant computations. Many such execution engines exist, but they embed assumptions about the computations they execute, or only target certain types of computations. Understanding these assumptions involves substantial study and experimentation. Thus, developers find it difficult to determine which execution engine is best, and even if they did, they become “locked in” because engineering effort is required to port workflows. In this dissertation, I first argue that in order to execute data analysis computations efficiently, and to flexibly choose the best engines, the way we specify data analysis computations should be decoupled from the execution engines that run the computations. I propose an architecture for decoupling data processing, together with Musketeer, my proof-of-concept implementation of this architecture. In Musketeer, developers express data analysis computations using their preferred programming interface. These are translated into a common intermediate representation from which code is generated and executed on the most appropriate execution engine. I show that Musketeer can be used to write data analysis computations directly, and these can execute on many execution engines because Musketeer automatically generates code that is competitive with optimised hand-written implementations. The diverse execution engines cause different workflow types to coexist within a data centre, opening up both opportunities for sharing and potenti

关键词： large scale data processing cluster management dataflow systems cluster scheduling

来源：评论

学校读者我要写书评

暂无评论

preAssemble: a tool for automatic sequencer trace data processing

引用

BMC BIOINFORMATICS 2006年第1期7卷 22-22页

作者： Adzhubei, AA Laerdahl, JK Vlasova, AV Norwegian Sch Vet Sci NO-0033 Oslo Norway Univ Oslo Rikshosp Ctr Mol Biol & Neurosci CMBN NO-0027 Oslo Norway VA Engelhardt Mol Biol Inst Moscow 117984 Russia Univ Oslo Ctr Biotechnol NO-0317 Oslo Norway

Background: Trace or chromatogram files ( raw data) are produced by automatic nucleic acid sequencing equipment or sequencers. Each file contains information which can be interpreted by specialised software to reveal the sequence ( base calling). This is done by the sequencer proprietary software or publicly available programs. Depending on the size of a sequencing project the number of trace files can vary from just a few to thousands of files. Sequencing quality assessment on various criteria is important at the stage preceding clustering and contig assembly. Two major publicly available packages - Phred and Staden are used by preAssemble to perform sequence quality processing. Results: The preAssemble pre-assembly sequence processing pipeline has been developed for small to large scale automatic processing of DNA sequencer chromatogram ( trace) data. The Staden Package Pregap4 module and base-calling program Phred are utilized in the pipeline, which produces detailed and self-explanatory output that can be displayed with a web browser. preAssemble can be used successfully with very little previous experience, however options for parameter tuning are provided for advanced users. preAssemble runs under UNIX and LINUX operating systems. It is available for downloading and will run as stand-alone software. It can also be accessed on the Norwegian Salmon Genome Project web site where preAssemble jobs can be run on the project server. Conclusion: preAssemble is a tool allowing to perform quality assessment of sequences generated by automatic sequencing equipment. preAssemble is flexible since both interactive jobs on the preAssemble server and the stand alone downloadable version are available. Virtually no previous experience is necessary to run a default preAssemble job, on the other hand options for parameter tuning are provided. Consequently preAssemble can be used as efficiently for just several trace files as for large scale sequence processing.

关键词： Trace File Staden Package Small scale Sequence large scale data processing Sequencing Equipment

来源：评论

学校读者我要写书评

暂无评论

TempWeb 2018 Chairs' Welcome and Organization 18

TempWeb 2018 Chairs' Welcome and Organization

引用

Companion Proceedings of the The Web Conference 2018

作者： Marc Spaniol Ricardo Baeza-Yates Julien Masanès University of Caen Normandy Caen France NTENT Carlsbad CA USA USA Internet Memory Foundation Paris France

ISBN: (纸本)9781450356404

Time is a key dimension to understand the Web. It is fair to say that it has not received yet all the attention it deserves and TempWeb is an attempt to help remedy this situation by putting time as the center of its reflection. Studying time in this context actually covers a large spectrum, from the extraction of temporal information and knowledge, to diachronic studies for the design of infrastructural and experimental settings enabling a proper observation of this dimension.

关键词： topic mining distributed data analytics web spam evolution systematic exploitation of web archives web science large scale data storage content evolution on the web web trends web scale data analytics data quality metrics community detection and evolution time aware web archiving data aggregation terminology evolution temporal web analytics web dynamics large scale data processing

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：