检索结果-内蒙古大学图书馆

Integrating data and compute-intensive workflows for uncertainty quantification in large-scale simulation: application to model-based hazard analysis

引用

INTERNATIONAL JOURNAL OF COMPUTER MATHEMATICS 2014年第4期91卷 730-747页

作者： Rohit, Shivaswamy Patra, Abani Chaudhary, Vipin SUNY Buffalo Dept Mech & Aerosp Engn Buffalo NY 14260 USA SUNY Buffalo Dept Comp Sci Buffalo NY 14260 USA

Complex inference from simulation ensembles used in uncertainty quantification leads to twin computational challenges of managing large amount of data and performing CPU-intensive computing. While algorithmic innovations using surrogates, localization and parallelization can make the problem feasible, one still has very large data and compute tasks. The problem of dealing with large data gets compounded when data warehousing and data mining are intertwined with computationally expensive tasks. We present here an approach to solving this problem by using a mix of hardware suitable for each task in a carefully orchestrated workflow. The computing environment is essentially an integration of Netezza database and high-performance cluster. It is based on the simple idea of segregating the data-intensive and compute-intensive tasks and assigning the right architecture for them. We present here the layout of the computing model and the new computational scheme adopted to generate probabilistic hazard maps.

关键词： data-intensive computing high-performance computing uncertainty quantification workflow Netezza

来源：评论

学校读者我要写书评

暂无评论

Toward efficient execution of data-intensive workflows

引用

JOURNAL OF SUPERcomputing 2021年第8期77卷 7989-8012页

作者： Sukhoroslov, Oleg Russian Acad Sci Inst Informat Transmiss Problems Moscow Russia

Workflows that consume and produce large amounts of data are being widely used in modern scientific computing and data processing pipelines. Scheduling of data-intensive workflows requires a careful management of data transfers between tasks, since network contention can significantly impact the workflow execution time. The paper presents and evaluates several scheduling algorithms, data transfer strategies and optimizations aimed at efficient execution of data-intensive workflows. The studied approaches reduce or completely avoid network contention by explicit scheduling of data transfers and incorporate several optimizations, such as data caching, chunked and peer-to-peer data transfers. The results of experimental study demonstrate that the relative performance of different approaches depends on the workflow properties, data staging strategy and system configuration. The proposed CAS-L1 heuristic with additional data transfer optimizations achieves the best results.

关键词： Workflows data-intensive computing Task scheduling data management Simulation

来源：评论

学校读者我要写书评

暂无评论

Accelerating tropical cyclone analysis using LambdaRAM, a distributed data cache over wide-area ultra-fast networks

引用

FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE 2009年第2期25卷 184-191页

作者： Vishwanath, Venkatrarn Burns, Robert Leigh, Jason Seablom, Michael Univ Illinois EVL Chicago IL 60607 USA NASA GSFC SIVO Greenbelt MD 20771 USA

data-intensive scientific applications require rapid access to local and geographically distributed data, however, there are significant I/O latency bottlenecks associated with storage systems and wide-area networking. LambdaRAM is a high-performance, multi-dimensional, distributed cache, that takes advantage of memory from multiple clusters interconnected by ultra-high-speed networking, to provide applications with rapid access to both local and remote data. It mitigates latency bottlenecks by employing proactive latency-mitigation heuristics based on an application's access patterns. We present results using LambdaRAM to rapidly stride through remote multi-dimensional NASA Modeling, Analysis and Prediction (MAP) 2006 project datasets, based on time and geographical coordinates, to compute wind shear for cyclone and hurricane and tropical cyclone analysis. Our current experiments have demonstrated up to a 20-fold speedup in the computation of wind shear with LambdaRAM. (C) 2008 Elsevier B.V. All rights reserved.

关键词： data-intensive computing Multi-dimensional remote data striding Climate modeling and analysis Distributed data caches LambdaGrids Tropical cyclone analysis Hurricane analysis

来源：评论

学校读者我要写书评

暂无评论

Moving huge scientific datasets over the Internet

引用

CONCURRENCY AND COMPUTATION-PRACTICE & EXPERIENCE 2011年第18期23卷 2404-2420页

作者： Liu, Wantao Tieman, Brian Kettimuthu, Rajkumar Foster, Ian Beihang Univ Sch Engn & Comp Sci Beijing Peoples R China Univ Chicago Dept Comp Sci Chicago IL 60637 USA Argonne Natl Lab Adv Photon Source Downers Grove Township IL USA Argonne Natl Lab Div Math & Comp Sci Downers Grove Township IL USA Univ Chicago Computat Inst Argonne Natl Lab Chicago IL 60637 USA

Modern scientific experiments can generate hundreds of gigabytes to terabytes or even petabytes of data that may be maintained in large numbers of relatively small files. Frequently, these data must be disseminated to remote collaborators or computational centers for data analysis. Moving this dataset with high performance and strong robustness and providing a simple interface for users are challenging tasks. We present a data transfer framework comprising a high-performance data transfer library based on GridFTP, an extensible data scheduler with four data scheduling policies, and a GUI that allows users to transfer their dataset easily, reliably, and securely. This system incorporates automatic tuning mechanisms to select at runtime the number of concurrent threads to be used for transfers. Also included are restart mechanisms for handling client, network, and server failures. Experimental results indicate that our data transfer system can significantly improve data transfer performance and can recover well from failures. Copyright (C) 2011 John Wiley & Sons, Ltd.

关键词： data transfer data-intensive computing Internet e-science

来源：评论

学校读者我要写书评

暂无评论

Quantitative retrieval of geophysical parameters using satellite data

引用

COMPUTER 2008年第4期41卷 33-+页

作者： Xue, Yong Wan, Wei Li, Yingjie Guang, Jie Bai, Linyian Wang, Ying Ai, Jianwen Chinese Acad Sci Satte Key Lab Remote Sensing Sci Inst Remote Sensing Applicat Beijing 100864 Peoples R China Beijing Normal Univ Beijing 100875 Peoples R China Chinese Acad Sci IRSA Beijing 100864 Peoples R China Grad Univ Chinese Acad Sci Beijing Peoples R China

A reliable atmospheric remote-sensing monitor uses physical or statistical models for which the parameters must be retrieved quantitatively. However, such retrieval is data-intensive. High-resolution, wide-range, and long-duration observations produce several terabytes of data each day. Processing such massive volumes of data into scientific aerosol products involves addressing several computational problems. Processing Level 1B data for Level 2 aerosol products for a single day requires 13 Gbytes total to achieve full coverage of China's main land surface.

关键词： data-intensive computing distributed system grid computing remote sensing

来源：评论

学校读者我要写书评

暂无评论

Collective input/output under memory constraints

引用

INTERNATIONAL JOURNAL OF HIGH PERFORMANCE computing APPLICATIONS 2015年第1期29卷 21-36页

作者： Lu, Yin Chen, Yong Zhuang, Yu Liu, Jialin Thakur, Rajeev Texas Tech Univ Dept Comp Sci Lubbock TX 79409 USA Texas Tech Univ Dept Comp Sci Data Intens Scalable Comp Lab Lubbock TX 79409 USA Argonne Natl Lab Div Math & Comp Sci Argonne IL 60439 USA

Compared with current high-performance computing (HPC) systems, exascale systems are expected to have much less memory per node, which can significantly reduce necessary collective input/output (I/O) performance. In this study, we introduce a memory-conscious collective I/O strategy that takes into account memory capacity and bandwidth constraints. The new strategy restricts aggregation data traffic within disjointed subgroups, coordinates I/O accesses in intranode and internode layers, and determines I/O aggregators at run time considering memory consumption among processes. We have prototyped the design and evaluated it with commonly used benchmarks to verify its potential. The evaluation results demonstrate that this strategy holds promise in mitigating the memory pressure, alleviating the contention for memory bandwidth, and improving the I/O performance for projected extreme-scale systems. Given the importance of supporting increasingly data-intensive workloads and projected memory constraints on increasingly larger scale HPC systems, this new memory-conscious collective I/O can have a significant positive impact on scientific discovery productivity.

关键词： Exascale system parallel input output collective input output many-core architecture data-intensive computing high-performance computing

来源：评论

学校读者我要写书评

暂无评论

ActiveSort: Efficient external sorting using active SSDs in the MapReduce framework

引用

FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE 2016年第0期65卷 76-89页

作者： Lee, Young-Sik Quero, Luis Cavazos Kim, Sang-Hoon Kim, Jin-Soo Maeng, Seungryoul Korea Adv Inst Sci & Technol Sch Comp 291 Daehak Ro Daejeon 34141 South Korea Sungkyunkwan Univ Coll Informat & Commun Engn 2066 Seobu Ro Suwon 16419 South Korea

In the last decades, there has been an explosion in the volume of data to be processed by data-intensive computing applications. As a result, processing I/O operations efficiently has become an important challenge. SSDs (solid state drives) are an effective solution that not only improves the I/O throughput but also reduces the amount of I/O transfer by adopting the concept of active SSDs. Active SSDs offload a part of the data-processing tasks usually performed in the host to the SSD. Offloading data-processing tasks removes extra data transfer and improves the overall data processing performance. In this work, we propose ActiveSort, a novel mechanism to improve the external sorting algorithm using the concept of active SSDs. External sorting is used extensively in the data-intensive computing frameworks such as Hadoop. By performing merge operations on-the-fly within the SSD, ActiveSort reduces the amount of I/O transfer and improves the performance of external sorting in Hadoop. Our evaluation results on a real SSD platform indicate that the Hadoop applications using ActiveSort outperform the original Hadoop by up to 36.1%. ActiveSort reduces the amount of write by up to 40.4%, thereby improving the lifetime of the SSD. (C) 2016 Elsevier B.V. All rights reserved.

关键词： data-intensive computing MapReduce External sorting Solid state drives In-storage processing

来源：评论

学校读者我要写书评

暂无评论

Analysis and semantic querying in large biomedical image datasets

引用

COMPUTER 2008年第4期41卷 52-+页

作者： Kumar, Vijay S. Narayanan, Sivaramakrishnan Kurc, Tahsin Kong, Jun Gurcan, Metin N. Saltz, Joel H. Ohio State Univ Columbus OH 43210 USA Ohio State Univ Dept Biomed Informat Columbus OH 43210 USA Ohio State Univ Dept Elect & Comp Engn Columbus OH 43210 USA

Digital microscopy opens new opportunities to study a disease's tissue characteristics at the cellular level. Traditionally, human experts visually examine tissue and classify images, then make a diagnosis. This process is time-consuming and the sheer size of image datasets makes gleaning information from digital microscopy slides dataintensive. The authors' work addresses problems in two areas: the processing of large digitized slides for analysis and the semantic query of annotated images and image regions in a large image dataset.

关键词： processor architectures parallel architectures information storage and retrieval artificial intelligence computer vision computer-assisted prognosis data-intensive computing

来源：评论

学校读者我要写书评

暂无评论

Hadoop Extensions for Distributed computing on Reconfigurable Active SSD Clusters

引用

ACM TRANSACTIONS ON ARCHITECTURE AND CODE OPTIMIZATION 2014年第2期11卷 191-216页

作者： Kaitoua, Abdulrahman Hajj, Hazem Saghir, Mazen A. R. Artail, Hassan Akkary, Haitham Awad, Mariette Sharafeddine, Mageda Mershad, Khaleel Amer Univ Beirut Dept Elect & Comp Engn Beirut Lebanon Texas A&M Univ Qatar Dept Comp Engn Doha Qatar

In this article, we propose new extensions to Hadoop to enable clusters of reconfigurable active solid-state drives (RASSDs) to process streaming data from SSDs using FPGAs. We also develop an analytical model to estimate the performance of RASSD clusters running under Hadoop. Using the Hadoop RASSD platform and network simulators, we validate our design and demonstrate its impact on performance for different worldoads taken from Stanford's Phoenix MapReduce project. Our results show that for a hardware acceleration factor of 20 x, compute-intensive workloads processing 153MB of data can run up to 11 x faster than a standard Hadoop cluster.

关键词： Reconfigurable computing RASSD Hadoop data-intensive computing middleware active storage

来源：评论

学校读者我要写书评

暂无评论

ProDA: An end-to-end wavelet-based OLAP system for massive datasets

引用

COMPUTER 2008年第4期41卷 69-+页

作者： Shahabi, Cyrus Jahangiri, Mehrdad Banaei-Kashani, Farnoush Univ So Calif Dept Comp Sci InfoLAB Los Angeles CA 90089 USA

By design, developers optimize traditional databases for transactional rather than analytical query processing. These databases support only a few basic analytical queries with nonoptimal performance and, therefore, provide inappropriate tools for analyzing massive datasets. Online analytical processing tools have emerged to address the limitations of traditional databases and spreadsheet applications. OLAP tools support complex analytical queries and handle massive datasets. The authors' ProDA system uses these tools to enable exploratory analysis of massive multidimensional datasets.

关键词： data-intensive computing database management scientific applications online query processing wavelet transformation range aggregate queries information technology and systems OLAP tools

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：