检索结果-内蒙古大学图书馆

Review of classical dimensionality reduction and sample selection methods for large-scale data processing

NEUROCOMPUTING 2019年 328卷 5-15页

作者： Xu, Xinzheng Liang, Tianming Zhu, Jiong Zheng, Dong Sun, Tongfeng China Univ Min & Technol Sch Comp Sci & Technol Xuzhou 221116 Jiangsu Peoples R China Guangxi High Sch Key Lab Complex Syst & Computat Nanning 530006 Guangxi Peoples R China

In the era of big data, all types of data with increasing samples and high-dimensional attributes are demonstrating their important roles in various fields, such as data mining, pattern recognition and machine learning, etc. Meanwhile, machine learning algorithms are being effectively applied in large-scale data processing. This paper mainly reviews the classical dimensionality reduction and sample selection methods based on machine learning algorithms for large-scale data processing. Firstly, the paper provides a brief overview to the classical sample selection and dimensionality reduction methods. Then, it pays attention to the applications of those methods and their combinations with the classical machine learning methods, such as clustering, random forest, fuzzy set, and heuristic algorithms, particularly deep leaning methods. Furthermore, the paper primarily introduces the application frameworks that combine sample selection and dimensionality reduction in the context of two aspects: sequential and simultaneous, which almost all get the ideal results in the processing of the large-scale training data contrasting to the original models. Lastly, we further conclude that sample selection and dimensionality reduction methods are essential and effective for the modern large-scale data processing. In the future work, the machine learning algorithms, especially the deep learning methods, will play a more important role in the processing of large-scale data. (c) 2018 Elsevier B.V. All rights reserved.

关键词： large-scale data processing Sample selection Dimensional reduction Machine learning methods

来源：评论

学校读者我要写书评

暂无评论

large-scale data processing software and performance instabilities within HEP grid environments

引用

INTERNATIONAL JOURNAL OF GRID AND UTILITY COMPUTING 2019年第4期10卷 402-414页

作者： Datskova, Olga Shi, Wedong Univ Houston Dept Comp Sci Houston TX 77204 USA

large tasks running on grids and clouds have introduced a need for stability guarantees from geographically spanning resources, where failures are handled pre-emptively. Detecting performance inefficiencies in such cases is difficult. While individual services implement fault-tolerance, the behaviour of interacting failures within tightly-coupled systems is less understood. This paper describes an approach to modelling performance of production tasks running within the ALICE grid. We provide an overview of the ALICE data and software workflow for production jobs. Event states are then constructed, based on data centre job, computing, storage and user behaviour. We then address the question of analysing failures within the context of operational instabilities, occurring in production grid environments. The results demonstrate that operational issues can be detected and described according to the principle service layers involved. This can guide users, central and data centre experts to take action in advance of service failure effects.

关键词： HEP grid computing large-scale data processing performance modelling failure fault-tolerance software development

来源：评论

学校读者我要写书评

暂无评论

On the large-scale Graph data processing for User Interface Testing in Big data Science Projects 8

On the Large-scale Graph Data Processing for User Interface ...

引用

8th IEEE International Conference on Big data (Big data)

作者： Uygun, Yasin Oguz, Ramazan Faruk Olmezogullari, Erdi Aktas, Mehmet S. Yildiz Tecn Univ Comp Engn Istanbul Turkey Microsoft Dev Ctr Oslo Norway

ISBN: (纸本)9781728162515

In functional User Interface testing, test scenarios are written with respect to the requirements that are specified by test analysts. Usually, a test analyst focuses on base URLs and HTML components while collecting requirements of User Interface test scenarios. A base URL is essentially a unit segment of large scale graph data. It has mostly dynamic shape and is used to navigate pages amongst application's pages. We argue that even though dynamic URLs have additional important information about the content of the page, they are not being utilized in generating User Interface test scenarios. In this study, we address this lack of capability and focus on the development of a methodology that can support the usage of large-scale dynamic URL datasets in UI test script generation. Our proposed methodology is designed as an add-on tool that can be used on the top of the existing UI test automation tools to improve testing quality. We introduce a higher quality testing methodology to make the results more accurate, and we discuss the proposed methodology and give an overview of the implementation details followed by the evaluation results. We perform various performance evaluations to investigate how well the proposed algorithms scale under increasing data sizes. The results are promising and show the usability of the proposed methodology.

关键词： Big data Science Projects large-scale data processing Dynamic URLs NLP Clustering HDFS Map Reduce Word2vec Software Testing

来源：评论

学校读者我要写书评

暂无评论

Efficient and Portable Distribution Modeling for large-scale Scientific data processing with data-Parallel Primitives

引用

ALGORITHMS 2021年第10期14卷 285页

作者： Yang, Hao-Yi Lin, Zhi-Rong Wang, Ko-Chih Natl Taiwan Normal Univ Dept Comp Sci & Informat Engn Taipei 11677 Taiwan

The use of distribution-based data representation to handle large-scale scientific datasets is a promising approach. Distribution-based approaches often transform a scientific dataset into many distributions, each of which is calculated from a small number of samples. Most of the proposed parallel algorithms focus on modeling single distributions from many input samples efficiently, but these may not fit the large-scale scientific data processing scenario because they cannot utilize computing resources effectively. Histograms and the Gaussian Mixture Model (GMM) are the most popular distribution representations used to model scientific datasets. Therefore, we propose the use of multi-set histogram and GMM modeling algorithms for the scenario of large-scale scientific data processing. Our algorithms are developed by data-parallel primitives to achieve portability across different hardware architectures. We evaluate the performance of the proposed algorithms in detail and demonstrate use cases for scientific data processing.

关键词： large-scale data processing scientific dataset distribution-based approach parallel algorithm data-parallel primitive

来源：评论

学校读者我要写书评

暂无评论

Text Filtering for Harmful Document Classification Using Three-Word Co-Occurrence and large-scale data processing

引用

ELECTRONICS AND COMMUNICATIONS IN JAPAN 2015年第10期98卷 31-40页

作者： Otsuka, Takanobu Deng, Deyue Ito, Takayuki Nagoya Inst Technol Gokiso Grad Sch Informat Sci Nagoya Aichi Japan Nagoya Inst Technol Gokiso Master Technobusiness Adm Program Nagoya Aichi Japan

Young people are increasingly using the Internet. However, this creates the problem that the material they encounter may adversely affect them. Therefore we propose a method of automatically classifying harmful sentences. Research on information filtering has been improving the performance of filters by introducing co-occurrence information. Extended from two words co-occurrence information, which is commonly studied in such work, we have created training data using co-occurrence information with three words. However, compared with two-word co-occurrence information, the processing time becomes a problem because of the increase in the amount of training data. In addition, we have found that noise caused by the increase in co-occurrences exceeds the capacity of double-precision floating-point calculations. We improved processing speed by implementing a text filtering system with three-word co-occurrence using a Bayesian filter in order to parallelize a fast MyISAM database. In addition, by using BigDecimal to remove the noise caused by the increase in the number of co-occurrences, we achieved a high F-value. (C) 2015 Wiley Periodicals, Inc.

关键词： text filtering three-word co-occurrence large-scale data processing

来源：评论

学校读者我要写书评

暂无评论

data processing Pipeline of Short-Term Depression Detection with large-scale dataset

Data Processing Pipeline of Short-Term Depression Detection ...

引用

IEEE International Conference on Big data and Smart Computing (BigComp)

作者： Lee, Yonggeon Noh, Youngtae Lee, Uichin KENTECH Energy AI Naju Jeollanam Do South Korea Korea Adv Inst Sci & Technol Sch Comp Daejeon South Korea

ISBN: (纸本)9781665475785

Depression is a common, recurring mental disorder that causes significant impairment in people's lives. In recent years, ubiquitous computing using mobile phones can monitor behavioral patterns relevant to depressive symptoms in-the-wild. In this paper, we propose data processing pipeline of short-term depression detection using mobile sensor data. We build a group model classified by depression severity for capturing depressive mood in a short-period time to handle data quality and data imbalance problem in a large-scale dataset. We expect the group model to identify and characterize digital phenotype representing each depressive group as a middle step toward personalization.

关键词： large-scale data processing Short-Term Depression Detection Positive Computing Mobile Computing

来源：评论

学校读者我要写书评

暂无评论

Parallel Optimization for large scale Interferometric Synthetic Aperture Radar data processing

引用

REMOTE SENSING 2023年第7期15卷 1850页

作者： Zhang, Weikang You, Haihang Wang, Chao Zhang, Hong Tang, Yixian Chinese Acad Sci Inst Comp Technol State Key Lab Processors Beijing 100190 Peoples R China Zhongguancun Lab Beijing 100094 Peoples R China Chinese Acad Sci Aerosp Informat Res Inst Key Lab Digital Earth Sci Beijing 100094 Peoples R China

Interferometric synthetic aperture radar (InSAR) has developed rapidly over the past years and is considered as an important method for surface deformation monitoring, benefiting from growing data quantities and improving data quality. However, the handing of SAR big data poses significant challenges for related algorithms and pipeline, particularly in large-scale SAR data processing. In addition, InSAR algorithms are highly complex, and their task dependencies are intricate. There is a lack of efficient optimization models and task scheduling for InSAR pipeline. In this paper, we design parallel time-series InSAR processing models based on multi-thread technology for high efficiency in processing InSAR big data. These models concentrate on parallelizing critical algorithms that have high complexity, with a focus on deconstructing two computationally intensive algorithms through loop unrolling. Our parallel models have shown a significant improvement of 10-20 times in performance. We have also developed a parallel optimization tool, Simultaneous Task Automatic Runtime (STAR), which utilizes a data flow optimization strategy with thread pool technology to address the problem of low CPU utilization resulting from multiple modules and task dependencies in the InSAR processing pipeline. STAR provides a data-driven pipeline and enables concurrent execution of multiple tasks, with greater flexibility to keep the CPU busy and further improve CPU utilization through predetermined task flow. Additionally, a supercomputing-based system has been constructed for processing massive InSAR scientific big data and providing technical support for nationwide surface deformation measurement, in accordance with the framework of time series InSAR data processing. Using this system, we processed InSAR data with the volumes of 500 TB and 700 TB in 5 and 7 days, respectively. Finally we generated two maps of land surface deformation all over China.

关键词： InSAR large-scale data processing parallel optimization task scheduling STAR supercomputing

来源：评论

学校读者我要写书评

暂无评论

An automated parallel genetic algorithm with parametric adaptation for distributed data analysis

引用

SCIENTIFIC REPORTS 2025年第1期15卷 1-16页

作者： Al-Terkawi, Laila Migliavacca, Matteo Int Univ Kuwait IUK Al Ardiya Kuwait Univ Kent Sch Comp Canterbury England

Unleashing the potential of large-scale data analysis requires advanced computational methods capable of managing the immense size and complexity of distributed data. Genetic algorithms (GAs), known for their adaptability, benefit significantly from parallelization, prompting ongoing enhancements to boost performance further. This study proposes the integration of automatic termination and population sizing mechanisms into parallel GAs to augment their flexibility and effectiveness. We extend PDMS-BioHEL and PDMD-BioHEL, two parallel GA-based classifiers implemented on the Spark platform, and through extensive experimentation, demonstrate the efficacy of our approach in enhancing computational efficiency and user-friendliness. However, while these automated strategies significantly reduce the need for manual parameter tuning, thereby increasing time efficiency, they may sometimes lead to a slight reduction in final solution accuracy, particularly under complex scenario conditions. This trade-off between efficiency and accuracy is critical, especially when high precision is paramount. Our techniques enable more efficient and effective large-scale data analysis using parallel GAs, providing a robust foundation for future advancements and inviting further investigation into balancing these aspects.

关键词： Genetic algorithms (GAs) GAs parameter control Classification large-scale data processing Spark

来源：评论

学校读者我要写书评

暂无评论

Efficient Management and processing of Massive InSAR Images Using an HPC-Based Cloud Platform

引用

IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING 2024年 17卷 2866-2876页

作者： Wu, Zherong Ma, Peifeng Zhang, Xinyang Ye, Guangen Cornell Univ Sch Integrat Plant Sci Ithaca NY 14853 USA Chinese Univ Hong Kong Inst Space & Earth Informat Sci Shatin Hong Kong Peoples R China Chinese Univ Hong Kong Dept Geog & Resource Management Hong Kong Peoples R China Chinese Univ Hong Kong Shenzhen Res Inst Shenzhen 518057 Peoples R China

Significant progress has occurred in interferometric synthetic aperture radar (InSAR), emerging as a crucial technique for monitoring surface deformation. This evolution is attributed to expanded synthetic aperture radar (SAR) data availability and improved data quality. However, effectively managing and processing SAR big data presents substantial challenges for algorithms and pipelines, especially in large-scale contexts. In this article, we introduce a parallel time-series InSAR processing platform that leverages high-performance computing (HPC) clusters for efficiently managing and processing large-scale SAR data and incorporates graphics processing unit (GPU) acceleration to significantly enhance the speed and efficiency of specific InSAR processing algorithms. Our approach encompasses high-quality data compression, integration of classic InSAR models, and the introduction of a robust distributed scatterer InSAR method for time-series processing. The platform efficiently handles massive data, featuring a parallel optimization tool for acceleration. In addition, it provides web-based two-dimensional (2-D) result visualization and 3-D outcome representation for comprehensive user understanding. To illustrate our platform's capabilities, we applied it to 40 Sentinel-1 SAR data scenes from Tibet (2017-2019). Our data compression technique notably reduces data size, reducing mask data by 87.5% and coherence data to 25% of its original size. Leveraging HPC and GPU, we achieved a 50% reduction in registration computation time. This study offers valuable insights and a comprehensive platform for InSAR practitioners, facilitating calculations and enhancing comprehension of surface deformation processes. Our system's improved processing efficiency, coupled with a variety of InSAR methods, makes it an alternative choice for InSAR data handling and analysis.

关键词： High-performance computing (HPC) large-scale data processing time-series interferometric synthetic aperture radar (InSAR)

来源：评论

学校读者我要写书评

暂无评论

Towards efficient and accurate approximation: tensor decomposition based on randomized block Krylov iteration

引用

SIGNAL IMAGE AND VIDEO processing 2024年第8-9期18卷 6287-6297页

作者： Qiu, Yichun Sun, Weijun Zhou, Guoxu Zhao, Qibin Guangdong Univ Technol Sch Automat Guangzhou 510006 Peoples R China Guangdong Hong Kong Macao Joint Lab Smart Discrete Hong Kong 510006 Guangdong Peoples R China Minist Educ Key Lab Intelligent Detect & Internet Things Mfg Guangzhou 510006 Peoples R China Minist Educ Joint Int Res Lab Intelligent Informat Proc & Syst Guangzhou 510006 Peoples R China RIKEN Ctr Adv Intelligence Project AIP Tokyo 1030027 Japan

Tensor decomposition methods are inefficient when dealing with low-rank approximation of large-scale data. Randomized tensor decomposition has emerged to meet this need, but most existing methods exhibit high computational costs in handling large-scale tensors and poor approximation accuracy in noisy data scenarios. In this work, a Tucker decomposition method based on randomized block Krylov iteration (rBKI-TK) is proposed to reduce computational complexity and guarantee approximation accuracy by employing cumulative sketches rather than randomized initialization to construct a better projection space with fewer iterations. In addition, a hierarchical tensor ring decomposition based on rBKI-TK is proposed to enhance the scalability of the rBKI-TK method. Numerical results demonstrate the efficiency and accuracy of the proposed methods in large-scale and noisy data processing.

关键词： Tensor decomposition Low-rank approximation Randomized block Krylov iteration Noise reduction large-scale data processing

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：