检索结果-内蒙古大学图书馆

Conference on Observatory Operations - Strategies, Processes, and Systems VIII

作者： Teplitz, Harry, I Rusholme, Benjamin Aussel, Herve Dabin, Christophe Helou, George Holliman, Mark Zacchei, Andrea CALTECH IPAC Pasadena CA 91125 USA Univ Paris Saclay Univ Paris Diderot Sorbonne Paris Cite AIMCEACNRS F-91191 Gif Sur Yvette France Inst Astrophys Paris 98bis Blvd Arago F-75014 Paris France Ctr Natl Etud Spatiales CNES Toulouse France Univ Edinburgh Edinburgh Midlothian Scotland INAF Osservatorio Astron Trieste Via GB Tiepolo 11 I-34131 Trieste Italy

ISBN: (纸本)9781510636866

Euclid is an ESA M-class mission to study the geometry and nature of the dark universe, slated for launch in mid-2022. NASA is participating in the mission through the contribution of the near-infrared detectors and associated electronics, the nomination of scientists for membership in the Euclid Consortium, and by establishing the Euclid NASA Science Center at IPAC (ENSCI) to support the US community As part of ENSCI' s work, we will participate in the Euclid Science Ground Segment (SGS) and build and operate the US Science data Center (SDC-US), which will be a node in the distributed data processing system for the mission. SDC-US is one of 10 data centers, and will contribute about 5% of the computing and data storage for the distributed system. We discuss lessons learned in developing a node in a distributed system. For example, there is a significant advantage to SDC-US development in sharing of knowledge, problem solving, and resource burden with other parts of the system. On the other hand, fitting into a system that is distributed geographically and relies on diverse computing environments results in added complexity in constructing SDC-US.

关键词： Euclid distributed data processing lessons learned US Science data Center

来源：评论

学校读者我要写书评

暂无评论

Learning Representations for Signal and data processing on Directed Graphs

Learning Representations for Signal and Data Processing on D...

引用

作者： Shafipour, Rasoul University of Rochester

学位级别：Ph.D.

Network processes are becoming increasingly ubiquitous, with examples ranging from the measurements of neural activities at different regions of the brain to infectious states of individuals in a population affected by an epidemic. Such network data can be conceptualized as graph signals supported on the vertices of the adopted graph abstraction to the network. Under the natural assumption that the signal properties relate to the underlying graph topology, the goal of graph signal processing (GSP) is to develop algorithms that fruitfully exploit this relational structure. This dissertation contributes to this effort by advancing signal representations for information processing on (possibly directed) networks. An instrumental GSP tool is the graph Fourier transform (GFT), which decomposes a graph signal into orthonormal components describing different modes of variation with respect to the graph topology. In the first part of this dissertation, we study the problem of constructing a graph Fourier transform (GFT) for directed graphs (digraphs). Unlike existing approaches, to capture low, medium, and high frequencies, we seek a digraph (D)GFT such that the orthonormal frequency components are as spread as possible in the graph spectral domain. To that end, we advocate a two-step design whereby we: (i) find the maximum directed variation (i.e., a novel notion of frequency on a digraph) a candidate basis vector can attain; and (ii) minimize a smooth spectral dispersion function over the achievable frequency range to obtain a spread DGFT basis. Both steps involve non-convex, orthonormality-constrained optimization problems, which are tackled via a provably-convergent feasible optimization method on the Stiefel manifold. We discuss a data-adaptive variant whereby a sparsifying orthonormal transform is learnt to encourage parsimonious representations of bandlimited signals. distributed graph filtering based on the learnt transform is investigated as well. Graph frequency a

关键词： Dictionary learning distributed data processing Graph Fourier transform Graph learning Graph signal processing Online optimization

来源：评论

学校读者我要写书评

暂无评论

Leveraging resource management for efficient performance of Apache Spark

引用

JOURNAL OF BIG data 2019年第1期6卷 1页

作者： Aziz, Khadija Zaidouni, Dounia Bellafkih, Mostafa Natl Inst Posts & Telecommun STRS Lab Rabat Morocco

Apache Spark is one of the most widely used open source processing framework for big data, it allows to process large datasets in parallel using a large number of nodes. Often, applications of this framework use resource management systems like YARN, which provide jobs a specific amount of resources for their execution. In addition, a distributed file system such as HDFS stores the data that is to be analyzed by the framework. This design allows sharing cluster resources effectively by running jobs on a single-node cluster or multi-nodes cluster infrastructure. Thus, one challenging issue is to realize effective resource management of these large cluster infrastructures in order to run distributed data analytics in an economically viable way. In this study, we use the Machine Learning library (MLlib) of Spark to implement different machine learning algorithms, then we manage the resources (CPU, memory, and Disk) in order to assess the performance of Apache Spark. In this paper, we present a review of various works that focus on resource management and data processing in Big data platforms. Furthermore, we perform a scalability analysis using Spark. We analyze the speedup and processing time. We deduce that from a certain number of nodes in the cluster, it is no longer necessary to add additional nodes to improve the speedup and the processing Time. Then, we investigate the tuning of the resource allocation in Spark. We showed that it is not only by allocating all the available resources we get better performance but it depends on how to tune the resource allocation. We propose new managed parameters and we show that they give better total processing time than the default parameters used by Spark. Finally, we study the Persistence of Resilient distributed datasets (RDDs) in Spark using machine learning algorithms. We show that one storage level gives the best execution time among all tested storage levels.

关键词： Resource management Performance Tuning distributed data processing Machine learning algorithms Apache Spark MLlib

来源：评论

学校读者我要写书评

暂无评论

A self-adapted swarm architecture to handle big data for "factories of the future"

引用

IFAC-PapersOnLine 2019年第13期52卷 916-921页

作者： Guerreiro, Guilherme Costa, Ruben Figueiras, Paulo Graça, Diogo Jardim-Gonçalves, Ricardo CTS UNINOVA Dep. de Eng. Eletrotécnica Faculdade de Ciências e Tecnologia FCT Universidade Nova de Lisboa Caparica2829-516 Portugal Volkswagen Autoeuropa Quinta da Marquesa Quinta do Anjo2954-024 Portugal

Currently, the manufacturing sector is facing a technological evolution with the so-called Industry 4.0. This poses a paradigm shift, enabling companies to be more competitive by taking advantage of innovative technologies(cloud computing, cyber-physical systems, big data analytics and deep learning), pursuing near-zero fault, near real-time reactivity to any problem, better traceability, more predictability in manufacturing, while working to achieve cheaper product customization. The challenges arise when the dimensionality of the data generated by manufacturing processes grows, affecting the performance of algorithms, decreasing it quickly as the dimension of the search space increases. Handling large datasets with a good performance in a limited time should be the main concern in Big data analytics. This paper focuses on a logistic process of car manufacturing, where batteries are unloaded from trucks to warehouse, and then to the point of fit, where they are assembled into the car. It presents a complete data-driven architecture, using a swarm approach for distributed data processing among all data stages, where processing nodes with different tasks and technologies can work cooperatively to complete a job. The work presented in this paper is funded by the EU project BOOST4.0, focusing on a smart manufacturing scenario for the automotive sector. © 2019, IFAC (International Federation of Automatic Control) Hosting by Elsevier Ltd. All rights reserved.

关键词： Manufacturing data processing Advanced Analytics Automotive industry Big data Cyber Physical System data Analytics data handling Deep learning Embedded systems Engineering education Large dataset Manufacture Real time systems Swarm intelligence data driven architectures distributed data processing Factories of the futures Industry4.0 Innovative technology Performance of algorithm Product customization Technological evolution

来源：评论

学校读者我要写书评

暂无评论

SANE: Smart Networks for Urban Citizen Participation 26

SANE: Smart Networks for Urban Citizen Participation

引用

26th International Conference on Telecommunications (ICT)

作者： Bornholdt, Heiko Jost, David Kisters, Philipp Rottleuthner, Michel Bade, Dirk Lamersdorf, Winfried Schmidt, Thomas C. Fischer, Mathias Univ Hamburg Hamburg Germany Hamburg Univ Appl Sci Hamburg Germany

ISBN: (纸本)9781728102733

The increasing urbanization results in a rising demand for smart city platforms that optimize limited resources and thus save resources and increase the quality of living at the same time. Several systems have evolved in the past to address such issues by providing platforms for collecting, sharing, and processing urban data. However, these existing platforms only partially address challenges like fostering the participation of citizens, protecting their privacy, and assuring certain levels of quality of information. In this paper, we present the architecture of SANE as an open, citizen-centric, scalable, and privacy-preserving smart city platform. SANE is intended as open platform on which citizens can contribute data but also hardware, without any central authority or control. Moreover, citizens maintain full control on their data and its usage. SANE comes with rich and distributed data analytics functionality that is intended to help citizens to answer data-related question in the context of smart cities.

关键词： Smart City Internet of Things Service-oriented Middleware Participatory Sensing distributed data processing Quality of Information

来源：评论

学校读者我要写书评

暂无评论

Information Spaces for Big data processing: Unification and Parallelization of Sequential Information Accumulation Procedures 21

Information Spaces for Big Data Processing: Unification and ...

引用

21st IEEE Conference on Business Informatics (IEEE CBI)

作者： Golubtsov, Peter Lomonosov Moscow State Univ Fac Phys Moscow Russia Natl Res Univ Higher Sch Econ Fac Business & Management Moscow Russia

ISBN: (纸本)9781728106502

In large-scale research, data are usually collected on many sites, have a huge volume, and new data are constantly generated. Since it is often impossible to collect all the relevant data on a single computer, much attention is paid to the algorithms that provide sequential or parallel accumulation of information and do not need to store all the original data. As an example of information accumulation, the Bayesian updating procedure for linear experiments is analyzed. The corresponding information spaces are defined and the relations between them are studied. It is shown that processing can be unified and simplified by introducing a special canonical form of information representation and transforming all the data and the original prior information into this form. Thanks to the rich algebraic properties of the canonical information space, the sequential Bayesian procedure allows various parallelization options that are ideally suited for distributed data processing platforms, such as Hadoop MapReduce. This opens up the possibility of a flexible and efficient scaling of information accumulation in distributed data processing systems.

关键词： distributed data processing Bayesian information update big data information spaces

来源：评论

学校读者我要写书评

暂无评论

Big data processing System for Analysis of GitHub Events 22

Big Data Processing System for Analysis of GitHub Events

引用

22nd International Conference on Soft Computing and Measurements (SCM)

作者： Voinov, Nikita Garzon, Katterine Rodriguez Nikiforov, Igor Drobintsev, Pavel Peter Great St Petersburg Polytech Univ Inst Comp Sci & Technol St Petersburg Russia

ISBN: (纸本)9781728136028

The article describes architecture of a big data processing system based on Apache Hadoop, Apache Flume and Apache Spark toolset. Application of the developed system is shown for storage and analysis of dataset containing generated events within GitHub repository - the world's largest web-service for version control using Git. System performance results are evaluated using chosen metrics.

关键词： Big data distributed data processing Apache Hadoop MapReduce GitHub repository

来源：评论

学校读者我要写书评

暂无评论

Industrial track: Architecting railway KPIs data processing with Big data technologies

Industrial track: Architecting railway KPIs data processing ...

引用

IEEE International Conference on Big data (Big data)

作者： Suleykin, Alexander Panfilov, Peter Bakhtadze, Natalya Russian Acad Sci VA Trapeznikov Inst Control Sci Moscow Russia Natl Res Univ Higher Sch Econ Sch Business Informat Moscow Russia Bauman Moscow State Tech Univ Moscow Russia

ISBN: (纸本)9781728108582

in our conducted research we have built the data processing pipeline for storing railway KPIs data based on Big data open-source technologies - Apache Hadoop, Kafka, Karim HDFS Connector. Spark, Airflow and PostgreSQL. Created methodology for data load testing allowed to iteratively perform data load tests with increased data size and evaluate needed cluster software and hardware resources and, finally, detected bottlenecks of solution. As a result of the research we proposed architecture for data processing and storage, gave recommendations on data pipeline optimization. In addition, we calculated approximate cluster machines sizing for current dataset volume for data processing and storage services.

关键词： Big data technologies distributed data processing Hadoop Spark railway KPIs

来源：评论

学校读者我要写书评

暂无评论

CONCEPTS AND TOOLS FOR THE EFFECTIVE AND EFFICIENT USE OF WEB ARCHIVES

CONCEPTS AND TOOLS FOR THE EFFECTIVE AND EFFICIENT USE OF WE...

引用

作者： Helge Holzmann Gottfried Wilhelm Leibniz Universitat Hannover

学位级别：博士

Web archives constitute valuable sources for researchers in various disciplines. How- ever, their sheer size, the typically broad scope and their temporal dimension make them difficult to work with. We have identified three views to access and explore Web archives from different perspectives: user-, data- and graph-centric. The natural way to look at the information in a Web archive is through a Web browser, just like the live Web is consumed. This is what we consider the user-centric view. The most commonly used tool to access a Web archive this way is the Wayback Machine, the Internet Archive's replay tool to render archived webpages. To facilitate the discovery of a page if the URL or timestamp of interest is unknown, we propose effective approaches to search Web archives by keyword with a temporal dimension through social bookmarks and labeled hyperlinks. Another way for users to find and access archived pages is past information on the current Web that is linked to the corresponding evidence in a Web archive. A presented tool for this purpose ensures coherent archived states of webpages related to a common object as rich temporal representations to be referenced and shared. Besides accessing a Web archive by closely reading individual pages like users do, distant reading methods enable analyzing archival collections at scale. This data-centric view enables analysis of the Web and its dynamics itself as well as the contents of archived pages. We address both angles: 1. by presenting a retrospective analysis of crawl meta- data on the size, age and growth of a Web dataset, 2. by proposing a programming framework for efficiently processing archival collections. ArchiveSpark operates on stan- dard formats to build research corpora from Web archives and facilitates the process of filtering as well as data extraction and derivation at scale. The third perspective is what we call the graph-centric view. Here, websites, pages or extracted facts are considered nodes in

关键词： Web archives temporal search distributed data processing Web analysis

来源：评论

学校读者我要写书评

暂无评论

DPFMDA: distributed and privatized framework for miRNA-Disease association prediction

引用

PATTERN RECOGNITION LETTERS 2018年第Jul.15期109卷 4-11页

作者： Chen, Lixin Liu, Bingtao Yan, Chenggang Hangzhou Dianzi Univ Hangzhou Zhejiang Peoples R China

In this paper, we developed a novel distributed and privatized framework for miRNA-disease association prediction (DPFMDA) to predict potential miRNA-disease associations. DPFMDA assumes that diseases and miRNAs can be represented as feature matrices. Through independent factorization, collaborative analysis and matrix re-factorization, feature matrices are estimated and potential associations in miRNA-disease association matrix are predicted. The proposed computational model is tested and verified by leave-one-out cross validation (LOOCV). The AUCs of global and local LOOCV are 0.8859 and 0.8267 respectively, which outperform the state-of-the-art computational models. DPFMDA is further evaluated with case studies of four important human complex diseases. We test Esophageal Neoplasms and Prostate Neoplasms using DPFMDA in the HMDD v2.0 database. For Breast Neoplasms, we use the HMDD v2.0 database which removes all the known miRNAs-disease associations on Breast Neoplasms. For Hepatocellular Carcinoma, HMDD v1.0 database is used as the input of DPFMDA and predicted results are verified with HMDD v2.0, dbDEMC and miR2Disease databases. The results show that 90% (Esophageal Neoplasms), 88% (Prostate Neoplasms), 98% (Breast Neoplasms) and 86% (Hepatocellular Carcinoma) of top 50 predicted miRNA-disease associations are confirmed by recent experimental studies respectively. It is anticipated that DPFMDA would be an effective and extensible method for potential miRNA-disease association prediction. (C) 2017 Elsevier B.V. All rights reserved.

关键词： miRNA-disease association prediction Computational model distributed data processing

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：