检索结果-内蒙古大学图书馆

Trusted Provenance of Collaborative, Adaptive, Process-Based data processing pipelines

Trusted Provenance of Collaborative, Adaptive, Process-Based...

Enterprise Design, Operations, and Computing (EDOC) Forum / IDAMS Workshop / SoEA4EE Workshop / IRESEARCH Workshop / MIDas4CS Workshop

作者： Stage, Ludwig Univ Groningen Informat Syst Grp Groningen Netherlands

ISBN: (纸本)9783031547119;9783031547126

The abundance of data nowadays provides a lot of opportunities to gain insights in many domains. data processing pipelines are one of the ways used to automate different data processing approaches and are widely used by both industry and academia. In many cases data and processing are available in distributed environments and the workflow technology is a suitable one to deal with the automation of data processing pipelines and support at the same time collaborative, trial-and-error experimentation in term of pipeline architecture for different application and scientific domains. In addition to the need for flexibility during the execution of the pipelines, there is a lack of trust in such collaborative settings where interactions cross organisational boundaries. Capturing provenance information related to the pipeline execution and the processed data is common and certainly a first step towards enabling trusted collaborations. However, current solutions do not capture change of any aspect of the processing pipelines themselves or changes in the data used, and thus do not allow for provenance of change. Therefore, the objective of this work is to investigate how provenance of workflow or data change during execution can be enabled. As a first step we have developed a preliminary architecture of a service - the Provenance Holder - which enables provenance of collaborative, adaptive data processing pipelines in a trusted manner. In our future work, we will focus on the concepts necessary to enable trusted provenance of change, as well as on the detailed service design, realization and evaluation.

关键词： Provenance of Change Reproducibility Trust Collaborative Processes data processing pipelines Workflow evolution provenance Provenance of ad-hoc workflow change

来源：评论

学校读者我要写书评

暂无评论

Visualization System Requirements for data processing Pipeline Design and Optimization

引用

IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 2017年第8期23卷 2028-2041页

作者： von Landesberger, Tatiana Fellner, Dieter W. Ruddle, Roy A. Tech Univ Darmstadt Interact Graph Syst Grp D-64289 Darmstadt Germany Tech Univ Darmstadt CGV Inst D-64289 Darmstadt Germany Univ Leeds Leeds LS2 9JT W Yorkshire England

The rising quantity and complexity of data creates a need to design and optimize data processing pipelines-the set of data processing steps, parameters and algorithms that perform operations on the data. Visualization can support this process but, although there are many examples of systems for visual parameter analysis, there remains a need to systematically assess users' requirements and match those requirements to exemplar visualization methods. This article presents a new characterization of the requirements for pipeline design and optimization. This characterization is based on both a review of the literature and first-hand assessment of eight application case studies. We also match these requirements with exemplar functionality provided by existing visualization tools. Thus, we provide end-users and visualization developers with a way of identifying functionality that addresses data processing problems in an application. We also identify seven future challenges for visualization research that are not met by the capabilities of today's systems.

关键词： Visualization systems requirement analysis data processing pipelines

来源：评论

学校读者我要写书评

暂无评论

VERTEXICA: Your Relational Friend for Graph Analytics!

引用

PROCEEDINGS OF THE VLDB ENDOWMENT 2014年第13期7卷 1669-1672页

作者： Jindal, Alekh Rawlani, Praynaa Wu, Eugene Madden, Samuel Deshpande, Amol Stonebraker, Mike MIT CSAIL Cambridge MA 02139 USA Univ Maryland Baltimore MD 21201 USA

In this paper, we present VERTEXICA, a graph analytics tools on top of a relational database, which is user friendly and yet highly efficient. Instead of constraining programmers to SQL, VERTEXICA offers a popular vertex-centric query interface, which is more natural for analysts to express many graph queries. The programmers simply provide their vertex-compute functions and VERTEXICA takes care of efficiently executing them in the standard SQL engine. The advantage of using VERTEXICA is its ability to leverage the relational features and enable much more sophisticated graph analysis. These include expressing graph algorithms which are difficult in vertex centric but straightforward in SQL and the ability to compose end to-end data processing pipelines, including pre- and post-processing of graphs as well as combining multiple algorithms for deeper insights. VERTEXICA has a graphical user interface and we outline several demonstration scenarios including, interactive graph analysis, complex graph analysis, and continuous and time series analysis.

关键词： Graph theory data handling Graphical user interfaces Pipeline processing systems Query processing Time series analysis data processing pipelines Graph algorithms Graph analytics Multiple algorithms Post processing Query interfaces Relational database Relational features

来源：评论

学校读者我要写书评

暂无评论

Development of the Program Execution System Architecture (PESA) for MSE 7

Development of the Program Execution System Architecture (PE...

引用

Conference on Software and Cyberinfrastructure for Astronomy VII

作者： Surace, C. Marshall, J. Sobeck, J. Szeto, K. Ho, K. Vermeulen, T. Wells, L. Wipper, C. Withington, K. Aix Marseille Univ CNRS Lab Astrophys Marseille Marseille France Natl Res Council Canada Herzberg Astron & Astrophys 5071 West Saanich Rd Victoria BC V9E 2E7 Canada CFHT Corp 65-1238 Mamalahoa Hwy Kamuela HI 96743 USA

ISBN: (纸本)9781510653603;9781510653597

Maunakea Spectroscopic Explorer (MSE) is a telescope dedicated to multi-fibers spectroscopy and IFUs observations of the sky. Program Execution System Architecture (PESA) is one of the systems of MSE, responsible for planning, executing, reducing, and distributing science products from survey programs. Work is being done to design PESA in a modular way to include several sophisticated software tools, organized into an operational framework. This paper describes the first step of its organization and the concepts that will be used in the development of PESA.

关键词： MSE data processing pipelines Computing Infrastructure Observation organization

来源：评论

学校读者我要写书评

暂无评论

JWST Science data Products 5

JWST Science Data Products

引用

Conference on Observatory Operations - Strategies, Processes, and Systems V

作者： Swade, Daryl Bushouse, Howard Greene, Gretchen Swam, Michael Space Telescope Sci Inst Baltimore MD 21218 USA

ISBN: (纸本)9780819496171

Science data products for James Webb Space Telescope (JWST) observations will be generated by the data Management Subsystem (DMS) within the JWST Science and Operations Center (S&OC) at the Space Telescope Science Institute (STScI). data processing pipelines within the DMS will produce uncalibrated and calibrated exposure files, as well as higher level data products that result from combined exposures, such as mosaic images. Information to support the science observations, for example data from engineering telemetry, proposer inputs, and observation planning will be captured and incorporated into the science data products. All files will be generated in Flexible Image Transport System (FITS) format. The data products will be made available through the Mikulski Archive for Space Telescopes (MAST) and adhere to International Virtual Observatory Alliance (IVOA) standard data protocols.

关键词： science data products data Management James Webb Space Telescope data processing pipelines

来源：评论

学校读者我要写书评

暂无评论

Text processing and Analysis Pipeline for Scientific Literature 3

Text Processing and Analysis Pipeline for Scientific Literat...

引用

International Conference on Advances in Computing, Communication and Applied Informatics (ACCAI)

作者： Singh, Inderjeet Satyam Semwal, Ashutosh Singh, Shivansh Gupta, Gouri Chandigarh Univ Mohali Dept Comp Sci Mohali Punjab India

ISBN: (纸本)9798350389432;9798350389449;9798350389456

This paper presents an exploratory data analysis-driven text processing pipeline for scientific publications based on the Scopus dataset. Sorting texts, spotting trends, and making recommendations are the goals. Pre-processing, cluster, and classification analyses are all incorporated into the suggested methodology, which produces a user recommendation software system. A parametrical method that takes user preferences into account extracts semantic information to overcome the obstacles associated with data preparation. An ensemble approach is used for user profile clustering, and an entropy-based ensemble technique is used for classification. The NLoN software performs exceptionally well when it comes to textual data classification. The analysis of the Scopus dataset highlights the importance of data processing pipelines by assisting with entity extraction and sentence classification. By combining information retrieval, data mining, and natural language processing, the work improves text processing for scientific publications.

关键词： Scopus dataset Text processing Exploratory data analysis (EDA) Classification Use-recommendations Preprocessing Cluster analysis Parametrical approach Semantic features data preparation Entropy-based technique NloN package Categorization Entity extraction data processing pipelines Information retrieval data mining Natural language processing Scientific literature

来源：评论

学校读者我要写书评

暂无评论

Fastai: A Layered API for Deep Learning

引用

INFORMATION 2020年第2期11卷 108-108页

作者： Howard, Jeremy Gugger, Sylvain Fastai San Francisco CA 94117 USA Univ San Francisco Data Sci Inst San Francisco CA 94117 USA

fastai is a deep learning library which provides practitioners with high-level components that can quickly and easily provide state-of-the-art results in standard deep learning domains, and provides researchers with low-level components that can be mixed and matched to build new approaches. It aims to do both things without substantial compromises in ease of use, flexibility, or performance. This is possible thanks to a carefully layered architecture, which expresses common underlying patterns of many deep learning and data processing techniques in terms of decoupled abstractions. These abstractions can be expressed concisely and clearly by leveraging the dynamism of the underlying Python language and the flexibility of the PyTorch library. fastai includes: a new type dispatch system for Python along with a semantic type hierarchy for tensors;a GPU-optimized computer vision library which can be extended in pure Python;an optimizer which refactors out the common functionality of modern optimizers into two basic pieces, allowing optimization algorithms to be implemented in 4-5 lines of code;a novel 2-way callback system that can access any part of the data, model, or optimizer and change it at any point during training;a new data block API;and much more. We used this library to successfully create a complete deep learning course, which we were able to write more quickly than using previous approaches, and the code was more clear. The library is already in wide use in research, industry, and teaching.

关键词： deep learning data processing pipelines

来源：评论

学校读者我要写书评

暂无评论

Access Control with Delegated Authorization Policy Evaluation for data-Driven Microservice Workflows

引用

FUTURE INTERNET 2017年第4期9卷 58-58页

作者： Preuveneers, Davy Joosen, Wouter Katholieke Univ Leuven IMEC DistriNet Celestijnenlaan 200A B-3001 Heverlee Belgium

Microservices offer a compelling competitive advantage for building data flow systems as a choreography of self-contained data endpoints that each implement a specific data processing functionality. Such a 'single responsibility principle' design makes them well suited for constructing scalable and flexible data integration and real-time data flow applications. In this paper, we investigate microservice based data processing workflows from a security point of view, i.e., (1) how to constrain data processing workflows with respect to dynamic authorization policies granting or denying access to certain microservice results depending on the flow of the data;(2) how to let multiple microservices contribute to a collective data-driven authorization decision and (3) how to put adequate measures in place such that the data within each individual microservice is protected against illegitimate access from unauthorized users or other microservices. Due to this multifold objective, enforcing access control on the data endpoints to prevent information leakage or preserve one's privacy becomes far more challenging, as authorization policies can have dependencies and decision outcomes cross-cutting data in multiple microservices. To address this challenge, we present and evaluate a workflow-oriented authorization framework that enforces authorization policies in a decentralized manner and where the delegated policy evaluation leverages feature toggles that are managed at runtime by software circuit breakers to secure the distributed data processing workflows. The benefit of our solution is that, on the one hand, authorization policies restrict access to the data endpoints of the microservices, and on the other hand, microservices can safely rely on other data endpoints to collectively evaluate cross-cutting access control decisions without having to rely on a shared storage backend holding all the necessary information for the policy evaluation.

关键词： authorization policy-based access control microservices data processing pipelines workflows feature toggles circuit breakers

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：