检索结果-内蒙古大学图书馆

Technology for Designing Tools for the Process and Analysis of data from Very Large Scale distributed Satellite Archives

引用

ATMOSPHERIC AND OCEANIC OPTICS 2017年第1期30卷 84-88页

作者： Kashnitskii, A. V. Lupyan, E. A. Balashov, I. V. Konstantinova, A. M. Russian Acad Sci Space Res Inst Moscow 117997 Russia

The rapid increase in the volume of Earth satellite observation data over recent years makes more necessary the problem of developing new technologies for effective data search, selection, and processing within very large constantly updated distributed archives. The paper describes the features of such technologies developed at the Space Research Institute, Russian Academy of Sciences (IKI RAS). These techniques provide the design of various data processing tools for satellite data analysis with the use of distributed computing resources of remote sensing data processing and archiving centers. Advantages and capabilities of the approaches suggested are described, as well as examples of implemented tools for distributed processing of data from various satellite remote sensing systems. The examples given show the capabilities of using the tools for the analysis of various atmospheric and ocean surface phenomena.

关键词： remote sensing information systems distributed data processing satellite data processing very large data archives data management technologies ocean and atmosphere remote observation approaches

来源：评论

学校读者我要写书评

暂无评论

distributed Meta-Learning for Context Based on Culture Heritage Collection data in an Immersive Reality System

引用

Procedia Computer Science 2022年 210卷 70-77页

作者： Rahim Rahmani Susanne Tienken Department of Computer and Systems Sciences Stockholm University Sweden Department of Slavic and Baltic Languages Finnish Dutch and German Stockholm University Sweden

This article proposes a new workflow for Culture Heritage (CH) Collection data in which distributed meta-learning with context-based fusion is required. The multidisciplinary research outlines a new paradigm for collaborative creation of a context- based data fusion CH workflow. Whose criterion restricts the search space to the cultural sector to include multiple other CH object types, texts and predominantly analyzes visualizations of CH object metadata. The approach promotes user-driven content creation and offsets economic models, thereby rewarding creators and performers. In response to these challenges, we propose a framework for bringing about massive and real-time presence and awareness on the Internet through an Internet of Things infrastructure to connect objects, context and intangible assets. We enable an online virtual world that incorporates proximity of pervasive information, objects, people, processes, data and places. Finally, we investigate some new ways to achieve immersive experiences via distributed meta-learning computing and point out the necessity to do more with regard to collaborative interaction creation.

关键词： distributed data processing Creative technologies distributed system Immersive participation IoT

来源：评论

学校读者我要写书评

暂无评论

Benchmarking geospatial database on Kubernetes cluster

引用

EURASIP JOURNAL ON ADVANCES IN SIGNAL processing 2021年第1期2021卷 1-29页

作者： Sharma, Bharti Bansal, Poonam Chugh, Mohak Chauhan, Adisakshya Anand, Prateek Hua, Qiaozhi Jain, Achin GGSIPU MSIT Delhi India NSUT East Campus Delhi India Hubei Univ Arts & Sci Comp Sch Xiangyang 441000 Peoples R China Bharati Vidyapeeths Coll Engn Delhi India

Kubernetes is an open-source container orchestration system for automating container application operations and has been considered to deploy various kinds of container workloads. Traditional geo-databases face frequent scalability issues while dealing with dense and complex spatial data. Despite plenty of research work in the comparison of relational and NoSQL databases in handling geospatial data, there is a shortage of existing knowledge about the performance of geo-database in a clustered environment like Kubernetes. This paper presents benchmarking of PostgreSQL/PostGIS geospatial databases operating on a clustered environment against non-clustered environments. The benchmarking process considers the average execution times of geospatial structured query language (SQL) queries on multiple hardware configurations to compare the environments based on handling computationally expensive queries involving SQL operations and PostGIS functions. The geospatial queries operate on data imported from OpenStreetMap into PostgreSQL/PostGIS. The clustered environment powered by Kubernetes demonstrated promising improvements in the average execution times of computationally expensive geospatial SQL queries on all considered hardware configurations compared to their average execution times in non-clustered environments.

关键词： distributed data processing Geospatial databases Cluster computing Geospatial-data Geospatial-databases Benchmarking database-as-a-service (DBaaS)

来源：评论

学校读者我要写书评

暂无评论

Market-based frequency domain decomposition for automated mode shape estimation in wireless sensor networks

引用

STRUCTURAL CONTROL & HEALTH MONITORING 2010年第7期17卷 808-824页

作者： Zimmerman, Andrew T. Lynch, Jerome P. Univ Michigan Dept Civil & Environm Engn Ann Arbor MI 48105 USA

Wireless sensing technology has paved the way for the cost-effective deployment of dense networks of sensing transducers within large structural systems. By leveraging the embedded computing power residing within networks of wireless sensors, it has been shown that powerful data analyses can be performed autonomously and in-network, without the need for central data processing. In this study, the power and flexibility of agent-based data processing in the wireless structural monitoring environment is illuminated through the application of market-based techniques to in-network mode shape estimation. Specifically, by drawing on previous wireless sensor work in both decentralized frequency domain decomposition (FDD) and market-based resource allocation, an algorithm derived from free-market principles is developed through which an agent-based wireless sensor network can autonomously and optimally shift emphasis between improving the accuracy of its mode shape calculations and reducing its dependency on any of the traditional limitations of wireless sensor networks: processing time, storage capacity, and power consumption. The developed algorithm is validated by estimating mode shapes using a network of wireless sensors deployed on the mezzanine balcony of Hill Auditorium located at the University of Michigan. Copyright (C) 2010 John Wiley & Sons, Ltd.

关键词： structural health monitoring wireless sensor networks distributed data processing agent-based systems frequency domain decomposition modal identification

来源：评论

学校读者我要写书评

暂无评论

Leveraging resource management for efficient performance of Apache Spark

引用

JOURNAL OF BIG data 2019年第1期6卷 1页

作者： Aziz, Khadija Zaidouni, Dounia Bellafkih, Mostafa Natl Inst Posts & Telecommun STRS Lab Rabat Morocco

Apache Spark is one of the most widely used open source processing framework for big data, it allows to process large datasets in parallel using a large number of nodes. Often, applications of this framework use resource management systems like YARN, which provide jobs a specific amount of resources for their execution. In addition, a distributed file system such as HDFS stores the data that is to be analyzed by the framework. This design allows sharing cluster resources effectively by running jobs on a single-node cluster or multi-nodes cluster infrastructure. Thus, one challenging issue is to realize effective resource management of these large cluster infrastructures in order to run distributed data analytics in an economically viable way. In this study, we use the Machine Learning library (MLlib) of Spark to implement different machine learning algorithms, then we manage the resources (CPU, memory, and Disk) in order to assess the performance of Apache Spark. In this paper, we present a review of various works that focus on resource management and data processing in Big data platforms. Furthermore, we perform a scalability analysis using Spark. We analyze the speedup and processing time. We deduce that from a certain number of nodes in the cluster, it is no longer necessary to add additional nodes to improve the speedup and the processing Time. Then, we investigate the tuning of the resource allocation in Spark. We showed that it is not only by allocating all the available resources we get better performance but it depends on how to tune the resource allocation. We propose new managed parameters and we show that they give better total processing time than the default parameters used by Spark. Finally, we study the Persistence of Resilient distributed datasets (RDDs) in Spark using machine learning algorithms. We show that one storage level gives the best execution time among all tested storage levels.

关键词： Resource management Performance Tuning distributed data processing Machine learning algorithms Apache Spark MLlib

来源：评论

学校读者我要写书评

暂无评论

Learning-Based Dynamic Memory Allocation Schemes for Apache Spark data processing

引用

IEEE TRANSACTIONS ON CLOUD COMPUTING 2024年第1期12卷 13-25页

作者： Jia, Danlin Wang, Li Valencia, Natalia Bhimani, Janki Sheng, Bo Mi, Ningfang Northeastern Univ Dept Elect & Comp Engn Boston MA 02115 USA Florida Int Univ Sch Comp & Informat Sci Miami FL 33199 USA Univ Massachusetts Dept Comp Sci Boston MA 02125 USA

Apache Spark is an in-memory analytic framework that has been adopted in the industry and research fields. Two memory managers, Static and Unified, are available in Spark to allocate memory for caching Resilient distributed datasets (RDDs) and executing tasks. However, we find that the static memory manager (SMM) lacks flexibility, while the unified memory manager (UMM) puts heavy pressure on the garbage collection of the JVM on which Spark resides. To address these issues, we design a learning-based bidirectional usage-bounded memory allocation scheme to support dynamic memory allocation with the consideration of both memory demands and latency introduced by garbage collection. We first develop an auto-tuning memory manager (ATuMm) that adopts an intuitive feedback-based learning solution. However, ATuMm is a slow learner that can only alter the states of Java Virtual Memory (JVM) Heap in a limited range. That is, ATuMm decides to increase or decrease the boundary between the execution and storage memory pools by a fixed portion of JVM Heap size. To overcome this shortcoming, we further develop a new reinforcement learning-based memory manager (Q-ATuMm) that uses a Q-learning intelligent agent to dynamically learn and tune the partition of JVM Heap. We implement our new memory managers in Spark 2.4.0 and evaluate them by conducting experiments in a real Spark cluster. Our experimental results show that our memory manager can reduce the total garbage collection time and thus further improve Spark applications' performance (i.e., reduced latency) compared to the existing Spark memory management solutions. By integrating our machine learning-driven memory manager into Spark, we can further obtain around 1.3x times reduction in the latency.

关键词： Sparks Memory management Q-learning Java Task analysis Cluster computing Heuristic algorithms JVM memory management distributed data processing machine learning Apache Spark

来源：评论

学校读者我要写书评

暂无评论

Facilitating rapid prototyping in the distributed data analytics platform OODIDA via active-code replacement

引用

ARRAY 2020年 8卷

作者： Ulm, Gregor Smith, Simon Nilsson, Adrian Gustavsson, Emil Jirstrand, Mats Fraunhofer Chalmers Res Ctr Ind Math Chalmers Sci Pk S-41288 Gothenburg Sweden Fraunhofer Ctr Machine Learning Chalmers Sci Pk S-41288 Gothenburg Sweden

OODIDA (On-board/Off-board distributed data Analytics) is a platform for distributed real-time analytics, targeting fleets of reference vehicles in the automotive industry. Its users are data analysts. The bulk of the data analytics tasks are performed by clients (on-board), while a central cloud server performs supplementary tasks (off-board). OODIDA can be automatically packaged and deployed, which necessitates restarting parts of the system, or all of it. As this is potentially disruptive, we added the ability to execute user-defined Python modules on clients as well as the server. These modules can be replaced without restarting any part of the system;they can even be replaced between iterations of an ongoing assignment. This feature is referred to as active-code replacement. It facilitates use cases such as iterative A/B testing of machine learning algorithms or modifying experimental algorithms on-the-fly. Various safeguards are in place to ensure that custom code does not have harmful consequences, for instance by limiting the allowed types for return values or prohibiting importing of certain modules of the Python standard library. Consistency of results is achieved by majority vote, which prevents tainted state. Our evaluation shows that active-code replacement can be done in less than a second in an idealized setting whereas a standard deployment takes many orders of magnitude more time. The main contribution of this paper is the description of a relatively straightforward approach to active-code replacement that is very user-friendly. It enables a data analyst to quickly execute custom code on the cloud server as well as on client devices. Sensible safeguards and design decisions ensure that this feature can be used by non-specialists who are not familiar with the implementation of OODIDA in general or this feature in particular. As a consequence of adding the active-code replacement feature, OODIDA is now very well-suited for rapid prototyping.

关键词： distributed computing Concurrent computing distributed data processing Hot swapping Code replacement Erlang

来源：评论

学校读者我要写书评

暂无评论

Secure Multiparty Computation of Approximations

引用

ACM TRANSACTIONS ON ALGORITHMS 2006年第3期2卷 435-472页

作者： Feigenbaum, Joan Ishai, Yuval Malkin, Tal Nissim, Kobbi Strauss, Martin J. Wright, Rebecca N. Yale Univ Dept Comp Sci POB 2158 New Haven CT 06520 USA Technion Dept Comp Sci IL-32000 Haifa Israel Columbia Univ Dept Comp Sci New York NY 10027 USA Ben Gurion Univ Negev Dept Comp Sci IL-84105 Beer Sheva Israel Univ Michigan Dept Math Ann Arbor MI 48109 USA Univ Michigan Dept EECS Ann Arbor MI 48109 USA Stevens Inst Technol Dept Comp Sci Hoboken NJ 07030 USA

Approximation algorithms can sometimes provide efficient solutions when no efficient exact computation is known. In particular, approximations are often useful in a distributed setting where the inputs are held by different parties and may be extremely large. Furthermore, for some applications, the parties want to compute a function of their inputs securely without revealing more information than necessary. In this work, we study the question of simultaneously addressing the above efficiency and security concerns via what we call secure approximations. We start by extending standard definitions of secure (exact) computation to the setting of secure approximations. Our definitions guarantee that no additional information is revealed by the approximation beyond what follows from the output of the function being approximated. We then study the complexity of specific secure approximation problems. In particular, we obtain a sublinear-communication protocol for securely approximating the Hamming distance and a polynomial-time protocol for securely approximating the permanent and related #P-hard problems.

关键词： Privacy sublinear communication distributed data processing

来源：评论

学校读者我要写书评

暂无评论

A Survey on IoT Big data Analytic Systems: Current and Future

引用

IEEE INTERNET OF THINGS JOURNAL 2022年第2期9卷 1024-1036页

作者： Sasaki, Yuya Osaka Univ Grad Sch Informat Sci Technol Suita Osaka 5650871 Japan

The Internet of Things (IoT) has become widespread around the world. Since a large number of diverse devices, such as vehicles, household electrical appliances, smart phones, and environmental sensors are connected to the Internet, we can obtain a large volume of diverse IoT data, known as IoT big data. The generation of IoT big data means that efficient analytic systems are needed for many application scenarios, for example, to optimize urban planning, solve air pollution problems, and improve business decisions. In this survey, we review current systems that can efficiently analyze IoT data. Existing systems can be categorized into batch and stream processing systems. We explore Hadoop- and Spark-based batch processing systems for spatiotemporal and trajectory data. We also review fog- and edge-aware stream processing systems. Although many existing systems can efficiently and effectively analyze specific data and tasks, no system exists that can handle all characteristics of IoT big data: volume, velocity, variety, veracity, and variability. We present some open issues and discuss the future of IoT big data analytic systems. This survey aims to help researchers and practitioners better understand current systems and develop new IoT big data analytic systems.

关键词： Internet of Things Sensors Big data data analysis Smart cities Batch production systems Meteorology distributed data processing edge computing smart city spatiotemporal data streaming processing trajectory data

来源：评论

学校读者我要写书评

暂无评论

ARCog-NET: An Aerial Robot Cognitive Network Architecture for Swarm Applications Development

引用

IEEE ACCESS 2024年 12卷 129040-129063页

作者： Ramos, Gabryel S. Henriques, Felipe Da R. Haddad, Diego B. Andrade, Fabio A. A. Pinto, Milena F. Fed Ctr Technol Educ Celso Suckow da Fonseca CEFET BR-20271110 Rio De Janeiro Brazil Univ South Eastern Norway USN Fac Technol Nat Sci & Maritime Sci Dept Microsyst N-3184 Borre Norway NORCE Norwegian Res Ctr Drones & Autonomous Syst N-9294 Tromso Norway

This work presents an advanced cognitive architecture for networked aerial robots to implement autonomous swarm systems effectively. It focuses on designing, implementing, and evaluating an architecture that enables unmanned aerial vehicles (UAVs) to coordinate and cooperate for complex tasks, with or without human intervention. Inspired by artificial intelligence, cognitive science, and robotics, the architecture integrates perception, planning, decision-making, and adaptive learning to optimize swarm behavior in dynamic environments. The architecture uses a distributed processing model based on the "edge-fog-cloud" (EFC) concept. Edge-level robots handle real-time data collection, local decision-making, and environmental perception. Fog-level vehicles manage intermediate processing and supervision of the groups, while cloud servers perform comprehensive data analysis and long-term storage, being the higher-level hierarchy of the framework. This structure allows efficient distribution of computational tasks, with critical decisions made at the robot level and complex analysis done in the fog or cloud. The implementation of ARCog-NET involves deploying a multi-agent simulation system using the Robot Operating System (ROS) and Gazebo simulator, facilitating the integration of sensors, communication protocols, and data processing algorithms. The performance evaluation demonstrates the architecture's effectiveness in a wind farm inspection scenario, where the UAV swarm exhibits improved trajectory planning, collision avoidance, and data processing efficiency. Simulation results show that ARCog-NET reduces latency, increases data throughput, and enhances operational effectiveness, providing a robust platform for future developers to focus on applications and direct robot control methods.

关键词： Cognitive architecture edge-fog-cloud technologies swarm systems distributed data processing autonomy of aerial robots Cognitive architecture edge-fog-cloud technologies swarm systems distributed data processing autonomy of aerial robots

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：