检索结果-内蒙古大学图书馆

7th International Conference on Cloud Computing and Services Science (CLOSER)

作者： Verginadis, Yiannis Alshabani, Iyad Mentzas, Gregoris Stojanovic, Nenad Natl Tech Univ Athens Inst Commun & Comp Syst Athens Greece Activeeon Valbonne France Nissatech Innovat Ctr Nish Serbia

ISBN: (纸本)9789897582431

Among the greatest challenges of cloud computing is to automatically and efficiently exploit infrastructural resources in a way that minimises cloud fees without compromising the performance of resource demanding cloud applications. In this aspect the consideration of using processing nodes at the edge of the network, increases considerably the complexity of these challenges. PrEstoCloud idea encapsulates a dynamic, distributed, self-adaptive and proactively configurable architecture for processing big data streams. In particular, PrEstoCloud aims to combine real-time big data, mobile processing and cloud computing research in a unique way that entails proactiveness of cloud resources use and extension of the fog computing paradigm to the extreme edge of the network. The envisioned PrEstoCloud solution is driven by the microservices paradigm and has been structured across five different conceptual layers: i) Meta-management;ii) Control;iii) Cloud infrastructure;iv) Cloud/Edge communication and v) Devices, layers.

关键词： big data processing Dynamic Resources Management Edge Computing

来源：评论

学校读者我要写书评

暂无评论

Job Scheduling without Prior Information in big data processing Systems 37

Job Scheduling without Prior Information in Big Data Process...

引用

37th IEEE International Conference on Distributed Computing Systems (ICDCS)

作者： Hu, Zhiming Li, Baochun Qin, Zheng Goh, Rick Siow Mong Univ Toronto Dept Elect & Comp Engn Toronto ON Canada ASTAR Inst High Performance Comp Singapore Singapore

ISBN: (纸本)9781538617915

Job scheduling plays an important role in improving the overall system performance in big data processing frameworks. Simple job scheduling policies, such as Fair and FIFO scheduling, do not consider job sizes and may degrade the performance when jobs of varying sizes arrive. More elaborate job scheduling policies make the convenient assumption that jobs are recurring, and complete information about their sizes is available from their prior runs. In this paper, we design and implement an efficient and practical job scheduler for big data processing systems to achieve better performance even without prior information about job sizes. The superior performance of our job scheduler originates from the design of multiple level priority queues, where jobs are demoted to lower priority queues if the amount of service consumed so far reaches a certain threshold. In this case, jobs in need of a small amount of service can finish in the topmost several levels of queues, while jobs that need a large amount of service to complete are moved to lower priority queues to avoid head-of-line blocking. Our new job scheduler can effectively mimic the shortest job first scheduling policy without knowing the job sizes in advance. To demonstrate its performance, we have implemented our new job scheduler in YARN, a popular resource manager used by Hadoop/ Spark, and validated its performance with both experiments on real datasets and large-scale trace-driven simulations. Our experimental and simulation results have strongly confirmed the effectiveness of our design: our new job scheduler can reduce the average job response time of the Fair scheduler by up to 45%.

关键词： job scheduling big data processing multilevel feedback queue

来源：评论

学校读者我要写书评

暂无评论

Security Intelligence Centers for big data processing 5

Security Intelligence Centers for Big Data Processing

引用

IEEE 5th International Conference on Future Internet of Things and Cloud (FiCloud)

作者： Miloslavskaya, Natalia Natl Res Nucl Univ MEPhI Moscow Engn Phys Inst Moscow Russia

ISBN: (纸本)9781538632819

Today numerous information security (IS) incidents in organizations' networks have become not only more sophisticated but also damaging. Hence the systems with proper security services in place to mitigate and promptly respond to IS threats by helping organizations better understand their current network situation, as well as to perform routine work in big IS-related data processing in automatic mode are needed as never before. They are known as Security Operations Centers (SOCs) and Security Intelligence Centers (SICs) as their next evolution step. The key features of SICs are summarized. The SIC business logic and data architecture are proposed. These results lead to the main area of further research.

关键词： big data IT big data processing Information Security Security Ingelligence Security Intelligence Center

来源：评论

学校读者我要写书评

暂无评论

big data processing of School Shooting Archives 16

Big Data Processing of School Shooting Archives

引用

16th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL)

作者： Farag, Mohamed Nakate, Pranav Fox, Edward A. Virginia Tech Blacksburg VA 24061 USA

ISBN: (纸本)9781450342292

Web archives about school shootings consist of webpages that may or may not be relevant to the events of interest. There are 3 main goals of this work;first is to clean the webpages, which involves getting rid of the stop words and non-relevant parts of a webpage. The second goal is to select just webpages relevant to the events of interest. The third goal is to upload the cleaned and relevant webpages to Apache Solr so that they are easily accessible. We show the details of all the steps required to achieve these goals. The results show that representative Web archives are noisy, with 2% - 40% relevant content. By cleaning the archives, we aid researchers to focus on relevant content for their analysis.

关键词： Web Archives big data processing Classification Digital Libraries

来源：评论

学校读者我要写书评

暂无评论

A flexible I/O arbitration framework for netCDF-based big data processing workflows on high-end supercomputers

引用

CONCURRENCY AND COMPUTATION-PRACTICE & EXPERIENCE 2017年第15期29卷 1-12页

作者： Liao, Jianwei Gerofi, Balazs Lien, Guo-Yuan Miyoshi, Takemasa Nishizawa, Seiya Tomita, Hirofumi Liao, Wei-Keng Choudhary, Alok Ishikawa, Yutaka Southwest Univ China Coll Comp & Informat Sci Tianshen Rd 2 Chongqing Peoples R China Nanjing Univ State Key Lab Novel Software Technol Nanjing Jiangsu Peoples R China RIKEN Adv Inst Computat Sci Kobe Hyogo Japan Northwestern Univ Dept Elect Engn & Comp Sci Evanston IL USA

On the verge of the convergence between high-performance computing and big data processing, it has become increasingly prevalent to deploy large-scale data analytics workloads on high-end supercomputers. Such applications often come in the form of complex workflows with various different components, assimilating data from scientific simulations as well as from measurements streamed from sensor networks, such as radars and satellites. For example, as part of the Flagship 2020 (post-K) supercomputer project of Japan, RIKEN is investigating the feasibility of a highly accurate weather forecasting system that would provide a real-time outlook for severe guerrilla rainstorms. One of the main performance bottlenecks of this application is the lack of efficient communication among workflow components, which currently takes place over the parallel file *** this paper, we present an initial study of a direct communication framework designed for complex workflows that eliminates unnecessary file I/O among components. Specifically, we propose an I/O arbitration layer that provides direct parallel data transfer (both synchronous and asynchronous) among job components that rely on the netCDF interface for performing I/O operations. Our solution requires only minimal modifications to application code. Moreover, we propose a configuration file-based approach that allows users to specify the desired data transfer pattern among workflow components, offering a general solution for different application contexts. We present a preliminary evaluation of the proposed framework on the K Computer (running on up to 4800 compute nodes) using RIKEN's experimental weather forecasting workflow as a case study.

关键词： asynchronous transfer big data processing customizability netCDF parallel direct data transfer real time

来源：评论

学校读者我要写书评

暂无评论

Research and Application of Spark Platform on big data processing in Intelligent Agriculture of Jilin Province

Research and Application of Spark Platform on Big Data Proce...

引用

The International Conference on Intelligent Agriculture 2017(ICIA2017)

作者： Siwei Fu Guifen Chen Shan Zhao Enze Xiao Jilin Agricultural University

Aiming at the demand of real-time massive data processing of Intelligent Agriculture in Jilin Province, this paper studies the big data processing of Intelligent Agriculture in Jilin Province based on Spark platform by acquiring real-time data through monitoring platform. This study first conducted the performance comparison experiment of Hadoop and Spark data processing platform, then used the Spark distributed cluster computing platform, real-time processing the big data of monitoring area. The experimental results show that the Spark platform speeds up 11.4 times faster than the Hadoop platform in the case of 100 million data sizes;and based on the Spark platform for real-time processing of big data intelligent agricultural monitoring network, not only provides memory calculations to reduce IO overhead, but also the results are faster and more accurate. The research results provide strong support for the implementation of precision agriculture technology in intelligent agriculture.

关键词： Spark big data processing MapReduce Intelligent Agriculture in Jilin Province

来源：评论

学校读者我要写书评

暂无评论

Workflow Transformation for Real-Time big data processing 5

Workflow Transformation for Real-Time Big Data Processing

引用

IEEE International Congress on big data (bigdata Congress)

作者： Ishizuka, Yuji Chen, Wuhui Paik, Incheon Univ Aizu Tsuruga Comp Sci & Engn Aizu Wakamatsu Fukushima 9658580 Japan

ISBN: (纸本)9781509026227

With the explosion of big data, processing and analyzing large numbers of continuous data streams in real-time, such as social media stream, sensor data streams, log streams, stock exchanges streams, etc., has become a crucial requirement for many scientific and industrial applications in recent years. Increased volume of streaming data as well as the demand for more complex real-time analytics require for execution of processing pipelines among heterogeneous event processing engines as a workflow. In this paper, we propose a workflow transformation for cost minimization in real-time big data processing on the heterogeneous systems. We first give the definition of stream-based workflow;and then we define eight different patterns as rules for workflow transformation;next, we give our workflow transformation algorithm based on our designed rules. Finally, our experiment shows that our proposed workflow transformation method can reduce the communication and computation cost effectively.

关键词： big data processing stream data workflow transformation geo-distributed data centers

来源：评论

学校读者我要写书评

暂无评论

An Architecture for big data processing on Intelligent Transportation Systems An application scenario on highway traffic flows 8

An Architecture for Big Data Processing on Intelligent Trans...

引用

8th IEEE International Conference on Intelligent Systems (IS)

作者： Guerreiro, Guilherme Figueiras, Paulo Silva, Ricardo Costa, Ruben Jardim-Goncalves, Ricardo Univ Nova Lisboa Fac Ciencias & Tecnol UNINOVA CTSDept Engn Electrotecn P-2829516 Caparica Portugal

ISBN: (纸本)9781509013531

\The transportation sector, and in particularly intelligent transportation systems, generate large volumes of realtime data that needs to be managed, communicated, interpreted, aggregated, and analyzed. To this end, innovative big data processing and mining as well as optimization techniques, need to be developed and applied in order to support real-time decision-making capabilities. Towards this end, this paper presents an ETL (extract, transform and load) architecture for intelligent transportation systems, addressing an application scenario on dynamic toll charging for highways. The ETL approach presented here, is responsible for preparing the data to be used by traffic prediction services, which will dynamically affect toll prices within different contexts. The proposed architecture relies on the adoption of "big data" technologies, to process and store large volumes of data from heterogeneous sources, provided by different highway operators. The proposed architecture is capable of handling real-time and historical data using big data technologies such as Spark on Hadoop and MongoDB. The DATEX-II data model is adopted, in order to harmonize traffic data provided by the highway operators. The work presented here, is still part of ongoing work currently addressed under the EU H2020 OPTIMUM project. Preliminary results achieved so far do not address the final conclusions of the project, but enabled us to demonstrate considerable gains in performance, when compared to other traditional ETL approaches, and also form the basis for pointing out and discuss future work directions and opportunities in the area of the development of big data processing and mining methods under the ITS domain.

关键词： ITS Tolling Systems big data processing data Mining ETL

来源：评论

学校读者我要写书评

暂无评论

An exploratory study on big data processing: a case study from a biomedical informatics 3

An exploratory study on Big data processing: a case study fr...

引用

3rd MEC International Conference on big data and Smart City (ICBDSC)

作者： Nair, Smitha Sunil Kumaran Ganesh, Nandita Middle East Coll Dept Comp Knowledge Oasis Muscat Oman Middle East Coll Qual Assurance Off Knowledge Oasis Muscat Oman

ISBN: (纸本)9781467395847

Recent studies confirm that big data processing will be a leading trend in the next several years in the biomedical field. Technical advancements in the field of medicine and life science demand for high storage and processing power. This paper presents an exploratory study on big data processing from a biomedical informatics perspective. In an attempt to perform this task, two case studies are considered from medical imaging and bioinformatics specifically proteomics aspect. It has been identified that there is scope for the growing medical imaging data and proteomic data be efficiently stored and processed through the use of big data technology. This study will also benefit the researchers working on big data tools in conjunction with data mining tools.

关键词： big data big data processing Bioinformatics Medical imaging Neurodegenerative diseases

来源：评论

学校读者我要写书评

暂无评论

Multiple big data processing Platforms

Multiple Big Data Processing Platforms

引用

Conference on Technologies and Applications of Artificial Intelligence (TAAI)

作者： Chang, Bao Rong Tsai, Hsiu-Fen Chang, Yi-Sheng Huang, Chien-Feng Natl Univ Kaohsiung Dept Comp Sci & Informat Engn 700 Kaohsiung Univ Rd Kaohsiung 811 Taiwan Shu Te Univ Dept Mkt Management 59 Hun Shang Rd Kaohsiung 824 Taiwan

ISBN: (纸本)9781509057320

The integration of Hive, Impala and Spark SQL platforms has achieved to perform rapid data retrieval using SQL query in big data environment. This paper is to design the optimized platform selection for highly improving the response of data retrieval. It can automatically choose the best-perform platform to best perform SQL commands. In addition, the distributed memory storage systems using Memcached and the distributed file system Hadoop HDFS have implemented the caching so that the fastest data retrieval has done once the repeated SQL command has applied.

关键词： big data processing Multi-Platform data processing System Optimized Platform Selection Distributed Memory Cache and Distributed File System Cache

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：