检索结果-内蒙古大学图书馆

A Modeling Language for MapReduce Programing in a Storage System Perspective

JOURNAL OF SIGNAL processing SYSTEMS FOR SIGNAL IMAGE AND VIDEO TECHNOLOGY 2018年第8-9期90卷 1133-1150页

作者： Jing, Yuxin Wang, Hanpin Huang, Yu Zhang, Lei Cao, Yongzhi Peking Univ Sch Elect Engn & Comp Sci Inst Software Beijing 100871 Peoples R China Peking Univ Natl Engn Res Ctr Software Engn Beijing 100871 Peoples R China

MapReduce is a powerful distributed data analysis programming model. It runs on big data storage systems and processes data in a parallel way. An appropriate way to ensure the correctness of MapReduce programs is formal method analysis, which requires firstly a formal model of MapReduce. In this paper we propose a modeling language to establish the formal model of the MapReduce framework. Unlike other approaches, our language describes the processing of data in the MapReduce programs from a perspective of underlying files and blocks, so that the details of data processing can be clearly demonstrated. The language is based on our previous work, a language describing the management of massive data storage systems, with extensions from two aspects: block content data refinement and concurrency support. Based on our language, the features of the MapReduce programming model can be discussed.

关键词： big data processing Modeling langugage MapReduce Concurrency Formal methods

来源：评论

学校读者我要写书评

暂无评论

Measurement of Medium-Duty Plug-In Hybrid Electric Vehicle Fuel Economy Sensitivity to Ambient Temperature

引用

IEEE TRANSACTIONS ON TRANSPORTATION ELECTRIFICATION 2018年第1期4卷 184-189页

作者： Vore, Spencer E. Kosowski, Mark Reid, Michael L. Wilkins, Zachary Minicucci, Joseph Bradley, Thomas H. Colorado State Univ Dept Mech Engn Ft Collins CO 80523 USA Elect Power Res Inst Elect Transportat Dept Palo Alto CA 94304 USA Colorado State Univ Dept Comp Sci Ft Collins CO 80523 USA

The correlation between ambient temperature and equivalent real-world fuel economy (FE) for plug-in hybrid electric vehicles (PHEVs) is the subject of an investigation seeking to inform economic, environmental, and policy assessments of these vehicles and technologies. This paper collected seven months of 1-Hz vehicle operating data from 119 Odyne medium-duty PHEV work trucks in commercial service across the United States. The data used for this paper were collected from the vehicle's controller area networks using mobile data collection and transmission devices. The raw data were aggregated into a central database for analysis. We present the methods of data processing, filtering, intermediate data manipulation, and analysis implemented using MATLAB, R, SQL, and Java-based Hadoop MapReduce. Regression analysis is used to calculate models describing the sensitivity of PHEV FE to ambient temperature, and the models are found to be similar to regressions for conventional vehicles (CVs) from other studies. This paper provides evidence that the sensitivity of medium-duty PHEV FE is similar to that of CVs. These results can provide guidance to researchers seeking to model real-world PHEV energy consumption.

关键词： big data processing controller area network (CAN)-communication system data medium-duty trucks mild hybrid plug-in hybrid electric vehicles (PHEVs)

来源：评论

学校读者我要写书评

暂无评论

Preemptive cloud resource allocation modeling of processing jobs

引用

JOURNAL OF SUPERCOMPUTING 2018年第5期74卷 2116-2150页

作者： Vakilinia, Shahin Cheriet, Mohamed Ecole Technol Super Dept Comp Engn Montreal PQ Canada

Cloud computing allows execution and deployment of different types of applications such as interactive databases or web-based services which require distinctive types of resources. These applications lease cloud resources for a considerably long period and usually occupy various resources to maintain a high quality of service (QoS) factor. On the other hand, general big data batch processing workloads are less QoS-sensitive and require massively parallel cloud resources for short period. Despite the elasticity feature of cloud computing, fine-scale characteristics of cloud-based applications may cause temporal low resource utilization in the cloud computing systems, while process-intensive highly utilized workload suffers from performance issues. Therefore, ability of utilization efficient scheduling of heterogeneous workload is one challenging issue for cloud owners. In this paper, addressing the heterogeneity issue impact on low utilization of cloud computing system, conjunct resource allocation scheme of cloud applications and processing jobs is presented to enhance the cloud utilization. The main idea behind this paper is to apply processing jobs and cloud applications jointly in a preemptive way. However, utilization efficient resource allocation requires exact modeling of workloads. So, first, a novel methodology to model the processing jobs and other cloud applications is proposed. Such jobs are modeled as a collection of parallel and sequential tasks in a Markovian process. This enables us to analyze and calculate the efficient resources required to serve the tasks. The next step makes use of the proposed model to develop a preemptive scheduling algorithm for the processing jobs in order to improve resource utilization and its associated costs in the cloud computing system. Accordingly, a preemption-based resource allocation architecture is proposed to effectively and efficiently utilize the idle reserved resources for the processing jobs in the cloud paradi

关键词： big data processing Markov chain Cloud computing Queuing theory

来源：评论

学校读者我要写书评

暂无评论

Multi-Regional Online Car-Hailing Order Quantity Forecasting Based on the Convolutional Neural Network

引用

INFORMATION 2019年第6期10卷 193-193页

作者： Huang, Zihao Huang, Gang Chen, Zhijun Wu, Chaozhong Ma, Xiaofeng Wang, Haobo Wuhan Univ Technol Sch Energy & Power Engn Wuhan 430063 Hubei Peoples R China Wuhan Univ Technol Intelligent Transportat Syst Res Ctr Wuhan 430063 Hubei Peoples R China Minist Publ Secur Traff Management Res Inst Wuhan 430063 Hubei Peoples R China

With the development of online cars, the demand for travel prediction is increasing in order to reduce the information asymmetry between passengers and drivers of online car-hailing. This paper proposes a travel demand forecasting model named OC-CNN based on the convolutional neural network to forecast the travel demand. In order to make full use of the spatial characteristics of the travel demand distribution, this paper meshes the prediction area and creates a travel demand data set of the graphical structure to preserve its spatial properties. Taking advantage of the convolutional neural network in image feature extraction, the historical demand data of the first twenty-five minutes of the entire region are used as a model input to predict the travel demand for the next five minutes. In order to verify the performance of the proposed method, one-month data from online car-hailing of the Chengdu Fourth Ring Road are used. The results show that the model successfully extracts the spatiotemporal features of the data, and the prediction accuracies of the proposed method are superior to those of the representative methods, including the Bayesian Ridge Model, Linear Regression, Support Vector Regression, and Long Short-Term Memory networks.

关键词： mobility as a service (MaaS) traffic demand forecasting convolutional neural networks big data processing

来源：评论

学校读者我要写书评

暂无评论

From Network Perspective to Optimize Shuffling Phase in MapReduce 6

From Network Perspective to Optimize Shuffling Phase in MapR...

引用

6th International Conference on Advanced Cloud and bigdata (CBD)

作者： Wang, Haifeng Cao, Yunpeng Linyi Univ Coll Informat Sci & Engn Linyi Shandong Peoples R China Shandong Prov Key Lab Network Based Intelligent C Linda Inst Linyi Shandong Peoples R China

ISBN: (纸本)9781538680346

For MapReduce jobs in data center, network traffic is generated in shuffling phase, causing east-west communication bottleneck. Aiming at this problem, an optimization scheme is proposed to aggregate relevant network traffic flows into local areas. Firstly, the characteristics of pre-scheduling of MapReduce jobs are extracted, the communication activity degree of jobs is defined and the computing jobs are divided into two types: active or inactive communication job. Then the Bayesian classification with active learning is used as the prediction model, and this model after training by sample data can determine job type. The active communication jobs are deployed in the same rack to improve network bandwidth utilization. The experiment results of small-scale data center show that the proposed communication optimization scheme has a significant effect on shuffling intensive jobs, reaching 4.2%-5.6%. In the case of larger amount of input data, this scheme has better robustness and can effectively reduce east-west communication delay in data center.

关键词： data Center MapReduce big data processing Communication Optimization Shuffle Communication

来源：评论

学校读者我要写书评

暂无评论

FlowTime: Dynamic Scheduling of Deadline-Aware Workflows and Ad-hoc Jobs 38

FlowTime: Dynamic Scheduling of Deadline-Aware Workflows and...

引用

38th IEEE International Conference on Distributed Computing Systems (ICDCS)

作者： Hu, Zhiming Li, Baochun Chen, Chen Ke, Xiaodi Univ Toronto Dept Elect & Comp Engn Toronto ON Canada Huawei Canada Res Ctr Toronto ON Canada

ISBN: (纸本)9781538668719

With rapidly increasing volumes of data to be processed in modern data analytics, it is commonplace to run multiple data processing jobs with inter-job dependencies in a datacenter cluster, typically as recurring data processing workloads. Such a group of inter-dependent data analytic jobs is referred to as a workflow, and may have a deadline due to its mission-critical nature. In contrast, non-recurring ad-hoc jobs are typically best-effort in nature, and rather than meeting deadlines, it is desirable to minimize their average job turnaround time. The state-of-the-art scheduling mechanisms focused on meeting deadlines for individual jobs only, and are oblivious to workflow deadlines. In this paper, we present FlowTime, a new system framework designed to make scheduling decisions for workflows so that their deadlines are met, while simultaneously optimizing the performance of ad-hoc jobs. To achieve this objective, we first adopt a divide-and-conquer strategy to transform the problem of workflow scheduling to a deadline-aware job scheduling problem, and then design an efficient algorithm that tackles the scheduling problem with both deadline-aware jobs and ad-hoc jobs by solving its corresponding optimization problem directly using a linear program solver. Our experimental results have clearly demonstrated that FlowTime achieves the lowest deadline-miss rates for deadline-aware workflows and 210 times shorter average job turnaround time, as compared to the state-of-the-art scheduling algorithms.

关键词： workflow scheduling big data processing deadline-aware scheduling

来源：评论

学校读者我要写书评

暂无评论

LADD: A Length-Adaptive Approach to Detecting Taxi Anomalous Detours

LADD: A Length-Adaptive Approach to Detecting Taxi Anomalous...

引用

International Conference on Parallel and Distributed Systems (ICPADS)

作者： Xuansheng Lu Weiyu Cheng Yanyan Shen Yanmin Zhu Department of Computer Science and Engineering Shanghai Jiao Tong University

ISBN: (纸本)9781728125848

This paper focuses on the practical problem of detecting taxi anomalous detours. In this paper, we propose a length-adaptive detection approach called LADD to detect if a taxi conducted an anomalous detour by investigating the taxi journey's trajectory. We employ a similarity-based detection strategy, which takes trajectory length into consideration. Our approach can deal with both long and short trajectories by applying weighted similarity on trajectories and using trajectory concatenation or clipping. For quantitative performance comparison, we propose a performance metric, and compare our approach with the state-of-the-art methods on a real-world dataset. The results verify the superior performance of LADD on taxi detour detection.

关键词： big data processing Spatial databases Detour detecting GPS trajectories Intelligent transportation systems visual databases Intelligent transportation systems Taxi Lacrimoauriculodentodigital syndrome Public Transportation Detours trajectory Performance metrics Anomalous

来源：评论

学校读者我要写书评

暂无评论

A Mobile Application for Dog Breed Detection and Recognition based on Deep Learning 5

A Mobile Application for Dog Breed Detection and Recognition...

引用

11th IEEE/ACM International Conference on Utility and Cloud Computing (UCC-Companion) / 5th IEEE/ACM International Conference on big data Computing, Applications and Technologies (BDCAT)

作者： Wu, Fang Chen, Wenbin Sinnott, Richard O. Univ Melbourne Sch Comp & Informat Syst Melbourne Vic 3010 Australia

ISBN: (纸本)9781538655023

Deep learning provides the ability to train algorithms (models) that can tackle the problems of data classification and prediction based on deriving (learning) knowledge from raw data. Convolutional Neural Networks (CNNs) provides one commonly used approach for image classification and detection. In this work we describe a CNN-based method for detecting dogs in potentially complex images and subsequently consider the identification of the type/ breed of dogs. The results achieve nearly 85% accuracy for breed classification for a set of 50 classes of dogs and 64% accuracy for 120 other less common dog types. An iOS application and associated big data processing infrastructure utilizing a variety of GPUs was used to support the image classification algorithms.

关键词： Dog Breed Recognition Object Detection big data processing GPU Computing

来源：评论

学校读者我要写书评

暂无评论

CODE-DE - THE GERMAN OPERATIONAL ENVIRONMENT FOR ACCESSING AND processing COPERNICUS SENTINEL PRODUCTS 38

CODE-DE - THE GERMAN OPERATIONAL ENVIRONMENT FOR ACCESSING A...

引用

38th IEEE International Geoscience and Remote Sensing Symposium (IGARSS)

作者： Storch, Tobias Reck, Christoph Holzwarth, Stefanie Keuck, Vanessa DLR EOC Munchner Str 20 D-82234 Wessling Germany DLR Space Adm Konigswinterer Str 522-524 D-53227 Bonn Germany

ISBN: (纸本)9781538671504

We present and analyze the methodology, architecture, and various functionalities of CODE-DE (Copernicus data and Exploitation Platform - Deutschland, ***) which is the German operational environment for accessing and processing Copernicus Sentinel products, a so-called Copernicus collaborative ground segment activity. Since 9 March 2017 the element for online data access to Sentinel-1 and Sentinel-2 products is operational and tapped by over 650 registered users until 30 November 2017. During this period more than 47,000 products were downloaded and the global catalogue was continuously updated and includes a data volume of 800 TByte based on a rolling archive concept. Since 30 November 2017 also the element for big data processing is operational, where registered users automatically process and analyze data themselves specifically with the help of various methodologies to value-added products. Furthermore, optimizations are realized such as improved visualization of the catalogue of available Sentinel-2 products. They are presented with the catalogue client at full 10 meter resolution.

关键词： Copernicus Sentinel CODE-DE Online data Accessing big data processing

来源：评论

学校读者我要写书评

暂无评论

LDBAS: Location-aware data Block Allocation Strategy for HDFS-based Applications in the Cloud

引用

KSII TRANSACTIONS ON INTERNET AND INFORMATION SYSTEMS 2018年第1期12卷 204-226页

作者： Xu, Hua Liu, Weiqing Shu, Guansheng Li, Jing Univ Sci & Technol China Dept Comp Sci & Technol Hefei 230026 Anhui Peoples R China

big data processing applications have been migrated into cloud gradually, due to the advantages of cloud computing. Hadoop Distributed File System (HDFS) is one of the fundamental support systems for big data processing on MapReduce-like frameworks, such as Hadoop and Spark. Since HDFS is not aware of the co-location of virtual machines in the cloud, the default scheme of block allocation in HDFS does not fit well in the cloud environments behaving in two aspects: data reliability loss and performance degradation. In this paper, we present a novel location-aware data block allocation strategy (LDBAS). LDBAS jointly optimizes data reliability and performance for upper-layer applications by allocating data blocks according to the locations and different processing capacities of virtual nodes in the cloud. We apply LDBAS to two stages of data allocation of HDFS in the cloud (the initial data allocation and data recovery), and design the corresponding algorithms. Finally, we implement LDBAS into an actual Hadoop cluster and evaluate the performance with the benchmark suite bigdataBench. The experimental results show that LDBAS can guarantee the designed data reliability while reducing the job execution time of the I/O-intensive applications in Hadoop by 8.9% on average and up to 11.2% compared with the original Hadoop in the cloud.

关键词： Hadoop HDFS big data processing data block allocation cloud

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：