检索结果-内蒙古大学图书馆

Joint Workload Distribution and Capacity Augmentation in Hybrid Datacenter Networks

IEEE-ACM TRANSACTIONS ON NETWORKING 2021年第1期29卷 120-133页

作者： Ao, Weng Chon Huang, Po-Han Psounis, Konstantinos Univ Southern Calif Ming Hsieh Dept Elect & Comp Engn Los Angeles CA 90089 USA

In hybrid datacenter networks, wired connections are augmented with wireless links to facilitate data transfers between racks. The usage of mmWave/FSO wireless links enables dynamic bandwidth/capacity allocation with extremely small reconfiguration delay. Also, on-demand workload distribution, where the workload of a job is divided into multiple tasks that can be distributed/routed to different racks to be processed in parallel, allows better utilization of computational resources in data centers. In prior work, the dynamic wireless capacity augmentation and workload distribution decisions were mostly made independently and in a heuristic manner for serving distributed and parallel computing jobs. In this paper, we propose a novel analytical framework and algorithms to jointly optimize both the wireless capacity augmentation and the workload distribution, to minimize the job completion time. We consider workload that is not amenable to pipelining, fully amenable to pipelining, and partially amenable to pipelining. With extensive simulation studies, we show that the gain (in terms of the reduction in the job completion time) can be very substantial when allowing such joint optimization.

关键词： Wireless communication Task analysis Pipeline processing Optimization Resource management Data centers Servers On-demand workload distribution wireless capacity augmentation biconvex optimization distributed and parallel computing hybrid datacenter networks

来源：评论

学校读者我要写书评

暂无评论

An Asymptotic Ensemble Learning Framework for Big Data Analysis

引用

IEEE ACCESS 2019年 7卷 3675-3693页

作者： Salloum, Salman Huang, Joshua Zhexue He, Yulin Chen, Xiaojun Shenzhen Univ Natl Engn Lab Big Data Syst Comp Technol Shenzhen 518060 Peoples R China Shenzhen Univ Coll Comp Sci & Software Engn Big Data Inst Shenzhen 518060 Peoples R China

In order to enable big data analysis when data volume goes beyond the available computing resources, we propose a new method for big data analysis. This method uses only a few random sample data blocks of a big data set to obtain approximate results for the entire data set. The random sample partition (RSP) distributed data model is used to represent a big data set as a set of non-overlapping random sample data blocks. Each block is saved as an RSP data block file that can be used directly to estimate the statistical properties of the entire data set. A subset of RSP data blocks is randomly selected and analyzed with existing sequential algorithms in parallel. Then, the results from these blocks are combined to obtain ensemble estimates and models which can be improved gradually by appending new results from the newly analyzed RSP data blocks. To this end, we propose a distributed data-parallel framework (Alpha framework) and develop a prototype of this framework using Microsoft R Server packages and Hadoop distributed file system. The experimental results of three real data sets show that a subset of RSP data blocks of a data set is sufficient to obtain estimates and models which are equivalent to those computed from the entire data set.

关键词： Big data analysis cluster computing random sample partition block-level sampling distributed and parallel computing approximate computing random sampling ensemble methods

来源：评论

学校读者我要写书评

暂无评论

Online Balancing Two Independent Criteria upon Placements and Deletions

引用

IEEE TRANSACTIONS ON parallel AND distributed SYSTEMS 2013年第8期24卷 1644-1650页

作者： Tse, Savio S. H. Istanbul Univ Dept Comp Engn TR-34320 Istanbul Turkey

We study the online bicriteria load balancing problem in this paper. We choose a system of distributed homogeneous file servers located in a cluster as the scenario and propose an online approximate solution for balancing their loads and required storage spaces upon placements and deletions. By placement (resp. deletion), we mean to insert a document into (resp. remove a document from) a server system. The main technique is to keep two global quantities large enough. To the best of our knowledge, the technique is novel, and the result is the first one, in the literature. Our result works for any sequences of document placements and deletions. For each deletion, a limited number of documents are reallocated. The load and storage space bounds are 1.5 to 4 times those in the best existing result for sole placements. We refer sole placements to those placement algorithms that do not allow any reallocation and replication. The time complexity, for each operation, is O(log M N), where M is the number of servers, and N is the number of existing documents in the servers, plus the reallocation cost for document deletion. The price for handling document deletion is almost totally reflected by the reallocation cost, and the higher bounds of load and storage spaces, while the O(log N) additive term in the time complexity serves as the remainder.

关键词： Approximate distributed online algorithm load balancing scheduling document placement deletion reallocation distributed and parallel computing

来源：评论

学校读者我要写书评

暂无评论

distributed and parallel processing for real-time and dynamic spatio-temporal graph

引用

WORLD WIDE WEB-INTERNET AND WEB INFORMATION SYSTEMS 2020年第2期23卷 905-926页

作者： Fang, Junhua Ding, Jiafeng Zhao, Pengpeng Xu, Jiajie Liu, An Li, Zhixu Soochow Univ Inst Artificial Intelligence Sch Comp Sci & Technol Suzhou 215006 Peoples R China

As a non-linear data structure consisting of nodes and edges, the graph data span many different domains. In the real world, applications based on such data structures are always time-sensitive, that is, the value of graph data tends to decrease with time. Furthermore, the application based on spatio-temporal graph is one of the typical representatives of time-sensitive, since the time dimension is an inherent feature of spatio-temporal data. The distributed Stream Processing Engine (DSPE) seems an excellent choice for the above requirement, which is commonly partitioned and concurrently processed by a number of threads to maximize the throughput. However, it is not feasible to do such mission directly using the traditional DSPE. In this paper, we propose a computational model suitable for handling the spatio-temporal graph in DSPE, by reconstructing the DSPE's parallel processing slots. Specifically, our proposal includes a general processing framework to deal with the data structure of the spatio-temporal graph, a state information compensation mechanism to ensure the correctness of processing such stateful operation in DSPE, a lightweight summary information calculation method to ensure the performance of the system. Empirical studies on real-world stream applications validate the usefulness of our proposals and prove the considerable advantage of our approaches over state-of-the-art solutions in the literature.

关键词： distributed and parallel computing Real-time processing Spatio-temporal graph

来源：评论

学校读者我要写书评

暂无评论

Lightweight distributed computing framework for orchestrating high performance computing and big data

引用

TURKISH JOURNAL OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCES 2022年第4期30卷 1571-1585页

作者： Ince, Muhammed Numan Gunay, Melih Ledet, Joseph Akdeniz Univ Dept Comp Engn Antalya Turkey

In recent years, the need for the ability to work remotely and subsequently the need for the availability of remote computer-based systems has increased substantially. This trend has seen a dramatic increase with the onset of the 2020 pandemic. Often local data is produced, stored, and processed in the cloud to remedy this flood of computation and storage needs. Historically, HPC (high performance computing) and the concept of big data have been utilized for the storage and processing of large data. However, both HPC and Hadoop can be utilized as solutions for analytical work, though the differences between these may not be obvious. Both use parallel processing techniques and offer options for data to be stored in either a centralized or distributed manner. Recent studies have focused on using a hybrid approach with both technologies. Therefore, the convergence between HPC and big data technologies can be filled with distributed computing machines at the layer described. This paper results from the motivation that there exists a necessity for a distributed computing framework that can scale from SOC (system on chip) boards to desktop computers and servers. For this purpose, in this article, we propose a distributed computing environment that can scale up to devices with heterogeneous architecture, where devices can set up clusters with resource-limited nodes and then run on top of. The solution can be thought of as a minimalist hybrid approach between HPC and big data. Within the scope of this study, not only the design of the proposed system is detailed, but also critical modules and subsystems are implemented as proof of concept.

关键词： distributed and parallel computing big data high performance computing distributed programming resource management

来源：评论

学校读者我要写书评

暂无评论

Delay-Optimal distributed Computation Offloading in Wireless Edge Networks

引用

IEEE-ACM TRANSACTIONS ON NETWORKING 2024年第4期32卷 3376-3391页

作者： Gong, Xiaowen Chen, Mingyu Cao, Yang Li, Dongsheng Auburn Univ Dept Elect & Comp Engn Auburn AL 36849 USA Huazhong Univ Sci & Technol Sch Elect & Informat Engn Wuhan 430074 Hubei Peoples R China

In this paper, we explore distributed edge computation offloading (DECO) that offloads computation to distributed edge devices connected wirelessly, which perform the offloaded computation in parallel. By integrating edge computing with parallel computing, DECO can substantially reduce the total computation delay. In particular, we study the fundamental problem of minimizing the total completion time of DECO. We show that the time-sharing based communication resource allocation always outperforms the bandwidth-sharing scheme, so that it suffices to focus on the time-sharing based communication scheduling. Based on the time-sharing scheme, we first establish some structural properties of the optimal communication scheduling policy. Then, given these properties, we develop an efficient algorithm that finds the optimal allocation of computation workloads. Next, based on the optimal computation allocation, we characterize the optimal scheduling order of communications, which exhibits an elegant structure: the optimal order is in the non-decreasing order of the ratio between a device's computation rate and its communication time. Last, based on the optimal computation allocation and communication scheduling, we show that the optimal device selection problem is a submodular minimization problem, so that it can be solved efficiently using some existing methods. We further extend the study to the setting where devices are subject to maximum computation workload constraints, and develop an efficient algorithm that finds the optimal computation workload allocation. Our results provide useful insights for the optimal computation-communication co-design for DECO. We evaluate the theoretical findings using extensive simulations in both practical settings and controlled settings, which demonstrate the performance of DECO in practice and also the efficiency of our proposed schemes and algorithms for DECO.

关键词： Edge computing distributed and parallel computing communication scheduling delay minimization

来源：评论

学校读者我要写书评

暂无评论

Large-Scale Subspace Clustering by Independent distributed and parallel Coding

引用

IEEE TRANSACTIONS ON CYBERNETICS 2022年第9期52卷 9090-9100页

作者： Li, Jun Tao, Zhiqiang Wu, Yue Zhong, Bineng Fu, Yun Northeastern Univ Dept Elect & Comp Engn Coll Engn Boston MA 02115 USA Santa Clara Univ Dept Comp Sci & Engn Santa Clara CA 95053 USA Guangxi Normal Univ Dept Comp Sci Guilin 541004 Peoples R China Northeastern Univ Khoury Coll Comp Sci Boston MA 02115 USA

Subspace clustering is a popular method to discover underlying low-dimensional structures of high-dimensional multimedia data (e.g., images, videos, and texts). In this article, we consider a large-scale subspace clustering ((LSC)-C-2) problem, that is, partitioning million data points with a millon dimensions. To address this, we explore an independent distributed and parallel framework by dividing big data/variable matrices and regularization by both columns and rows. Specifically, (LSC)-C-2 is independently decomposed into many subproblems by distributing those matrices into different machines by columns since the regularization of the code matrix is equal to a sum of that of its submatrices (e.g., square-of-Frobenius/l(1)-norm). Consensus optimization is designed to solve these subproblems in a parallel way for saving communication costs. Moreover, we provide theoretical guarantees that (LSC)-C-2 can recover consensus subspace representations of high-dimensional data points under broad conditions. Compared with the state-of-the-art (LSC)-C-2 methods, our approach achieves better clustering results in public datasets, including a million images and videos.

关键词： distributed and parallel computing least-squares regression (LSR) low-rank representation (LRR) over-high dimensional big data sparse subspace clustering (SSC) subspace clustering

来源：评论

学校读者我要写书评

暂无评论

Multiarmed-Bandit-Based Decentralized Computation Offloading in Fog-Enabled IoT

引用

IEEE INTERNET OF THINGS JOURNAL 2021年第12期8卷 10010-10017页

作者： Misra, Sudip Rachuri, Pramodh Deb, Pallav Kumar Mukherjee, Anandarup Indian Inst Technol Kharagpur Dept Comp Sci & Engn Kharagpur 721302 W Bengal India Indian Inst Technol Kharagpur Smart Wireless Applicat & Networking Lab Kharagpur 721302 W Bengal India Indian Inst Technol Bhilai Dept Elect Engn Bhilai 492015 India

The Internet-of-Things (IoT) environments have hard real-time tasks that need execution within fixed deadlines. As IoT devices consist of a myriad of sensors, each task is composed of multiple interdependent subtasks. Toward this, the cloud and fog computing platforms have the potential of facilitating these IoT sensor nodes (SNs) in accommodating complex operations with minimum delay. To further reduce operational latencies, we breakdown the high-level tasks into smaller subtasks and form a directed acyclic task graph (DATG). Initially, the SNs offload their tasks to a nearby fog node (FN) based on a greedy choice. The greedy formulation helps in selecting the FN in linear time while avoiding combinatorial optimizations at the SN, which saves time as well as energy. IoT environments are highly dynamic, which mandates the need for adaptive solutions. At the chosen FN, depending on the dependencies on the DATGs, its corresponding deadlines, and the varying conditions of the other FNs, we propose an E -greedy nonstationary multiarmed bandit-based scheme (D2CIT) for online task allocation among them. The online learning D2CIT scheme allows the FN to autonomously select a set of FNs for distributing the subtasks among themselves and executes the subtasks in parallel with minimum latency, energy, and resource usage. Simulation results show that D2CIT offers a reduction in latency by 17% compared to traditional fog computing schemes. Additionally, upon comparison with existing online learning-based task offloading solutions in fog environments, D2CIT offers an improved speedup of 59% due to the induced parallelism.

关键词： Computation offloading distributed and parallel computing fog computing Internet of Things (IoT) reinforcement learning (RL)

来源：评论

学校读者我要写书评

暂无评论

distributed and parallel construction method for equi-width histogram in cloud database

引用

MULTIAGENT AND GRID SYSTEMS 2017年第3期13卷 311-329页

作者： Wang, Yang Zhong, Yong Ma, Qingshan Yang, Guanci Chinese Acad Sci Chengdu Inst Comp Applicat Chengdu Sichuan Peoples R China Univ Chinese Acad Sci Beijing Peoples R China Guizhou Univ Minist Educ Key Lab Adv Mfg Technol Guiyang 550025 Guizhou Peoples R China

Data distribution summary has been commonly used in databases to support query optimization, and histograms are of particular interest. A significant issue in histogram estimation is the large amount of data transmission. This paper presents a distributed and parallel construction method for equi-width histogram in cloud database (called DPHCD). Unlike previous methods, the DPHCD does not require the transfer of any table detail during histogram construction. Only small information about buckets and a few necessary data need to be transmitted over the network. The data transmission of DPHCD is unrelated with table size. DPHCD divides the histogram task into small tasks that could be simultaneously executed in a distributed cluster. It uses an innovative tablet-level sampling method to reduce the computing overhead in each cluster node. DPHCD is implemented in the Xugu cloud database management system. Experimental results demonstrate that DPHCD can achieve small data transmission and speed up histogram construction.

关键词： Cloud database equi-width histogram distributed and parallel computing data transmission

来源：评论

学校读者我要写书评

暂无评论

A Survey of Data Partitioning and Sampling Methods to Support Big Data Analysis

引用

Big Data Mining and Analytics 2020年第2期3卷 85-101页

作者： Mohammad Sultan Mahmud Joshua Zhexue Huang Salman Salloum Tamer Z.Emara Kuanishbay Sadatdiynov National Engineering Laboratory for Big Data System Computing Technology Shenzhen UniversityUniversityShenzhen 518060China Big Data Institute College of Computer Science and Software EngineeringShenzhen UniversityShenzhen 518060China.

Computer clusters with the shared-nothing architecture are the major computing platforms for big data processing and *** cluster computing,data partitioning and sampling are two fundamental strategies to speed up the computation of big data and increase *** this paper,we present a comprehensive survey of the methods and techniques of data partitioning and sampling with respect to big data processing and *** start with an overview of the mainstream big data frameworks on Hadoop *** basic methods of data partitioning are then discussed including three classical horizontal partitioning schemes:range,hash,and random *** partitioning on Hadoop clusters is also discussed with a summary of new strategies for big data partitioning,including the new Random Sample Partition(RSP)distributed *** classical methods of data sampling are then investigated,including simple random sampling,stratified sampling,and reservoir *** common methods of big data sampling on computing clusters are also discussed:record-level sampling and blocklevel ***-level sampling is not as efficient as block-level sampling on big distributed *** the other hand,block-level sampling on data blocks generated with the classical data partitioning methods does not necessarily produce good representative samples for approximate computing of big *** this survey,we also summarize the prevailing strategies and related work on sampling-based approximation on Hadoop *** believe that data partitioning and sampling should be considered together to build approximate cluster computing frameworks that are reliable in both the computational and statistical respects.

关键词： big data analysis data partitioning data sampling distributed and parallel computing approximate computing

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：