检索结果-内蒙古大学图书馆

Sparsity exploiting erasure coding for distributed storage of versioned data

COMPUTING 2016年第12期98卷 1305-1329页

作者： Harshan, J. Oggier, Frederique Datta, Anwitaman Nanyang Technol Univ Sch Phys & Math Sci Div Math Sci Singapore Singapore Nanyang Technol Univ Sch Comp Engn Singapore Singapore

In this paper we study the problem of storing reliably an archive of versioned data. Specifically, we focus on systems where the differences (deltas) between subsequent versions rather than the whole objects are stored-a typical model for storing versioned data. For reliability, we propose erasure encoding techniques that exploit the sparsity of information in the deltas while storing them reliably in a distributed back-end storage system, resulting in improved I/O read performance to retrieve the whole versioned archive. Along with the basic techniques, we propose a few optimization heuristics, and evaluate the techniques' efficacy analytically and with numerical simulations.

关键词： Distributed storage systems Version management Fault tolerance erasure coding

来源：评论

学校读者我要写书评

暂无评论

erasure coding BASED CAUSALLY CONSISTENT SHARED MEMORY EMULATION ALGORITHM

ERASURE CODING BASED CAUSALLY CONSISTENT SHARED MEMORY EMULA...

引用

作者： Lyu, Shihang PennState University Libraries

学位级别：Master of Science

This thesis develops CausalEC_exp, a causal read-write shared memory emulation algorithm for a distributed data store that uses erasure coding. In a message passing system with servers and clients, where every client is associated to a particular server to which it issues read and write operations, this thesis shows that every execution of CausalEC_exp is causally ***_exp retains a key benefit of previously designed replication-based algorithms - every write operation is "local", that is, a server performs only local actions before returning to the client that issued a write operation. Furthermore, for servers that store object values in uncoded form, CausalEC_exp also ensures that read operations issued to those objects are *** thesis shows that every operation issued to CausalEC_exp algorithm eventually terminates. In CausalEC_exp, some servers store object values corre- sponding to multiple versions temporarily, but we show that in an execution where, after a point where no new write operations are issued, the storage cost of every node eventually is equal to that of the underlying erasure coding *** thesis also discusses potential benefits for using this causal shared memory algorithm that uses erasure coding, and points out directions for future work.

关键词： erasure coding causal consistency shared memory

来源：评论

学校读者我要写书评

暂无评论

Brief Announcement: Racos: A Leaderless erasure coding State Machine Replication 24

Brief Announcement: Racos: A Leaderless Erasure Coding State...

引用

36th ACM Symposium on Parallelism in Algorithms and Architectures (SPAA)

作者： Zarnstorff, Jonathan Lebow, Lucas Remuck, Dillon Ruiz, Colin Tseng, Lewis Clark Univ Worcester MA 01610 USA

ISBN: (纸本)9798400704161

Cloud storage systems often use state machine replication (SMR) to ensure reliability and availability. erasure coding has recently been integrated with SMR to reduce disk and network I/O costs. This brief announcement shares our experience in developing a leaderless erasure coding SMR system. We integrate our system Racos with etcd, a distributed key-value storage that powers Kubernetes. Racos outperforms competitors by up to 3.36x in throughput.

关键词： State machine replication erasure coding Leaderless Rabia

来源：评论

学校读者我要写书评

暂无评论

HSPP: Load-Balanced and Low-Latency File Partition and Placement Strategy on Distributed Heterogeneous Storage with erasure coding 19th

HSPP: Load-Balanced and Low-Latency File Partition and Place...

引用

19th International Conference on Algorithms and Architectures for Parallel Processing (ICA3PP)

作者： Sun, Jiazhao Li, Yunchun Yang, Hailong Beihang Univ Sch Comp Sci & Engn Sino German Joint Software Inst Beijing 100191 Peoples R China

ISBN: (纸本)9783030389611;9783030389604

To speedup the accesses to massive amount of data, heterogeneous architecture has been widely adopted in the mainstream storage system. In such systems, load imbalance and scheduler overhead are the primary factors that slow down the I/O performance. In this paper, we propose an effective file scheduling strategy HSPP that includes statistic based file classification, partition with erasure coding and adaptive data placement to optimize load balance and read latency on the distributed heterogeneous storage system. The experiment results show that HSPP is superior than existing strategies in terms of load balance, read latency, and scheduling overhead.

关键词： Data partition and placement erasure coding Heterogeneous storage

来源：评论

学校读者我要写书评

暂无评论

TriEC: Tripartite Graph Based erasure coding NIC Offload 19

TriEC: Tripartite Graph Based Erasure Coding NIC Offload

引用

International Conference for High Performance Computing, Networking, Storage and Analysis (SC)

作者： Shi, Haiyang Lu, Xiaoyi Ohio State Univ Dept Comp Sci & Engn Columbus OH 43210 USA

ISBN: (数字)9781450362290

ISBN: (纸本)9781450362290

erasure coding (EC) NIC offload is a promising technology for designing next-generation distributed storage systems. However, this paper has identified three major limitations of current-generation EC NIC offload schemes on modern SmartNICs. Thus, this paper proposes a new EC NIC offload paradigm based on the tripartite graph model, namely TriEC. TriEC supports both encode-and-send and receive-and-decode operations efficiently. Through theorem-based proofs, co-designs with memcached (i.e., TriEC-Cache), and extensive experiments, we show that TriEC is correct and can deliver better performance than the state-of-the-art EC MC offload schemes (i.e., BiEC). Benchmark evaluations demonstrate that TriEC outperforms BiEC by up to 1.82x and 2.33x for encoding and recovering, respectively. With extended YCSB workloads, TriEC reduces the average write latency by up to 23.2% and the recovery time by up to 37.8%. TriEC outperforms BiEC by 1.32x for a full-node recovery with 8 million records.

关键词： erasure coding Bipartite Tripartite NIC Offload

来源：评论

学校读者我要写书评

暂无评论

EC-Shuffle: Dynamic erasure coding Optimization for Efficient and Reliable Shuffle in Spark 19

EC-Shuffle: Dynamic Erasure Coding Optimization for Efficien...

引用

19th Annual IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing (CCGRID)

作者： Yao, Xin Wang, Cho-Li Zhang, Mingzhe Univ Hong Kong Dept Comp Sci Hong Kong Peoples R China

ISBN: (纸本)9781728109121

Fault-tolerance capabilities attract increasing attention from existing data processing frameworks, such as Apache Spark. To avoid replaying costly distributed computation, like shuffle, local checkpoint and remote replication arc two popular approaches. They incur significant runtime overhead, such as extra storage cost or network traffic. erasure coding is another emerging technology, which also enables data resilience. It is perceived as capable of replacing the checkpoint and replication mechanisms for its high storage efficiency. However, it suffers heavy network traffic due to distributing data partitions to different locations. In this paper, we propose EC-Shuffle with two encoding schemes and optimize the shuffle-based operations in Spark or MapReduce-like frameworks. Specifically, our encoding schemes concentrate on optimizing the data traffic during the execution of shuffle operations. They only transfer the parity chunks generated via erasure coding, instead of a whole copy of all data chunks. EC-Shuffle also provides a strategy, which can dynamically select the per shuffle biased encoding scheme according to the number of senders and receivers in each shuffle. Our analyses indicate that this dynamic encoding selection can minimize the total size of parity chunks. The extensive experimental results using BigDataBench with hundreds of mappers and reducers shows this optimization can reduce up to 50% network traffic and achieve up to 38% performance improvement.

关键词： fault tolerance distributed computation erasure coding

来源：评论

学校读者我要写书评

暂无评论

Sabotage-proof Messenger Discovery for Protesters based on Brownian motion and erasure coding 19

Sabotage-proof Messenger Discovery for Protesters based on B...

引用

19th International Conference on Wireless and Mobile Computing, Networking and Communications (WiMob)

作者： Liang, Chunchao Cheng, Wen Dotter, Todd Parkhurst, Zachary Marywood Univ Dept Sci Scranton PA 18509 USA Southern Arkansas Univ Dept Math & Comp Sci Magnolia AR USA

ISBN: (纸本)9798350336672

Sabotage-proof message dissemination is critical for a successful protest. To successfully propagate the messages to the crowd, it is critical to develop a sabotage-proof protocol to discover the messengers for information relay. The protocol should be simple, efficient, and sabotage-proof. In this paper, we review the previous protocols and propose a new protocol that exploits the Brownian-motion of the protesters. The proposed distributed protocol is simple, but it is robust against sabotage. In every message propagation, the sender encodes the message with erasure coding into multiple packets, then the sender broadcasts one packet after another while the protesters are in Brownian-motion. The receivers that do not receive all packets but receive enough packets to decode the original message will be the potential messengers. Mathematical analysis and simulation verified the viability of the proposed protocol.

关键词： sabotage-proof messenger discovery Brownian motion erasure coding protest cellphone

来源：评论

学校读者我要写书评

暂无评论

EFLOG: A Full Stream-Logging Scheme with erasure coding in Cloud Storage Systems 15

EFLOG: A Full Stream-Logging Scheme with Erasure Coding in C...

引用

15th IEEE International Conference on Networking, Architecture and Storage (NAS)

作者： Sun, Lei Cao, Qiang Wang, Shucheng Xie, Changsheng Huazhong Univ Sci & Technol Wuhan Natl Lab Optoelect Wuhan Peoples R China

ISBN: (纸本)9781728177441

Large-scale cloud storage systems use the logging mechanism to sequentially write data in an append-only manner. The write stream needs to be first appended and persisted into logging files, and then encoded with erasure coding (EC) in underlying storage. This introduces significant overhead to small write operations. To solve this problem, we propose EFLOG, a full-streaming storage framework that combines Logging and inter-log EC mechanisms. EFLOG evenly schedules front-end write streams across log files in each disk with append-only manner. In background, EFLOG determines unprotected logged data and seals them into ECblocks. Afterwards, EFLOG concurrently encodes data with multi-threads and stores parity data into parity disks. Results of our trace-driven evaluation show that, EFLOG can achieve up to 1.01GB/s write throughput with RS(4, 2) codes built upon 6 SSD disks.

关键词： Large-scale cloud storage systems erasure coding logging

来源：评论

学校读者我要写书评

暂无评论

A Case for Common-Case: On FPGA Acceleration of erasure coding 25

A Case for Common-Case: On FPGA Acceleration of Erasure Codi...

引用

25th IEEE Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM)

作者： Nakhjavani, Reza Zhu, Jianwen Univ Toronto Dept Elect & Comp Engn Toronto ON Canada

ISBN: (纸本)9781538640371

Reliable storage is central component of data centers that support private or public cloud. erasure coding has becoming increasingly popular alternative to replication for its capability in substantially cutting disk cost while delivering the same reliability. This paper reports the comprehensive results of using FPGA for accelerating erasure encoding and decoding algorithms. In particular, to accomplish the best efficiency in throughput delivered per thousand LUTs, we argue it is best to allocate more resources to the common-case, which we show can be more than 90%, while reducing performance target for the general-case. With further innovations, we show, as an example, that for a RS(10,4) erasure code, and a 1.3% disk failure probability, a 6Gb/s/KLUT can be accomplished for 5 nines of reliability. In terms of power efficiency, our design is able to achieve 40Gb/s/Watt.

关键词： Common-case erasure coding FPGA Reliability

来源：评论

学校读者我要写书评

暂无评论

MICS : Mingling Chained Storage Combining Replication and erasure coding 34

MICS : Mingling Chained Storage Combining Replication and Er...

引用

SRDS 2015 34th Symposium on Reliable Distributed Systems

作者： Tang, Yan Yin, Jianwei Lo, Wei Li, Ying Deng, Shuiguang Dong, Kexiong Pu, Calton Zhejiang Univ Hangzhou Zhejiang Peoples R China LongTang Informat Technol Ltd Suzhou Peoples R China Georgia Inst Technol Atlanta GA 30332 USA

ISBN: (纸本)9781467393027

High reliability, low space cost, and efficient read/write performance are all desirable properties for cloud storage systems. Due to the inherent conflicts, however, simultaneously achieving optimality on these properties is unrealistic. Since reliable storage is indispensable prerequisite for services with high availability, tradeoff should therefore be made between space and read/write efficiency when storage scheme is designed. N-way Replication and erasure coding, two extensively-used storage schemes with high reliability, adopt opposite strategies on this tradeoff issue. However, unbalanced tradeoff designs of both schemes confine their effectiveness to limited types of workloads and system requirements. To mitigate such applicability penalty, we propose MICS, a MIngling Chained Storage scheme that combines structural and functional advantages from both Nway replication and erasure coding. Qualitatively, MICS provides efficient read/write performance and high reliability at reasonably low space cost. MICS stores each object in two forms: a full copy and certain amount of erasure-coded segments. We establish dedicated read/write protocols for MICS leveraging the unique structural advantages. Moreover, MICS provides high read/write efficiency with Pipeline Random-Access Memory consistency to guarantee reasonable semantics for services users. Evaluation results demonstrate that under same fault tolerance and consistency level, MICS outperforms N-way replication and pure erasure coding in I/O throughput by up to 34.1% and 51.3% respectively. Furthermore, MICS shows superior performance stability over diverse workload conditions, in which case the standard deviation of MICS is 70.1% and 29.3% smaller than those of other two schemes.

关键词： MICS Cloud Storage erasure coding Replication Tradeoff

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：