检索结果-内蒙古大学图书馆

Demand-Aware erasure coding for Distributed Storage Systems

IEEE TRANSACTIONS ON CLOUD COMPUTING 2021年第2期9卷 532-545页

作者： Li, Jun Li, Baochun Florida Int Univ Sch Comp & Informat Sci Miami FL 33199 USA Univ Toronto Dept Elect & Comp Engn Toronto ON M4P 1A6 Canada

Distributed storage systems provide cloud storage services by storing data on commodity storage servers. Conventionally, data are protected against failures of such commodity servers by replication. erasure coding consumes less storage overhead than replication to tolerate the same number of failures and thus has been replacing replication in many distributed storage systems. However, with erasure coding, the overhead of reconstructing data from failures also increases significantly. Under the ever-changing workload where data accesses can be highly skewed, it is challenging to deploy erasure coding with appropriate values of parameters to achieve a well trade-off between storage overhead and reconstruction overhead. In this paper, we propose Zebra, a framework that encodes data by their demand into multiple tiers that deploy erasure codes with different values of parameters. Zebra automatically determines the number of such tiers and dynamically assigns erasure codes with optimal values of parameters into corresponding tiers. With Zebra, a flexible trade-off between storage overhead and reconstruction overhead is achieved with multiple tiers. When demand changes, Zebra adjusts itself with a marginal amount of network transfer. We demonstrate that Zebra can work with two representative families of erasure codes in distributed storage systems, Reed-Solomon codes and local reconstruction codes.

关键词： Distributed storage system demand skewness erasure coding reconstruction storage overhead Reed-Solomon code local reconstruction code

来源：评论

学校读者我要写书评

暂无评论

Enabling Efficient erasure coding in Disaggregated Memory Systems

引用

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS 2024年第1期35卷 154-168页

作者： Li, Qiliang Xu, Liangliang Li, Yongkun Lyu, Min Wang, Wei Zuo, Pengfei Xu, Yinlong Univ Sci & Technol China Sch Comp Sci & Technol Hefei 230027 Peoples R China Huawei Cloud Sch Comp Sci & Technol Shanghai 201206 Peoples R China Anhui Prov Key Lab High Performance Comp Hefei 230026 Peoples R China Huawei Cloud Shenzhen 518100 Peoples R China

Disaggregated memory (DM) separates compute and memory resources to build a huge memory pool. erasure coding (EC) is expected to provide fault tolerance in DM with low memory cost. In DM with EC, objects are first coded in compute servers, then directly written to memory servers via high-speed networks like one-sided RDMA. However, as the one-sided RDMA latency goes down to the microsecond level, coding overhead degrades the performance in DM with EC. To enable efficient EC in DM, we thoroughly analyze the coding stack from the perspective of cache efficiency and RDMA transmission. We develop MicroEC, which optimizes the coding workflow by reusing the auxiliary coding data and coordinates the coding and RDMA transmission with an exponential pipeline, as well as carefully adjusting the coding and transmission threads to minimize the latency. We implement a prototype supporting common basic operations, such as write/read/degraded read/recovery. Experiments show that MicroEC reduces the write latency by up to 44.35% and 42.14% and achieves up to 1.80 x and 1.73 write throughput, compared with the state-of-the-art DM systems with EC and 3-way replication for objects not smaller than 1 MB, respectively. For small objects, MicroEC also evidently reduces the variation of latency, e.g., it reduces the P99 latency of writing 1 KB objects by 27.81%.

关键词： Encoding Servers Memory management Random access memory Fault tolerant systems Fault tolerance Throughput Disaggregated memory erasure coding pipeline reliability

来源：评论

学校读者我要写书评

暂无评论

Cooperative transmission control scheme using erasure coding for vehicular delay-tolerant networks

引用

JOURNAL OF SUPERCOMPUTING 2014年第3期68卷 1462-1486页

作者： Tang, Xiaolan Pu, Juhua Ma, Kun Xiong, Zhang Beihang Univ State Key Lab Software Dev Environm Beijing 100191 Peoples R China Beihang Univ Res Inst Shenzhen Peoples R China

Transmission control is a challenging problem in vehicular delay-tolerant networks because the network has a high vehicle mobility which leads to a dynamic and unreliable network connectivity. In this paper, we propose a COoperative Transmission control Scheme using erasure coding (COTS) to improve the transmission efficiency. In this scheme, the cooperative message distribution algorithm uses delivery capacities to decide the transmission direction and traffic. Besides, the cooperative task scheduling algorithm sets the transfer order for messages from both sides. This algorithm uses a comprehensive parameter, the number of data blocks to send, to simplify the computation of transmission priority. After analyzing the vehicle transmission behavior under different storage conditions, we apply local deleting and creating behaviors instead of sending and receiving behaviors in specific states to decrease communication overhead. Simulation results show that COTS, compared with other well-known schemes, not only increases the delivery ratio but also decreases the delivery latency and the transmission overhead.

关键词： Vehicular delay-tolerant networks Transmission control Vehicle cooperation erasure coding

来源：评论

学校读者我要写书评

暂无评论

Cost analysis of erasure coding for exa-scale storage

引用

JOURNAL OF SUPERCOMPUTING 2019年第8期75卷 4638-4656页

作者： Kim, Dong-Oh Kim, Hong-Yeon Kim, Young-Kyun Kim, Jeong-Joon Elect & Telecommun Res Inst High Performance Comp Res Dept Daejeon South Korea Korea Polytech Univ Dept Comp Engn Shihung Gyeonggi Do South Korea

With the increasing demand for mass storage, research on exa-scale storage is actively underway. When the scale of storage grows to the exa-scale, the space efficiency becomes very important. To maintain the storage reliability and improve the space efficiency, we have begun to introduce erasure coding instead of replication. However, erasure coding has many I/O performance degradation factors such as Parity Calculation, degraded I/O, Data Distribution cost, etc., whereas the existing research mainly focuses on improving the performance of the Parity *** this study, we identified the issues and bottlenecks of using erasure coding in real storage. First, we measured the I/O performance of various erasure codes to find the suitable erasure codes for real storage. Next, we analyzed the execution time for each processing step when I/O was performed and the issues when erasure coding was used in storage. Finally, we predicted the cost of EC-based I/O processing in the exa-scale storage and identified the expected problems.

关键词： erasure coding Parity calculation cost Exa-scale storage ExSTorus-FS Degradation factor

来源：评论

学校读者我要写书评

暂无评论

An erasure coding-based loss-tolerant file delivery protocol for deep space communications

引用

ACTA ASTRONAUTICA 2011年第7-8期68卷 1409-1416页

作者： Li, Hui Luo, Hao Zhejiang Univ Sch Aeronaut & Astronaut Hangzhou 310027 Peoples R China

In deep space communications, huge delay, asymmetric bandwidth, large BER and packet loss ratio and intermittent link hinder the adoption of terrestrial transmission protocols. To decrease file delivery time, consultative committee for space data systems (CCSDS) file delivery protocol (CFDP) replaces acknowledge (ACK) by negative acknowledge (NAK) and simplifies the procedures of link establishment and link removal. However, feedback information cannot be avoided in CFDP;hence the file delivery performance cannot be further improved. Motivated by the idea of erasure coding, a one-way reliable transmission protocol named loss-tolerant file deliver protocol (LTFDP) that is suitable for deep space long-range file delivery is proposed in this paper. For LTFDP, if a certain percent of the data segments interweaved by explorers is received, the original data can be de-interweaved successfully by Earth stations;hence reliable transmission is accomplished without NAK or ACK. Obviously, the Earth stations have enough power and memory resources for the iteration process of de-interweaving. In a shorter distance communications scenario such as from the Moon to the Earth, performance of LTFDP is greatly impacted by file size and transmission rate. In contrast, propagation delay is the main factor of file delivery time in a longer distance communication scenario, e.g. from the Mars to the Earth. This paper demonstrated that LTFDP can not only diminish file delivery time greatly and improve goodput by feedback-reduced transmission, but also improve the reliability of deep space communications. (C) 2010 Elsevier Ltd. All rights reserved.

关键词： Deep space communication File delivery protocol Loss-tolerant file delivery protocol Interweave erasure coding Delay Goodput

来源：评论

学校读者我要写书评

暂无评论

Fountain-inspired erasure coding for real-time traffic Performance analysis and simulation

引用

TELECOMMUNICATION SYSTEMS 2011年第1-2期48卷 219-232页

作者： Tirronen, Tuomas Virtamo, Jorma Aalto Univ Dept Commun & Networking Espoo Finland

An erasure correction strategy based on fountain coding is proposed for traffic with real-time requirements. A sliding window marks the range of non-expired data. Each new block entering the window is once sent as such, followed by probabilistically sending a repair packet. The repair packets are formed as a random combination of the blocks in the current window using a degree distribution as in LT coding. The performance of the method with a given channel loss probability is analyzed using a Markov chain model. The state space, however, has to be truncated for computational tractability. The truncation error is verified to be small enough by simulations. By using the analytical model the optimal degree distribution is found to be of single-degree type. The performance of the proposed scheme is compared with deterministic settings, in which repair packets are sent after fixed number of systematic packets. Further comparison is made against Raptor coding, and we note that using the presented strategy can result in better performance in some situations.

关键词： Fountain coding erasure coding Markov chain Real-time traffic Simulation

来源：评论

学校读者我要写书评

暂无评论

erasure coding Optimization for Data Storage: Acceleration Techniques and Delayed Parities Generation

Erasure Coding Optimization for Data Storage: Acceleration T...

引用

作者： Zhou, Tianli Texas A&M University

学位级别：Ph.D., Doctor of Philosophy

Various techniques have been proposed in the literature to improve erasure code computation efficiency, including optimizing bitmatrix design and computation schedule, common XOR operation reduction, caching management techniques, and vectorization techniques. These techniques were largely proposed individually, and in this work, we seek to use them jointly. To accomplish this task, these techniques need to be thoroughly evaluated individually, and their relation better understood. Building on extensive testing, we develop methods to systematically optimize the computation chain together with the underlying bitmatrix. This led to a simple design approach of optimizing the bitmatrix by minimizing a weighted computation cost function, and also a straightforward coding procedure: follow a computation schedule produced from the optimized bitmatrix to apply XOR-level vectorization. This procedure provides better performances than most existing techniques (e.g., those used in ISA-L and Jerasure libraries), and sometimes can even compete against well-known but less general codes such as EVENODD, RDP, and STAR codes. One particularly important observation is that vectorizing the XOR operations is a better choice than directly vectorizing finite field operations, not only because of the flexibility in choosing finite field size and the better encoding throughput, but also its minimal migration efforts onto newer CPUs. A delayed parity generation technique for maximum distance separable (MDS) storage codes is proposed as well, for two possible applications: the first is to improve the write-speed during data intake where only a subset of the parities are initially produced and stored into the system, and the rest can be produced from the stored data during a later time of lower system load; the second is to provide better adaptivity, where a lower number of parities can be chosen initially in a storage system, and more parities can be produced when the existing ones are not s

关键词： erasure coding Delayed parity generation Data storage Bitmatrix Storage codes

来源：评论

学校读者我要写书评

暂无评论

Performance analysis of CCSDS File Delivery Protocol and erasure coding techniques in deep space environments

引用

COMPUTER NETWORKS 2007年第14期51卷 4032-4049页

作者： de Cola, Tomaso Ernst, Harald Marchese, Mario Univ Genoa Genoa Res Unit CNIT I-16145 Genoa Italy Inst Commun & Navigat German Aerosp Ctr DLR D-82234 Wessling Germany Univ Genoa Dept Commun Comp & Syst Sci I-16145 Genoa Italy

The rising demand for multimedia services even in hazardous environments, such as space missions and military theatres, and the consequent need of proper internetworking technologies have revealed the performance limits experienced by TCP protocol over long-delay and lossy links and highlighted the importance of the communication features provided by the protocol architectures proposed by the Consultative Committee for Space Data Systems (CCSDS). This paper proposes a CCSDS File Delivery Protocol (CFDP) extension, based on the implementation of erasure coding schemes, within the CFDP itself, in order to assure high reliability to the data communication even in presence of very critical conditions, such as hard shadowing, deep-fading periods and intermittent links. Different encoding techniques are considered and various channel conditions, in terms of Bit Error Ratio and bandwidth values, are tested. (C) 2007 Elsevier B.V. All rights reserved.

关键词： CFDP erasure coding transport layer satellite communication

来源：评论

学校读者我要写书评

暂无评论

FlexRaft: Exploiting Flexible erasure coding for Minimum-Cost Consensus and Fast Recovery

引用

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS 2024年第10期35卷 1826-1840页

作者： Zhang, Mi Kang, Qihan Lee, Patrick P. C. Chinese Acad Sci Inst Comp Technol Res Ctr Adv Comp Syst State Key Lab Processors Beijing 100045 Peoples R China Chinese Acad Sci Univ Chinese Acad Sci Inst Comp Technol Res Ctr Adv Comp Syst Beijing 101408 Peoples R China Chinese Univ Hong Kong Dept Comp Sci & Engn Hong Kong Peoples R China

Consensus protocols like Paxos and Raft provide data consistency and fault tolerance for distributed services. Log replication in these protocols can be supported by erasure coding, which incurs lower redundancy than full-copy replication and significantly saves network and storage costs for overall performance improvements. However, existing consensus protocols with erasure coding cannot achieve the minimum network and storage costs during log replication. We propose FlexRaft, which dynamically varies the coding scheme used in Raft based on the server status to always achieve the theoretically minimum redundancy ratio, while maintaining the same liveness as in Raft. To address the issue of an inconsistent coding scheme between the leader and its followers, we specify the prerequisite of overwriting a log entry and also allow the leader and its followers to exactly track the coding scheme being used. We further extend FlexRaft into FlexRaft+, which provides a different storage layout to vary the coding scheme through a novel technique called re-encoding-free replication, so as to enable fast server recovery. We prove that both FlexRaft and FlexRaft+ maintain Raft safety. We implement a prototype of FlexRaft and FlexRaft+, atop which we build a distributed key-value store to show its efficacy. Experiments on Alibaba Cloud show that FlexRaft achieves the theoretically minimum network and storage costs in practice, and reduces the commit latency by 44.51% and 19.37% compared with state-of-the-art CRaft and HRaft, respectively. FlexRaft+ further reduces the commit latency when the coding scheme is being varied and improves the server recovery performance.

关键词： Servers Encoding Costs Redundancy Consensus protocol Safety Distributed databases Consensus erasure coding

来源：评论

学校读者我要写书评

暂无评论

Delay control in MANETs with erasure coding and f-cast relay

引用

WIRELESS NETWORKS 2014年第8期20卷 2617-2631页

作者： Yang, Bin Gao, Juntao Zhou, Yuezhi Jiang, Xiaohong Future Univ Hakodate Sch Syst Informat Sci Hakodate Hokkaido 0418655 Japan Chuzhou Univ Sch Comp & Informat Engn Chuzhou 239000 Peoples R China Nara Inst Sci & Technol Sch Informat Sci Nara 6300192 Japan Tsinghua Univ Dept Comp Sci & Technol Beijing 100084 Peoples R China

Packet delay control in mobile ad hoc networks (MANETs) is critical to support delay-sensitive applications in such networks. By combining erasure coding and packet redundancy techniques, this paper proposes a general two-hop relay algorithm 2HR-(x, tau, f) for a flexible control of packet delivery delay in MANETs, where a group of x packets in source node are first encoded into x . tau encoded packets based erasure coding, and each encoded packet is then delivered to at most f distinct relay nodes (f-cast) that will help to forward the encoded packet to destination node. To understand the delay performance in a 2HR-(x, tau, f) MANET, we then develop a discrete time multi-dimensional Markov chain model to depict the packet delivery process in the network, based on which closed-form results on mean and variance of packet delivery delay are further derived. Finally, extensive simulation and theoretical results are provided to illustrate the efficiency of our delay models as well as the capability of the 2HR-(x, tau, f) algorithm in delay control.

关键词： Mobile ad hoc networks Two-hop relay erasure coding Packet redundancy Delay

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：