检索结果-内蒙古大学图书馆

Dynamic-EC:an efficient dynamic erasure coding method for permissioned blockchain systems

Frontiers of Computer Science 2025年第1期19卷 15-28页

作者： Mizhipeng ZHANG Chentao WU Jie LI Minyi GUO Department of Computer Science and Engineering Shanghai Jiao Tong UniversityShanghai 200200China Yancheng Blockchain Research Institute Hengyang 421200China

Blockchain as a decentralized storage technology is widely used in many *** has extremely strict requirements for reliability because there are many potentially malicious ***,blockchain is a chain storage structure formed by interconnecting blocks1),which are stored by full replication method,where each node stores a replica of all blocks and the data consistency is maintained by the consensus *** decrease the storage overhead,previous approaches such as BFT-Store and Partition Chain store blocks via erasure ***,existing erasure coding based methods utilize static encoding schema to tolerant f malicious nodes,but in the typical cases,the number of malicious nodes is much smaller than f as described in previous *** redundant parities to tolerate excessive malicious nodes introduces unnecessary storage *** solve the above problem,we propose Dynamic-EC,which is a Dynamic erasure coding method in permissioned blockchain *** key idea of Dynamic-EC is to reduce the storage overhead by dynamically adjusting the total number of parities according to the risk level of the whole system,which is determined by the number of perceived malicious nodes,while ensuring the system *** demonstrate the effectiveness of Dynamic-EC,we conduct several experiments on an open source blockchain software *** results show that,compared to the state-of-the-art erasure coding methods,Dynamic-EC reduces the storage overhead by up to 42%,and decreases the average write latency of blocks by up to 25%,respectively.

关键词： blockchain Byzantine Fault Tolerance(BFT) erasure coding consensus reputation evaluation

来源：评论

学校读者我要写书评

暂无评论

erasure coding Aware Block Placement for Data-Intensive Applications 5

Erasure Coding Aware Block Placement for Data-Intensive Appl...

引用

5th Workshop on Challenges and Opportunities of Efficient and Performant Storage Systems-CHEOPS

作者： Ibrahim, Shadi Darrous, Jad Univ Rennes INRIA CNRS IRISA Rennes France INRIA IMT Atlant LS2N Nantes France

ISBN: (纸本)9798400715297

erasure coding (EC) has recently been integrated and deployed in the Hadoop Distributed File System (HDFS) to provide the same fault tolerance guarantees as replication, but with significantly less storage overhead. When EC is used, data reads typically involve only data chunks. In this paper, we study the effect of data chunk distribution on the performance of reads and data-intensive applications, and present the design and evaluation of an erasure coding aware (EC-aware) block placement that balances the distribution of data chunks across nodes. Experimental results show that EC-aware block placement can reduce the execution time of Sort and WordCount applications by up to 25%.

关键词： erasure coding Storage systems Data analytic frameworks HDFS Reed-Solomon codes

来源：评论

学校读者我要写书评

暂无评论

Towards benchmarking erasure coding schemes in object storage system: A systematic review

引用

FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE 2025年 163卷

作者： Noor, Jannatun Upoma, Rezuana Imtiaz Sakif, Md. Sadiqul Islam Islam, A. B. M. Alim Al BRAC Univ Sch Data & Sci Comp Sustainabil & Social Good C2SG Res Grp Dhaka Bangladesh BRAC Univ Dept Comp Sci & Engn Sch Data & Sci Dhaka Bangladesh Bangladesh Univ Engn & Technol Dept CSE Next Generat Comp Res Grp Dhaka Bangladesh

erasure coding (EC) in cloud storage minimizes data replication by reconstructing data from parity fragments. This method enhances data redundancy and efficiency while reducing storage costs and improving fault tolerance. It is more advantageous than replication in Object Storage Systems. EC guarantees data integrity by ensuring lossless transmission of all coded pieces. As data volumes continue to increase rapidly, the time efficiency of the EC method becomes crucial in ensuring optimal system performance. Various variables, including the algorithm employed, data size, number of storage nodes, hardware resources, and network conditions, can influence the speed of EC operations. Although some literature covers various aspects, there is still a research gap in understanding the I/O activities, time efficiency, and fault tolerance of EC in object storage systems. Hence, our research aims to address these challenges in cloud-based object storage systems. We analyze and benchmark the data storage I/O performance of OpenStack Swift, focusing on the time efficiency of the Reed-Solomon (RS) algorithm across two datasets. Additionally, our contributions include benchmarking EC performance in both local and remote testbeds, utilizing the SimEDC simulator for comprehensive efficiency and fault tolerance assessments. Moreover, we create a comprehensive dataset (MCSD-100) for benchmarking and conduct a systematic literature review. Finally, we identify and discuss future opportunities for enhancing EC in cloud-based object storage systems.

关键词： erasure coding OpenStack swift Reed-Solomon codes Cloud storage system

来源：评论

学校读者我要写书评

暂无评论

EdgeHydra: Fault-Tolerant Edge Data Distribution Based on erasure coding

引用

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS 2025年第1期36卷 29-42页

作者： He, Qiang Zhang, Guobiao Wang, Jiawei Luo, Ruikun Dai, Xiaohai Hu, Yuchong Chen, Feifei Jin, Hai Yang, Yun Huazhong Univ Sci & Technol Natl Engn Res Ctr Big Data Technol & Syst Sch Comp Sci & Technol Serv Comp Technol & Syst LabCluster & Grid Comp L Wuhan 430074 Peoples R China Swinburne Univ Technol Dept Comp Technol Melbourne Vic 3122 Australia Huazhong Univ Sci & Technol Sch Comp Sci & Technol Key Lab Informat Storage Syst Minist Educ China Wuhan 430074 Peoples R China Deakin Univ Sch Informat Technol Geelong Vic 3125 Australia

In the edge computing environment, app vendors can distribute popular data from the cloud to edge servers to provide low-latency data retrieval. A key problem is how to distribute these data from the cloud to edge servers cost-effectively. Under current schemes, a file is divided into some data blocks for parallel transmissions from the cloud to target edge servers. Edge servers can then combine received data blocks to reconstruct the file. While this method expedites the data distribution process, it presents potential drawbacks. It is sensitive to transmission delays and transmission failures caused by runtime exceptions like network fluctuations and server failures. This paper presents EdgeHydra, the first edge data distribution scheme that tackles this challenge through fault tolerance based on erasure coding. Under EdgeHydra, a file is encoded into data blocks and parity blocks for parallel transmission from the cloud to target edge servers. An edge server can reconstruct the file upon the receipt of a sufficient number of these blocks without having to wait for all the blocks in transmission. It also innovatively employs a leaderless block supplement mechanism to ensure the receipt of sufficient blocks for individual target edge servers. These improve the robustness of the data distribution process significantly. Extensive experiments show that EdgeHydra can tolerate delays and failures in individual transmission links effectively, outperforming the state-of-the-art scheme by up to 50.54% in distribution time.

关键词： Servers Delays Costs Encoding Runtime Fluctuations Fault tolerant systems Fault tolerance Steiner trees Edge computing Edge data caching erasure coding data distribution distributed transmission

来源：评论

学校读者我要写书评

暂无评论

An Optimized Method for Batch Recovery Based on erasure coding in Heterogeneous Network

引用

SENSORS 2025年第2期25卷 346-346页

作者： Song, Ying Liu, Jialin Tian, Yingai Wang, Bo Beijing Informat Sci & Technol Univ Sch Comp Sci Beijing 102206 Peoples R China Zhengzhou Univ Light Ind Software Engn Coll Zhengzhou 450002 Peoples R China

With the rapid development of IoT technology, sensors are widely used for monitoring environmental parameters. The data collected by sensors needs to be stored, and distributed storage systems provide an excellent platform to handle this vast amount of data. To enhance data reliability and reduce storage costs, erasure coding technology can be employed within distributed storage systems. However, the process of recovering lost or damaged data inevitably generates significant cross-rack traffic. In recent years, various batch recovery methods have been designed to improve data recovery speed and reduce cross-rack recovery traffic, but they have limitations in different aspects, such as less consideration for heterogeneous cross-rack bandwidth and insufficient handling of recovery link scheduling. This paper proposes the HBRepair recovery framework, aimed at improving data recovery speed by balancing the recovery load and reducing cross-rack recovery traffic. Firstly, HBRepair strategically selects helper blocks and nodes for storing recovery blocks to find the optimal batch recovery plan. Then, it selectively and rationally schedules recovery links to saturate idle bandwidth resources and avoid network congestion. Experimental results show that by optimizing the use of cross-rack bandwidth, HBRepair can reduce cross-rack recovery time by up to 26.74%, effectively addressing the shortcomings of existing methods.

关键词： bandwidth management load balancing distributed storage system heterogeneous network erasure coding batch recovery Internet of Things

来源：评论

学校读者我要写书评

暂无评论

Enabling Efficient erasure coding in Disaggregated Memory Systems

引用

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS 2024年第1期35卷 154-168页

作者： Li, Qiliang Xu, Liangliang Li, Yongkun Lyu, Min Wang, Wei Zuo, Pengfei Xu, Yinlong Univ Sci & Technol China Sch Comp Sci & Technol Hefei 230027 Peoples R China Huawei Cloud Sch Comp Sci & Technol Shanghai 201206 Peoples R China Anhui Prov Key Lab High Performance Comp Hefei 230026 Peoples R China Huawei Cloud Shenzhen 518100 Peoples R China

Disaggregated memory (DM) separates compute and memory resources to build a huge memory pool. erasure coding (EC) is expected to provide fault tolerance in DM with low memory cost. In DM with EC, objects are first coded in compute servers, then directly written to memory servers via high-speed networks like one-sided RDMA. However, as the one-sided RDMA latency goes down to the microsecond level, coding overhead degrades the performance in DM with EC. To enable efficient EC in DM, we thoroughly analyze the coding stack from the perspective of cache efficiency and RDMA transmission. We develop MicroEC, which optimizes the coding workflow by reusing the auxiliary coding data and coordinates the coding and RDMA transmission with an exponential pipeline, as well as carefully adjusting the coding and transmission threads to minimize the latency. We implement a prototype supporting common basic operations, such as write/read/degraded read/recovery. Experiments show that MicroEC reduces the write latency by up to 44.35% and 42.14% and achieves up to 1.80 x and 1.73 write throughput, compared with the state-of-the-art DM systems with EC and 3-way replication for objects not smaller than 1 MB, respectively. For small objects, MicroEC also evidently reduces the variation of latency, e.g., it reduces the P99 latency of writing 1 KB objects by 27.81%.

关键词： Encoding Servers Memory management Random access memory Fault tolerant systems Fault tolerance Throughput Disaggregated memory erasure coding pipeline reliability

来源：评论

学校读者我要写书评

暂无评论

FlexRaft: Exploiting Flexible erasure coding for Minimum-Cost Consensus and Fast Recovery

引用

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS 2024年第10期35卷 1826-1840页

作者： Zhang, Mi Kang, Qihan Lee, Patrick P. C. Chinese Acad Sci Inst Comp Technol Res Ctr Adv Comp Syst State Key Lab Processors Beijing 100045 Peoples R China Chinese Acad Sci Univ Chinese Acad Sci Inst Comp Technol Res Ctr Adv Comp Syst Beijing 101408 Peoples R China Chinese Univ Hong Kong Dept Comp Sci & Engn Hong Kong Peoples R China

Consensus protocols like Paxos and Raft provide data consistency and fault tolerance for distributed services. Log replication in these protocols can be supported by erasure coding, which incurs lower redundancy than full-copy replication and significantly saves network and storage costs for overall performance improvements. However, existing consensus protocols with erasure coding cannot achieve the minimum network and storage costs during log replication. We propose FlexRaft, which dynamically varies the coding scheme used in Raft based on the server status to always achieve the theoretically minimum redundancy ratio, while maintaining the same liveness as in Raft. To address the issue of an inconsistent coding scheme between the leader and its followers, we specify the prerequisite of overwriting a log entry and also allow the leader and its followers to exactly track the coding scheme being used. We further extend FlexRaft into FlexRaft+, which provides a different storage layout to vary the coding scheme through a novel technique called re-encoding-free replication, so as to enable fast server recovery. We prove that both FlexRaft and FlexRaft+ maintain Raft safety. We implement a prototype of FlexRaft and FlexRaft+, atop which we build a distributed key-value store to show its efficacy. Experiments on Alibaba Cloud show that FlexRaft achieves the theoretically minimum network and storage costs in practice, and reduces the commit latency by 44.51% and 19.37% compared with state-of-the-art CRaft and HRaft, respectively. FlexRaft+ further reduces the commit latency when the coding scheme is being varied and improves the server recovery performance.

关键词： Servers Encoding Costs Redundancy Consensus protocol Safety Distributed databases Consensus erasure coding

来源：评论

学校读者我要写书评

暂无评论

Accelerating erasure coding by exploiting multiple repair paths in distributed storage systems

引用

CLUSTER COMPUTING-THE JOURNAL OF NETWORKS SOFTWARE TOOLS AND APPLICATIONS 2024年第6期27卷 8621-8635页

作者： Kim, Chanki Chon, Kang-Wook Jeonbuk Natl Univ Dept Comp Sci & Artificial Intelligence 567 Baekje Daero Jeonju Si 54896 Jeollabuk Do South Korea Korea Univ Technol & Educ Sch Comp Sci & Engn 1600 Chungjeol Ro Cheonan Si 31253 Chungcheongnam South Korea

High reliability must be ensured in distributed storage systems (DSSs) to maintain the stability of warehouse-scale computing and high-performance computing (HPC) systems. For system-level reliability, a repair operation using redundant storage nodes can be used in conjunction with erasure coding (EC), which can also affect the system performance. The existing EC design mainly focused on minimizing the required bandwidth for the repair and storage overheads. However, the computing performance for EC should be considered to achieve high bandwidth in order to exploit back-end network link capacity with heterogeneous and high-speed interconnects over 10 Gbps Ethernet. In this study, a new computing acceleration method for repair operation in EC is proposed using multiple repair paths and modifying the computation kernel on the graphics processing unit (GPU) device. For the Cauchy Reed-Solomon (CRS) codes, the proposed scheme is observed to achieve sufficient repair bandwidth compared to the theoretical bound or exceed the current maximum Ethernet link bandwidth.

关键词： Cauchy Reed-Solomon (CRS) codes Computing acceleration erasure coding Fault-tolerant system Graphics processing units (GPUs) High-performance computing (HPC)

来源：评论

学校读者我要写书评

暂无评论

erasure coding Based Optimization in Decentralized Distributed Storage Systems 17

Erasure Coding Based Optimization in Decentralized Distribut...

引用

17th International Conference on Networking, Architecture and Storage

作者： Gan, Yiwei Huang, Zhijie Shi, Yulong Zhang, Xiao Zhao, Nannan Northwestern Polytech Univ Sch Comp Sci Xian Peoples R China

ISBN: (纸本)9798331520519;9798331520502

Node failures in decentralized distributed storage systems are common. To ensure data availability, these systems employ data redundancy mechanisms, typically relying on replicas. This paper proposes a decentralized data redundancy scheme based on erasure coding, with Reed-Solomon and IPFS as examples. Compared with replication, the proposed scheme reduces storage space and enhances fault tolerance. Files are sharded and encoded across multiple nodes, avoiding the high redundancy of replicas. Users can retrieve any K shards from N nodes to reconstruct the original files. This erasure coding optimization combines efficient data exchange among decentralized nodes with erasure coding technology, significantly reducing storage space compared with the replica mechanism. The implementation involves truncating and sharding the blocks within the Merkle DAG generated by files, enabling flexible adjustments to the code rate of erasure codes and the allocation of storage nodes based on user needs and available resources. This method achieves a balance between storage efficiency and data availability.

关键词： Decentralized Distributed Storage System erasure coding Reed-Solomon IPFS Merkle DAG

来源：评论

学校读者我要写书评

暂无评论

BPR: An erasure coding Batch Parallel Repair Approach in Distributed Storage Systems

引用

IEEE ACCESS 2023年 11卷 44509-44518页

作者： Song, Ying Zhao, Wenxuan Wang, Bo Beijing Informat Sci & Technol Univ Beijing Key Lab Internet Culture & Digital Dissemi Beijing 100101 Peoples R China Beijing Informat Sci & Technol Univ Beijing Adv Innovat Ctr Mat Genome Engn Beijing 100101 Peoples R China Chinese Acad Sci Inst Comp Technol State Key Lab Comp Architecture Beijing 100086 Peoples R China Zhengzhou Univ Light Ind Software Engn Coll Zhengzhou 450002 Peoples R China

Today, erasure coding is one of the most significant techniques widely used in distributed systems because it can improve reliability for large amounts of data with low storage overhead. However, when the distributed system encounters a large number of data loss in stripes and requires batch-stripes data recovery, current data recovery methods either repeat the single-stripe recovery method or only optimize partial stripe recovery when recovering large-scale stripes, which incurs heavy upload and download repair traffics and imbalanced load, affecting the efficiency of fault recovery and wasting additional resources. In this paper, we propose BPR, an erasure coding batch parallel repair approach for distributed storage systems. BPR reduces cross-rack network transfer time and increases recovery throughput by classifying the stripes and recovering the data of stripes in batches through the forward and reverse parallel data recovery. The experiment results show that for large-scale stripes recovery, BPR reduces the cross-rack network transfer time by up to 10% and increases the recovery throughput by up to 8% compared with the rPDL in some scenarios.

关键词： Distributed processing Business process re-engineering Traffic congestion Encoding Full-duplex system Decoding Bandwidth Storage management Distributed storage system erasure coding data recovery

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：