检索结果-内蒙古大学图书馆

Data-Aware Adaptive compression for Stream Processing

IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING 2024年第9期36卷 4531-4549页

作者： Zhang, Yu Zhang, Feng Li, Hourun Zhang, Shuhao Guo, Xiaoguang Chen, Yuxing Pan, Anqun Du, Xiaoyong Renmin Univ China Sch Informat Key Lab Data Engn & Knowledge Engn MOE Beijing 100872 Peoples R China Nanyang Technol Univ NTU Sch Comp Sci & Engn SCSE Singapore 639798 Singapore Tencent Inc Database R&D Dept Shenzhen Peoples R China

Stream processing has been in widespread use, and one of the most common application scenarios is SQL query on streams. By 2021, the global deployment of IoT endpoints reached 12.3 billion, indicating a surge in data generation. However, the escalating demands for high throughput and low latency in stream processing systems have posed significant challenges due to the increasing data volume and evolving user requirements. We present a compression-based stream processing engine, called CompressStreamDB, which enables adaptive fine-grained stream processing directly on compressed streams, to significantly enhance the performance of existing stream processing solutions. CompressStreamDB utilizes nine diverse compression methods tailored for different stream data types and integrates a cost model to automatically select the most efficient compression schemes. CompressStreamDB provides high throughput with low latency in stream SQL processing by identifying and eliminating redundant data among streams. Our evaluation demonstrates that CompressStreamDB improves average performance by 3.84x and reduces average delay by 68.0% compared to the state-of-the-art stream processing solution for uncompressed streams, along with 68.7% space savings. Besides, our edge trials show an average throughput/price ratio of 9.95x and a throughput/power ratio of 7.32x compared to the cloud design.

关键词： Encoding compression algorithms Real-time systems Low latency communication Data processing Costs Delays Data compaction and compression stream processing edge computing

来源：评论

学校读者我要写书评

暂无评论

Generalized Biwords for Bitext compression and Translation Spotting

引用

JOURNAL OF ARTIFICIAL INTELLIGENCE RESEARCH 2012年第1期43卷 389-418页

作者： Sanchez-Martinez, Felipe Carrasco, Rafael C. Martinez-Prieto, Miguel A. Adiego, Joaquin Univ Alacant Dept Llenguatges & Sistemes Informat E-03071 Alacant Spain Univ Valladolid Dept Informat E-47011 Valladolid Spain

Large bilingual parallel texts (also known as bitexts) are usually stored in a compressed form, and previous work has shown that they can be more efficiently compressed if the fact that the two texts are mutual translations is exploited. For example, a bitext can be seen as a sequence of biwords pairs of parallel words with a high probability of co-occurrence that can be used as an intermediate representation in the compression process. However, the simple biword approach described in the literature can only exploit one-to-one word alignments and cannot tackle the reordering of words. We therefore introduce a generalization of biwords which can describe multi-word expressions and reorderings. We also describe some methods for the binary compression of generalized biword sequences, and compare their performance when different schemes are applied to the extraction of the biword sequence. In addition, we show that this generalization of biwords allows for the implementation of an efficient algorithm to look on the compressed bitext for words or text segments in one of the texts and retrieve their counterpart translations in the other text-an application usually referred to as translation spotting-with only some minor modifications in the compression algorithm.

关键词： compression algorithms Generalization Word COMPRESSED texts REORDERING efficient algorithm compression Metrorrhagia

来源：评论

学校读者我要写书评

暂无评论

Vessel AIS Trajectory Online compression Based on Scan-Pick-Move Algorithm Added Sliding Window

引用

IEEE ACCESS 2020年 8卷 109350-109359页

作者： Sun, Shuang Chen, Yan Piao, Zaiji Zhang, Jinsong Dalian Maritime Univ Sch Maritime Econ & Management Dalian 116026 Peoples R China Dalian Maritime Univ Sch Marine Elect Engn Dalian 116026 Peoples R China

The trajectory data of vessel AIS (automatic identification system) has important theoretical and application value for information supporting decisions. However, large sizes lead to difficulties in storing, querying, and processing. To solve the problems of high compression ratio and longtime consumption of the existing online trajectory compression algorithm, an SPM (scan-pick-move) trajectory data compression algorithm added sliding window is proposed. In order to better compress vessel trajectory data regarding compression efficiency, the sliding window is added to the classical SPM algorithm. In order to reduce trajectory data storage space, the maximum offset distance reference trajectory point is used as the criterion of whether the current trajectory point can be compressed. In this paper, the multi-dimensional space-time characteristics of trajectory data, such as distance error, compression ratio and compression time, are selected to evaluate the trajectory compression method from three levels: geometric characteristics, motion characteristics and compression efficiency. Compared with the existing SPM trajectory data compression algorithm, parallel experiments are conducted based on AIS data gathered over the duration of a month in the Japan Osaka Bay. The SPM trajectory compression algorithm added sliding window can significantly reduce the compression time and outperforms other existing trajectory compression algorithms in term of average compression error at high compression strengths. Also, the proposed method has high compression efficiency in the range of commonly used compression thresholds.

关键词： Trajectory Artificial intelligence compression algorithms Marine vehicles Data compression Microsoft Windows Windows AIS data compression ratio scan-pick-move sliding window vessel trajectory compression

来源：评论

学校读者我要写书评

暂无评论

A Rolling Hash Algorithm and the Implementation to LZ4 Data compression

引用

IEEE ACCESS 2020年 8卷 35529-35534页

作者： Jiang, Hao Lin, Sian-Jheng Univ Sci & Technol China Sch Informat Sci & Technol CAS Key Lab Electromagnet Space Informat Hefei 230027 Peoples R China

LZ77 is a dictionary compression algorithm by replacing the repeating sequence with the addresses of the previous referenced data in the stream. To find out these repetition, the LZ77 encoder maintains a hashing table, which have to frequently calculate hash values during the encoding process. In this paper, we present a class of rolling hash functions, that can calculate multiple hash values via a carry-less multiplication instruction. Then the proposed hash function is implemented in LZ4, which is a derivative of LZ77. The simulation shows that the encoding throughput of LZ4 has 15.7% improvement in average, and the compression ratio is +/- 1% in most cases.

关键词： Encoding Data compression compression algorithms Dictionaries Indexes Image coding Image restoration Carry-less multiplication LZ4 LZ77 rolling hash SIMD

来源：评论

学校读者我要写书评

暂无评论

Comprehensive approach to choice of compression and error-control coding algorithms for digital image transmission through communication channels

Comprehensive approach to choice of compression and error-co...

引用

International Symposium on Image and Signal Processing and Analysis

作者： N.I. Glumov Image Processing Systems Institute of the RAS Samara Russia

The problem of compression and error-control coding algorithms for digital image transmission through communication channels is considered. The criteria of the output image quality are discussed, and a technique for modeling faults in communication channels is proposed. In an example of the compression method based on hierarchical grid interpolation, a possibility of significant increase of noise immunity of compressed images is shown.

关键词： Image coding Digital images Communication channels Signal processing algorithms compression algorithms Error correction codes Interpolation Stability Redundancy Image processing

来源：评论

学校读者我要写书评

暂无评论

Multi-Channel Data Structure and Real-Time compression Algorithm Research

引用

IEEE AEROSPACE AND ELECTRONIC SYSTEMS MAGAZINE 2009年第11期24卷 28-35页

作者： Ren, Yongfeng Liu, Xiaohua Xu, Wenqiang Zhang, Wendong N Univ China Natl Key Lab Elect Measurement Technol Key Lab Instrumentat Sci & Dynam Measurement Minist Educ Taiyuan 030051 Peoples R China

The development of space telemetry technology has brought forward the need for large capacity memory of any solid-state recorder;data compression therefore, becomes more and more important. The compression feasibility and potentiality of telemetry data are examined by analyzing the statistical characteristics of actual telemetry data recovered from recorders. Aiming at the disadvantage of present data formats in data compression for multi-channel telemetry data acquisition systems, this introduces a data packet structure, and a real-time compression algorithm for low complex hardware design. The principles and implementations of data package compression are described. Simulation results show that this technology can meet the requirements of multi-channel real-time data compression with a high compression ratio and a fast compression speed, which possesses great application value.

关键词： Telemetry compression algorithms Hardware Data compression Encoding Real time systems

来源：评论

学校读者我要写书评

暂无评论

XCQ: A queriable XML compression system

引用

KNOWLEDGE AND INFORMATION SYSTEMS 2006年第4期10卷 421-452页

作者： Ng, Wilfred Lam, Wai-Yeung Wood, Peter T. Levene, Mark Hong Kong Univ Sci & Technol Dept Comp Sci Kowloon Hong Kong Peoples R China Univ London Sch Comp Sci & Informat Syst London England

XML has already become the de facto standard for specifying and exchanging data on the Web. However, XML is by nature verbose and thus XML documents are usually large in size, a factor that hinders its practical usage, since it substantially increases the costs of storing, processing, and exchanging data. In order to tackle this problem, many XML-specific compression systems, such as XMill, XGrind, XMLPPM, and Millau, have recently been proposed. However, these systems usually suffer from the following two inadequacies: They either sacrifice performance in terms of compression ratio and execution time in order to support a limited range of queries, or perform full decompression prior to processing queries over compressed documents. In this paper, we address the above problems by exploiting the information provided by a Document Type Definition (DTD) associated with an XML document. We show that a DTD is able to facilitate better compression as well as generate more usable compressed data to support querying. We present the architecture of the XCQ, which is a compression and querying tool for handling XML data. XCQ is based on a novel technique we have developed called DTD Tree and SAX Event Stream Parsing (DSP). The documents compressed by XCQ are stored in Partitioned Path-Based Grouping (PPG) data streams, which are equipped with a Block Statistics Signature (BSS) indexing scheme. The indexed PPG data streams support the processing of XML queries that involve selection and aggregation, without the need for full decompression. In order to study the compression performance of XCQ, we carry out comprehensive experiments over a set of XML benchmark datasets.

关键词： XML document type definitions compression algorithms query processing performance

来源：评论

学校读者我要写书评

暂无评论

A 65-nm CMOS Lossless Bio-Signal compression Circuit With 250 FemtoJoule Performance Per Bit

引用

IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEMS 2019年第5期13卷 1087-1100页

作者： Crispin-Bailey, Christopher Dai, Chenglaing Austin, Jim Univ York Dept Comp Sci York YO10 5GH N Yorkshire England Imperial Coll London London SW7 2AZ England

A 65nm CMOS integrated circuit implementation of a bio-physiological signal compression device is presented, reporting exceptionally low power, and extremely low silicon area cost, relative to state-of-the-art. A novel 'xor-log2-sub-band' data compression scheme is evaluated, achieving modest compression, but with very low resource cost. With the intent to design the simplest useful compression algorithm', the outcome is demonstrated to be very favourable where power must be saved by trading off compression effort against data storage capacity, or data transmission power, even where more complex algorithms can deliver higher compression ratios. A VLSI design and fabricated Integrated Circuit implementation are presented, and estimated performance gains and efficiency measures for various bio-medical use-cases are given. Power costs as low as 1.2 pJ per sample-bit are suggested for a 10kSa/s data-rate, whilst utilizing a power-gating scenario, and dropping to 250fJ/bit at continuous conversion data-rates of 5MSa/sec. This is achieved with a diminutive circuit area of 155um(2). Both power and area appear to be state-of-the-art in terms of compression versus resource cost, and this yields benefit for system optimization.

关键词： Encoding Integrated circuits Data compression compression algorithms Biomedical measurement Hardware ECG EEG lossless data compression power efficiency VLSI design wearable sensors

来源：评论

学校读者我要写书评

暂无评论

HADC: A Hybrid compression Approach for DNA Sequences

引用

IEEE ACCESS 2022年 10卷 106841-106848页

作者： Elnady, Sarah Sayed, Sabah Salah, Akram Cairo Univ Fac Comp & Artificial Intelligence Comp Sci Dept Cairo 12613 Egypt

In the blossoming age of Next Generation Sequencing (NGS) technologies, genome sequencing has become much easier and more affordable. The large number of enormous genomic sequences obtained demand the availability of huge storage space in order to be kept for analysis. Since the storage cost has become an impediment facing biologists, there is a constant need of software that provides efficient compression of genomic sequences. Most general-purpose compression algorithms do not exploit the inherent redundancies that exist in genomic sequences which is the reason for the success and popularity of reference-based compression algorithms. In this research, a new reference-based lossless compression technique is proposed for deoxyribonucleic acid (DNA) sequences stored in FASTA format which can act as a layer above gzip compression. Several experiments were performed to evaluate this technique and the experimental results show that it is able to obtain promising compression ratios saving up to 99.9% space and reaching a gain of 80% for some plant genomes. The proposed technique also succeeds in performing the compression at acceptable time;even saving more than 50% of the time taken by ERGC in most experiments.

关键词： Bioinformatics Genomics DNA Sequential analysis compression algorithms Encoding Redundancy Bioinformatics DNA sequences reference-based compression greedy alignment

来源：评论

学校读者我要写书评

暂无评论

Sub-block interchange for lossless image compression

引用

IEEE TRANSACTIONS ON CONSUMER ELECTRONICS 1999年第1期45卷 236-242页

作者： Ng, KS Cheng, LM City Univ Hong Kong Dept Elect Engn Kowloon SAR Hong Kong Peoples R China

Lempel-Ziv-Welch (LZW) technique for text compression has been successfully modified to lossless image compression such as GIF. Recently, a new class of text compression, namely, Burrows and Wheeler Transformation (BWT) has been developed which gives promising results for text compression. Here, we propose a sub-block interchange lossless compression method which belongs to this block sorting class. Our compression results have outperformed GIF in compression ratios and BWT in compression times when tested with 512x512 pixel 8-bit grey scale images. The comparison of compression ratios and times with GIF, BWT and other popular LZ based compression methods are discussed.

关键词： Image coding Sorting Testing Pixel compression algorithms Data compression Performance loss Decoding Entropy coding

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：