检索结果-内蒙古大学图书馆

Data-Aware Adaptive compression for Stream Processing

IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING 2024年第9期36卷 4531-4549页

作者： Zhang, Yu Zhang, Feng Li, Hourun Zhang, Shuhao Guo, Xiaoguang Chen, Yuxing Pan, Anqun Du, Xiaoyong Renmin Univ China Sch Informat Key Lab Data Engn & Knowledge Engn MOE Beijing 100872 Peoples R China Nanyang Technol Univ NTU Sch Comp Sci & Engn SCSE Singapore 639798 Singapore Tencent Inc Database R&D Dept Shenzhen Peoples R China

Stream processing has been in widespread use, and one of the most common application scenarios is SQL query on streams. By 2021, the global deployment of IoT endpoints reached 12.3 billion, indicating a surge in data generation. However, the escalating demands for high throughput and low latency in stream processing systems have posed significant challenges due to the increasing data volume and evolving user requirements. We present a compression-based stream processing engine, called CompressStreamDB, which enables adaptive fine-grained stream processing directly on compressed streams, to significantly enhance the performance of existing stream processing solutions. CompressStreamDB utilizes nine diverse compression methods tailored for different stream data types and integrates a cost model to automatically select the most efficient compression schemes. CompressStreamDB provides high throughput with low latency in stream SQL processing by identifying and eliminating redundant data among streams. Our evaluation demonstrates that CompressStreamDB improves average performance by 3.84x and reduces average delay by 68.0% compared to the state-of-the-art stream processing solution for uncompressed streams, along with 68.7% space savings. Besides, our edge trials show an average throughput/price ratio of 9.95x and a throughput/power ratio of 7.32x compared to the cloud design.

关键词： Encoding compression algorithms Real-time systems Low latency communication Data processing Costs Delays Data compaction and compression stream processing edge computing

来源：评论

学校读者我要写书评

暂无评论

compression of Multichannel Signals With Irregular Sampling Rates and Data Gaps

引用

IEEE ACCESS 2024年 12卷 193195-193211页

作者： Cervenansky, Pablo Martin, Alvaro Seroussi, Gadiel PEDECIBA Informat Montevideo 11800 Uruguay Univ Republica Fac Ingn Inst Comp Montevideo 11300 Uruguay Univ Republica Fac Ingn Inst Ingn Elect Montevideo 11300 Uruguay

In this work, we study the compression of multichannel signals with irregular sampling rates and data gaps. We consider state-of-the-art algorithms, which were originally designed to compress gapless signals with regular sampling, adapt them to operate with signals with irregular sampling rates and data gaps, and then evaluate their performance experimentally, through the compression of signals obtained from real-world datasets. Both the original algorithms and our schemes compress signals by exploiting their temporal, and, in some cases, spatial correlation. They work in a near-lossless fashion, guaranteeing a bounded absolute error between each decompressed sample and its original value. This includes the important lossless compression case, which corresponds to an error bound of zero. Our schemes first encode the position of the gaps, using arithmetic coding combined with a Krichevsky-Trofimov probability assignment on a Markov model, and then encode the data values separately. Our experimental analysis consists of comparing the compression performance of our schemes with each other, and with representative special-purpose and general-purpose lossless compression algorithms. We also measure and compare the schemes' running times, to assess their practicality. From the results we extract some general conclusions: in the lossless case, TS2Diff and LZMA attain the best compression performance, whereas our adaptation of algorithm APCA is preferred for positive error bounds. At the same time, our adaptation of APCA, and TS2Diff, attain some of the best running times.

关键词： Encoding Correlation Image coding Approximation algorithms Data models compression algorithms Principal component analysis Time series analysis Classification algorithms Arrays Multichannel signal compression time series compression near-lossless compression irregular sampling rate data gaps

来源：评论

学校读者我要写书评

暂无评论

DNA Sequences compression algorithms Based on the Two bits Codation Method

DNA Sequences Compression Algorithms Based on the Two bits C...

引用

IEEE International Conference on Bioinformatics and Biomedicine

作者： Bacem Saada Jing Zhang Harbin Engineering University

ISBN: (纸本)9781467368001

Nowadays a large number of DNA sequences is being stored on online databases. To reduce this quantity of information, researchers have been trying to implement new DNA sequences compression techniques based on the LOSSLESS algorithms. In this article we will compare two algorithms using the binary representation of DNA sequences. Those two algorithms are characterized by their ease of implementation. The first one transforms the DNA sequence into extended-ASCII representation while the second algorithm transforms it into a hexadecimal representation. Thereafter, we will apply the RLE technique to further enhance the compression of entire genomes.

关键词： DNA sequences Extended-ASCII code Hexadecimal compression 2-Bits compression method Horizontal compression Vertical compression DNA Sequence Horizontal compression Vertical compression binary representation compression algorithms online databases

来源：评论

学校读者我要写书评

暂无评论

Hybrid-Smash: A Heterogeneous CPU-GPU compression Library

引用

IEEE ACCESS 2024年 12卷 32706-32723页

作者： Penaranda, Cristian Reano, Carlos Silla, Federico Univ Politecn Valencia DISCA Valencia 46022 Spain Univ Valencia Dept Informat ETSE Burjassot 46100 Spain

compression algorithms are widely used to reduce data size and improve application performance. Nevertheless, data compression has a computational cost which can limit its use. GPUs could be leveraged to reduce compression time. However, existing GPU-based compression libraries expect data to compress in GPU memory, although it is usually stored in CPU memory. Additionally, setup time of GPUs could be a problem when compressing small data sizes. In this paper, we implement a new GPU-based compression library. Contrary to existing ones, our library uses data located in CPU memory. Performance results show that, for the same compression algorithms, GPUs are beneficial for larger data sizes whereas smaller data sizes are compressed faster using CPUs. Therefore, we enhance our proposal with Hybrid-Smash: a heterogeneous CPU-GPU compression library, which transparently uses CPU or GPU compression depending on data size, thus improving compression for any data size.

关键词： Graphics processing units Image coding compression algorithms Symbols Proposals Artificial intelligence Parallel processing Performance evaluation Central Processing Unit Hybrid intelligent systems Data systems Size control Lossless compression parallel computing GPU

来源：评论

学校读者我要写书评

暂无评论

Universal Graph compression: Stochastic Block Models

引用

IEEE TRANSACTIONS ON INFORMATION THEORY 2024年第2期70卷 1473-1497页

作者： Bhatt, Alankrita Wang, Ziao Wang, Chi Wang, Lele Calif Inst Technol Caltech Pasadena CA 91125 USA Univ British Columbia Dept Elect & Comp Engn Vancouver BC V6T 1Z4 Canada Microsoft Res Redmond WA 98052 USA

Motivated by the prevalent data science applications of processing large-scale graph data such as social networks and biological networks, this paper investigates lossless compression of data in the form of a labeled graph. Particularly, we consider a widely used random graph model, stochastic block model (SBM), which captures the clustering effects in social networks. An information-theoretic universal compression framework is applied, in which one aims to design a single compressor that achieves the asymptotically optimal compression rate, for every SBM distribution, without knowing the parameters of the SBM. Such a graph compressor is proposed in this paper, which universally achieves the optimal compression rate with polynomial time complexity for a wide class of SBMs. Existing universal compression techniques are developed mostly for stationary ergodic one-dimensional sequences. However, the adjacency matrix of SBM has complex two-dimensional correlations. The challenge is alleviated through a carefully designed transform that converts two-dimensional correlated data into almost i.i.d. submatrices. The sequence of submatrices is then compressed by a Krichevsky-Trofimov compressor, whose length analysis is generalized to identically distributed but arbitrarily correlated sequences. In four benchmark graph datasets, the compressed files from competing algorithms take 2.4 to 27 times the space needed by the proposed scheme.

关键词： compression algorithms source coding graph theory

来源：评论

学校读者我要写书评

暂无评论

Multi-Component Temporal-Correlation Seismic Data compression Algorithm Based on the PCA and DWT

引用

algorithms 2025年第1期18卷 33-33页

作者： de Lucena, Mateus Martinez Ribeiro, Josafat Leal Wagner, Matheus Frohlich, Antonio Augusto Univ Fed Santa Catarina Software Hardware Integrat Lab BR-88040900 Florianopolis Brazil UFSC CTC LISHA POB 476 BR-88040900 Florianopolis Brazil

Industrial application data acquisition systems can be sources of vast amounts of data. The seismic surveys conducted by oil and gas companies result in enormous datasets, often exceeding terabytes of data. The storage and communication demands these data require can only be achieved through compression. Careful consideration must be given to minimize the reconstruction error of compressed data caused by lossy compression. This paper investigates the combination of principal component analysis (PCA), discrete wavelet transform (DWT), thresholding, quantization, and entropy encoding to compress such datasets. The proposed method is a lossy compression algorithm tuned by evaluating the reconstruction error in frequency ranges of interest, namely 0-20 Hz and 15-65 Hz. The PCA compression and decompression acts as a noise filter while the DWT drives the compression. The proposed method can be tuned through threshold and quantization percentages and the number of principal components to achieve compression rates of up to 31:1 with reconstruction residues energy of less than 4% in the frequency ranges of 0-20 Hz, 15-65 Hz, and 60-105 Hz.

关键词： compression algorithms principal component analysis discrete wavelet transform

来源：评论

学校读者我要写书评

暂无评论

Efficient and Adaptive CUR Matrix Decomposition for Flexible compression of Network Monitoring Data

引用

IEEE TRANSACTIONS ON NETWORK SCIENCE AND ENGINEERING 2025年第3期12卷 2231-2242页

作者： Wen, Jigang Wang, Shiqin Xie, Kun Tian, Jiazheng Wang, Yixuan Hunan Univ Sci & Technol Sch Comp Sci & Engn Xiangtan 411201 Peoples R China Hunan Univ Coll Comp Sci & Elect Engn Changsha 410012 Peoples R China Hunan Univ Key Lab Convergence Comp Supercomp & Artificial In Minist Educ Changsha 410002 Peoples R China

Network-wide monitoring is indispensable for a variety of network applications. However, as network sizes increase and the demand for fine-grained, continuous measurements grows, the challenges associated with storing and transmitting such data intensify. Recent studies have shown that network-wide monitoring data exhibits a low-rank structure, which can be exploited using matrix decomposition techniques for compression. This paper presents a compression algorithm for low-rank matrices based on CUR decomposition, which offers enhanced interpretability compared to SVD-based compression. Existing CUR solutions, however, lack the capability for fast and flexible compression that can dynamically adjust to matrix size requirements while preserving maximal approximation accuracy. We address the challenges associated with CUR row and column selection by formulating it as a deterministic CUR matrix decomposition problem, involving a selection matrix $\mathbf{W}$. To achieve rapid compression, we propose an algorithm that effectively accelerates the process of solving for the parameter matrix $\mathbf{W}$. Our approach reveals that the vectors in $\mathbf{W}$ indicate the importance of each row and column in forming the respective row and column subspaces. Leveraging this insight, we develop a flexible compression algorithm based on the sorted vectors in the selection matrix $\mathbf{W}$. This method not only ensures the required compression ratio but also maintains maximal approximation accuracy. Extensive experiments on both synthesized and real data demonstrate that our algorithm can deliver fast and precise matrix compression, aligning with the desired compression ratio.

关键词： Matrix decomposition Monitoring Vectors Approximation algorithms Accuracy Sparse matrices Frequency measurement compression algorithms Electronic mail Computational complexity Matrix compression matrix decomposition CUR decomposition

来源：评论

学校读者我要写书评

暂无评论

AIS-Based Vessel Trajectory compression: A Systematic Review and Software Development

IEEE OPEN JOURNAL OF VEHICULAR TECHNOLOGY

引用

IEEE OPEN JOURNAL OF VEHICULAR TECHNOLOGY 2024年 5卷 1193-1214页

作者： Liu, Ryan Wen Zhou, Shiqi Yin, Shangkun Shu, Yaqing Liang, Maohan Wuhan Univ Technol Sch Nav Hubei Key Lab Inland Shipping Technol Wuhan 430063 Peoples R China Wuhan Univ Technol Hainan Inst Sanya 572000 Peoples R China Wuhan Univ Technol State Key Lab Maritime Technol & Safety Wuhan 430063 Peoples R China Natl Univ Singapore Dept Civil & Environm Engn Singapore 117576 Singapore

With the advancement of satellite and 5G communication technologies, vehicles can transmit and exchange data from anywhere in the world. It has resulted in the generation of massive spatial trajectories, particularly from the Automatic Identification System (AIS) for surface vehicles. The massive AIS data lead to high storage requirements and computing costs, as well as low data transmission efficiency. These challenges highlight the critical importance of vessel trajectory compression for surface vehicles. However, the complexity and diversity of vessel trajectories and behaviors make trajectory compression imperative and challenging in maritime applications. Therefore, trajectory compression has been one of the hot spots in research on trajectory data mining. The major purpose of this work is to provide a comprehensive reference source for beginners involved in vessel trajectory compression. The current trajectory compression methods could be broadly divided into two types, batch (offline) and online modes. The principles and pseudo-codes of these methods will be provided and discussed in detail. In addition, compressive experiments on several publicly available data sets have been implemented to evaluate the batch and online compression methods in terms of computation time, compression ratio, trajectory similarity, and trajectory length loss rate. Finally, we develop a flexible and open software, called AISCompress, for AIS-based batch and online vessel trajectory compression. The conclusions and associated future works are also given to inspire future applications in vessel trajectory compression.

关键词： Trajectory Artificial intelligence Safety Software Real-time systems compression algorithms Approximation algorithms Automatic identification system (AIS) vessel trajectory trajectory compression error metrics similarity measure

来源：评论

学校读者我要写书评

暂无评论

Lossy compression: An Online Multi-Stage Technology for High-Fidelity Synchro- Waveform Measurements

引用

IEEE TRANSACTIONS ON INDUSTRY APPLICATIONS 2025年第3期61卷 4290-4300页

作者： Qiu, Wei Yin, He Wu, Yuru Dong, Yuqing Zheng, Yao Yao, Wenxuan Liu, Yilu Hunan Univ Dept Elect Engn & Comp Sci Changsha 410082 Peoples R China Univ Tennessee Dept Elect Engn & Comp Sci Knoxville TN 37996 USA Oak Ridge Natl Lab Oak Ridge TN 37831 USA

Effective real-time monitoring and analysis of distributed grids necessitate the use of synchro-waveform measurements, which capture almost all high-frequency disturbances and transient phenomena. However, due to limitations in high-speed measurements and network bandwidth, it is challenging to transfer all high-fidelity synchro-waveforms losslessly and successfully. To cope with these challenges, a hybrid-based online multi-stage compression algorithm is proposed to significantly improve the compression efficiency for synchro-waveform measurements. Initially, the multiple discrete Wavelet transformation is deployed to deconstruct the waveform components. The delta encoding is further developed to decrease the magnitude. In conjunction with the Lempel-Ziv-Markov chain, the hybrid compression algorithm is implemented to achieve real-time compression for the synchro-waveform measurements. Moreover, an innovative error index that synergizes the time and frequency domain error and correlation is formulated to evaluate the waveform distortion. By integrating compression ratio, suitable parameters can be optimally selected. Finally, the simulation, laboratory experiments, as well as field tests across a spectrum of sampling frequencies and time intervals are conducted to substantiate the efficacy of the proposed method. The outcomes demonstrated that a compression ratio of approximately 15.5 and 17.83 can be reached for 0.5 s and 1 s data under both offline and online scenarios, which equates to a substantial 93.5% to 94.39% reduction in data storage requirements.

关键词： Encoding Discrete wavelet transforms Loss measurement compression algorithms Voltage measurement Real-time systems Image coding Extraterrestrial measurements Time measurement Noise Hybrid compression algorithm High-fidelity synchro-waveform measurements smart grid monitoring compression ratio

来源：评论

学校读者我要写书评

暂无评论

Efficient compression for Remote Sensing: Multispectral Transform and Deep Recurrent Neural Networks for Lossless Hyper-Spectral Imagine

引用

INTERNATIONAL JOURNAL OF ADVANCED COMPUTER SCIENCE AND APPLICATIONS 2024年第2期15卷 531-541页

作者： Anuradha, D. Sekhar, Gillala Chandra Mishra, Annapurna Thapar, Puneet El-Ebiary, Yousef A. Baker Syamala, Maganti Panimalar Engn Coll Dept CSBS Chennai Tamil Nadu India Inst Aeronaut Engn Dept Comp Sci & Engn Hyderabad 500043 India Silicon Inst Technol Dept Elect & Commun Engn Bhubaneswar 751024 India Lovely Profess Univ Comp Sci & Engn Dept Phagwara Punjab India UniSZA Univ Fac Informat & Comp Kuala Terengganu Malaysia Koneru Lakshmaiah Educ Fdn Dept Comp Sci & Engn Vaddeswaram 522302 Andhra Pradesh India

Remote sensing technologies, which are essential for everything from environmental monitoring to disaster relief, enable large-scale multispectral data collection. In the field of hyper-spectral imaging, where high-dimensional data is required for precise analysis, effective compression techniques are critical for transmission and storage. In the field of hyper-spectral imaging, the development of efficient compression techniques is critical because datasets containing high-dimensional information must be transmitted and stored efficiently without sacrificing analytical precision. The paper presents advanced compression techniques that combine deep Recurrent Neural Networks (RNNs) with multispectral transforms to achieve lossless compression in hyper-spectral imaging. The Discrete Wavelet Transform (DWT) is used to efficiently capture spectral and spatial information by utilizing the properties of multispectral transforms. Simultaneously, deep RNNs are used to model the hyper-spectral data with complex dependencies, allowing for sequential compression. The overall compression efficiency that is increased by the integration of spatial and spectral information allows for reduced storage requirements and improved transmission efficiency. Python software is used to implement the proposed model. When compared to Liner Spectral Mixture Analysis (LSMA) based compression, Spatial Orientation Tree Wavelet (STW)-Wavelet Difference Reduction (WDR), and DPCM, the proposed DWT-RNN-LSTM method has a better PSNR value of 45 dB and a lower MSE of 7.50%. Adaptive compression methods are presented in order to dynamically adapt to various data properties and ensure application in various hyperspectral scenes. Studies on hyper-spectral images of various sizes and resolutions demonstrate the approach's scalability and generalization, as well as the utility and adaptability of the proposed compression framework in a variety of remote sensing scenarios.

关键词： Multi-Spectral transform lossless compression hyper-spectral data deep recurrent neural network compression algorithms

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：