Fast and scalable data transfer is crucial in today's decentralized data ecosystems and data-driven applications. Example use cases include transferring data from operational systems to consolidated data warehouse...
详细信息
Fast and scalable data transfer is crucial in today's decentralized data ecosystems and data-driven applications. Example use cases include transferring data from operational systems to consolidated data warehouse environments, or from relational database systems to data lakes for exploratory data analysis or ML model training. Traditional data transfer approaches rely on efficient point-to-point connectors or general middleware with generic intermediate data representations. Physical environments (e.g., on-premise, cloud, or consumer nodes) also have become increasingly heterogeneous. Existing work still struggles to achieve both, fast and scalable data transfer as well as generality in terms of heterogeneous systems and environments. Hence, in this paper, we introduce a holistic data transfer framework. Our XDBC framework splits the data transfer pipeline into logical components and provides a wide variety of physical implementations for these components. This design allows a seamless integration of different systems as well as the automatic optimizations of data transfer configurations according to workload and environment characteristics. Our evaluation shows that XDBC outperforms state-of-the-art generic data transfer tools by up to 5x, while being on par with specialized approaches.
The RAPIDS Memory Manager (RMM) is developed by NVIDIA as a package that would enable developers to customize GPU memory allocation. RMM enables the use of pool allocation which could improve the performance greatly. ...
详细信息
The RAPIDS Memory Manager (RMM) is developed by NVIDIA as a package that would enable developers to customize GPU memory allocation. RMM enables the use of pool allocation which could improve the performance greatly. This paper proposes a systematic profiling and evaluation framework that leverages NVIDIA's RMM to optimize and understand dataloading performance of the ***_csv operation in GPU accelerated environments. It examines RMM's impact from multiple aspects, by measuring the execution time required to complete the operation, the memory consumption effect, and by profiling the operation with and without utilizing RMM across various dataset sizes. The finding demonstrates that RMM can have significant speedup of up to 24% by improving the memory management strategy of cuDF. As for other time series data preprocessing operations were overall improved by 14% when utilizing RMM. It could also improve the scalability of cuDF by utilizing managed memory to overcome the limited GPU memory constrains, allowing cuDF to handle datasets that exceeds the GPU memory while maintaining similar to 10x faster execution than the CPU based Pandas dataFrame. The effect of RMM on GPU memory consumption is also highlighted indicating a trade-off between faster execution and increased memory consumption.
暂无评论