RT3DHVC: A Real-time Human Holographic Video Conferencing System with a Consumer RGB-D Camera Array

作     者：Jiang, Yuqi Li, Jing Dai, Yanran Qin, Haidong Zhou, Xiaoshi Zhang, Yong Liu, Hongwei Yan, Kefan Yang, Tao 

作者机构：The School of Telecommuni cations Engineering Xidian University Xi’an710071 China The National Engineering Laboratory for Integrated AeroSpace-Ground-Ocean Big Data Application Technology SAIIP School of Computer Science Northwestern Polytechnical University Xi’an710129 China 

出 版 物：《IEEE Transactions on Circuits and Systems for Video Technology》 (IEEE Trans Circuits Syst Video Technol)

年 卷 期：2024年第35卷第5期

页      面：4226-4241页

核心收录：

学科分类：0810[工学-信息与通信工程] 0808[工学-电气工程] 081203[工学-计算机应用技术] 08[工学] 0805[工学-材料科学与工程（可授工学、理学学位）] 0816[工学-测绘科学与技术] 0835[工学-软件工程] 0813[工学-建筑学] 0803[工学-光学工程] 0814[工学-土木工程] 0812[工学-计算机科学与技术（可授工学、理学学位）] 

基　　金：National Natural Science Foundation of China Key Research and Development Projects of Shaanxi Province Huawei China 

主　　题：Video conferencing 

摘      要：In this paper, we present an end-to-end holographic video conferencing system that enables real-time high-quality free-viewpoint rendering of participants in different spatial regions, placing them in a unified virtual space for a more immersive display. Our system offers a cost-effective, complete holographic conferencing process, including multiview 3D data capture, RGB-D stream compression and transmission, high-quality rendering, and immersive display. It employs a sparse set of commodity RGB-D cameras that capture 3D geometric and textural information. We then remotely transmit color and depth maps via standard video encoding and transmission protocols. We propose a GPU-parallelized rendering pipeline based on an image-based virtual view synthesis algorithm to achieve real-time and high-quality scene rendering. This algorithm uses an on-the-fly Truncated Signed Distance Function (TSDF) approach, which marches along virtual rays within a computed precise search interval to determine surface intersections. We then design a multiweight projective texture mapping method to fuse color information from multiple views. Furthermore, we introduce a method that uses a depth confidence map to weight the rendering results from different views, which mitigates the impact of sensor noise and inaccurate measurements on the rendering results. Finally, our system places conference participants from different spaces into a virtual conference environment with a global coordinate system through coordinate transformation, which simulates a real conference scene in physical space, providing an immersive remote conferencing experience. Experimental evaluations confirm our system’s real-time, low-latency, high-quality, and immersive capabilities. ©1991-2012 IEEE.