InstanceGaussian: Appearance-Semantic Joint Gaussian Representation for 3D Instance-Level Perception

作     者：Li, Haijie Wu, Yanmin Meng, Jiarui Gao, Qiankun Zhang, Zhiyao Wang, Ronggang Zhang, Jian 

作者机构：School of Electronic and Computer Engineering Peking University China Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology Shenzhen Graduate School Peking University China College of Information Science and Engineering Northeastern University China 

出 版 物：《arXiv》 (arXiv)

年 卷 期：2024年

核心收录：

主　　题：Semantic Segmentation 

摘      要：3D scene understanding is vital for applications in autonomous driving, robotics, and augmented reality. However, scene understanding based on 3D Gaussian Splatting faces three key challenges: (i) an imbalance between appearance and semantics, (ii) inconsistencies in object boundaries, and (iii) difficulties with top-down instance segmentation. To address these challenges, we propose InstanceGaussian, a method that jointly learns appearance and semantic features while adaptively aggregating instances. Our contributions are as follows: (i) a new Semantic-Scaffold-GS representation to improve feature representation and boundary delineation, (ii) a progressive training strategy for enhanced stability and segmentation, and (iii) a category-agnostic, bottom-up instance aggregation approach for better segmentation. Experimental results demonstrate that our approach achieves state-of-the-art performance in category-agnostic, open-vocabulary 3D point-level segmentation, validating the effectiveness of our proposed method. Project page: https://***/InstanceGaussian/ Copyright © 2024, The Authors. All rights reserved.