Nguyen Phuc D. A. ; Ngo Tuan Duc ; Kalogerakis Evangelos ; Gan Chuang ; Tran Anh ; Pham Cuong ; Nguyen Khoi

Abstract

We introduce Open3DIS, a novel solution designed to tackle the problem ofOpen-Vocabulary Instance Segmentation within 3D scenes. Objects within 3Denvironments exhibit diverse shapes, scales, and colors, making preciseinstance-level identification a challenging task. Recent advancements inOpen-Vocabulary scene understanding have made significant strides in this areaby employing class-agnostic 3D instance proposal networks for objectlocalization and learning queryable features for each 3D mask. While thesemethods produce high-quality instance proposals, they struggle with identifyingsmall-scale and geometrically ambiguous objects. The key idea of our method isa new module that aggregates 2D instance masks across frames and maps them togeometrically coherent point cloud regions as high-quality object proposalsaddressing the above limitations. These are then combined with 3Dclass-agnostic instance proposals to include a wide range of objects in thereal world. To validate our approach, we conducted experiments on threeprominent datasets, including ScanNet200, S3DIS, and Replica, demonstratingsignificant performance gains in segmenting objects with diverse categoriesover the state-of-the-art approaches.