Kontogianni Theodora ; Celikkan Ekin ; Tang Siyu ; Schindler Konrad

Abstract

We propose an interactive approach for 3D instance segmentation, where userscan iteratively collaborate with a deep learning model to segment objects in a3D point cloud directly. Current methods for 3D instance segmentation aregenerally trained in a fully-supervised fashion, which requires large amountsof costly training labels, and does not generalize well to classes unseenduring training. Few works have attempted to obtain 3D segmentation masks usinghuman interactions. Existing methods rely on user feedback in the 2D imagedomain. As a consequence, users are required to constantly switch between 2Dimages and 3D representations, and custom architectures are employed to combinemultiple input modalities. Therefore, integration with existing standard 3Dmodels is not straightforward. The core idea of this work is to enable users tointeract directly with 3D point clouds by clicking on desired 3D objects ofinterest~(or their background) to interactively segment the scene in anopen-world setting. Specifically, our method does not require training datafrom any target domain, and can adapt to new environments where no appropriatetraining sets are available. Our system continuously adjusts the objectsegmentation based on the user feedback and achieves accurate dense 3Dsegmentation masks with minimal human effort (few clicks per object). Besidesits potential for efficient labeling of large-scale and varied 3D datasets, ourapproach, where the user directly interacts with the 3D environment, enablesnew applications in AR/VR and human-robot interaction.