Unal Ozan ; Dai Dengxin ; Hoyer Lukas ; Can Yigit Baran ; Van Gool Luc

Abstract

As 3D perception problems grow in popularity and the need for large-scalelabeled datasets for LiDAR semantic segmentation increase, new methods arisethat aim to reduce the necessity for dense annotations by employingweakly-supervised training. However these methods continue to show weakboundary estimation and high false negative rates for small objects and distantsparse regions. We argue that such weaknesses can be compensated by using RGBimages which provide a denser representation of the scene. We propose animage-guidance network (IGNet) which builds upon the idea of distilling highlevel feature information from a domain adapted synthetically trained 2Dsemantic segmentation network. We further utilize a one-way contrastivelearning scheme alongside a novel mixing strategy called FOVMix, to combat thehorizontal field-of-view mismatch between the two sensors and enhance theeffects of image guidance. IGNet achieves state-of-the-art results forweakly-supervised LiDAR semantic segmentation on ScribbleKITTI, boasting up to98% relative performance to fully supervised training with only 8% labeledpoints, while introducing no additional annotation burden orcomputational/memory cost during inference. Furthermore, we show that ourcontributions also prove effective for semi-supervised training, where IGNetclaims state-of-the-art results on both ScribbleKITTI and SemanticKITTI.