Cermelli Fabio ; Cord Matthieu ; Douillard Arthur

Abstract

Continual learning for segmentation has recently seen increasing interest.However, all previous works focus on narrow semantic segmentation and disregardpanoptic segmentation, an important task with real-world impacts. %a In thispaper, we present the first continual learning model capable of operating onboth semantic and panoptic segmentation. Inspired by recent transformerapproaches that consider segmentation as a mask-classification problem, wedesign CoMFormer. Our method carefully exploits the properties of transformerarchitectures to learn new classes over time. Specifically, we propose a noveladaptive distillation loss along with a mask-based pseudo-labeling technique toeffectively prevent forgetting. To evaluate our approach, we introduce a novelcontinual panoptic segmentation benchmark on the challenging ADE20K dataset.Our CoMFormer outperforms all the existing baselines by forgetting less oldclasses but also learning more effectively new classes. In addition, we alsoreport an extensive evaluation in the large-scale continual semanticsegmentation scenario showing that CoMFormer also significantly outperformsstate-of-the-art methods.