Cha Sungmin ; Kim Beomyoung ; Yoo Youngjoon ; Moon Taesup

Abstract

This paper introduces a solid state-of-the-art baseline for aclass-incremental semantic segmentation (CISS) problem. While the recent CISSalgorithms utilize variants of the knowledge distillation (KD) technique totackle the problem, they failed to fully address the critical challenges inCISS causing the catastrophic forgetting; the semantic drift of the backgroundclass and the multi-label prediction issue. To better address these challenges,we propose a new method, dubbed SSUL-M (Semantic Segmentation with UnknownLabel with Memory), by carefully combining techniques tailored for semanticsegmentation. Specifically, we claim three main contributions. (1) definingunknown classes within the background class to help to learn future classes(help plasticity), (2) freezing backbone network and past classifiers withbinary cross-entropy loss and pseudo-labeling to overcome catastrophicforgetting (help stability), and (3) utilizing tiny exemplar memory for thefirst time in CISS to improve both plasticity and stability. The extensivelyconducted experiments show the effectiveness of our method, achievingsignificantly better performance than the recent state-of-the-art baselines onthe standard benchmark datasets. Furthermore, we justify our contributions withthorough ablation analyses and discuss different natures of the CISS problemcompared to the traditional class-incremental learning targetingclassification. The official code is available athttps://github.com/clovaai/SSUL.