Michaelis Claudio ; Ustyuzhaninov Ivan ; Bethge Matthias ; Ecker Alexander S.

Abstract

We tackle the problem of one-shot instance segmentation: Given an exampleimage of a novel, previously unknown object category, find and segment allobjects of this category within a complex scene. To address this challengingnew task, we propose Siamese Mask R-CNN. It extends Mask R-CNN by a Siamesebackbone encoding both reference image and scene, allowing it to targetdetection and segmentation towards the reference category. We demonstrateempirical results on MS Coco highlighting challenges of the one-shot setting:while transferring knowledge about instance segmentation to novel objectcategories works very well, targeting the detection network towards thereference category appears to be more difficult. Our work provides a firststrong baseline for one-shot instance segmentation and will hopefully inspirefurther research into more powerful and flexible scene analysis algorithms.Code is available at: https://github.com/bethgelab/siamese-mask-rcnn