Pu Tao ; Chen Tianshui ; Wu Hefeng ; Shi Yukai ; Yang Zhijing ; Lin Liang

Abstract

Despite achieving impressive progress, current multi-label image recognition(MLR) algorithms heavily depend on large-scale datasets with complete labels,making collecting large-scale datasets extremely time-consuming andlabor-intensive. Training the multi-label image recognition models with partiallabels (MLR-PL) is an alternative way, in which merely some labels are knownwhile others are unknown for each image. However, current MLP-PL algorithmsrely on pre-trained image similarity models or iteratively updating the imageclassification models to generate pseudo labels for the unknown labels. Thus,they depend on a certain amount of annotations and inevitably suffer fromobvious performance drops, especially when the known label proportion is low.To address this dilemma, we propose a dual-perspective semantic-awarerepresentation blending (DSRB) that blends multi-granularity category-specificsemantic representation across different images, from instance and prototypeperspective respectively, to transfer information of known labels to complementunknown labels. Specifically, an instance-perspective representation blending(IPRB) module is designed to blend the representations of the known labels inan image with the representations of the corresponding unknown labels inanother image to complement these unknown labels. Meanwhile, aprototype-perspective representation blending (PPRB) module is introduced tolearn more stable representation prototypes for each category and blends therepresentation of unknown labels with the prototypes of corresponding labels,in a location-sensitive manner, to complement these unknown labels. Extensiveexperiments on the MS-COCO, Visual Genome, and Pascal VOC 2007 datasets showthat the proposed DSRB consistently outperforms current state-of-the-artalgorithms on all known label proportion settings.