Zheng Ruochen ; Hong Jiahao ; Gao Changxin ; Sang Nong

Abstract

The presence of noise in acquired data invariably leads to performancedegradation in cross-modal matching. Unfortunately, obtaining preciseannotations in the multimodal field is expensive, which has prompted somemethods to tackle the mismatched data pair issue in cross-modal matchingcontexts, termed as noisy correspondence. However, most of these existing noisycorrespondence methods exhibit the following limitations: a) the problem ofself-reinforcing error accumulation, and b) improper handling of noisy datapair. To tackle the two problems, we propose a generalized framework termed asRank corrElation and noisy Pair hAlf-replacing wIth memoRy (REPAIR), whichbenefits from maintaining a memory bank for features of matched pairs.Specifically, we calculate the distances between the features in the memorybank and those of the target pair for each respective modality, and use therank correlation of these two sets of distances to estimate the softcorrespondence label of the target pair. Estimating soft correspondence basedon memory bank features rather than using a similarity network can avoid theaccumulation of errors due to incorrect network identifications. For pairs thatare completely mismatched, REPAIR searches the memory bank for the mostmatching feature to replace one feature of one modality, instead of using theoriginal pair directly or merely discarding the mismatched pair. We conductexperiments on three cross-modal datasets, i.e., Flickr30K, MSCOCO, and CC152K,proving the effectiveness and robustness of our REPAIR on synthetic andreal-world noise.