Zhao Zihua ; Chen Mengxi ; Dai Tianjie ; Yao Jiangchao ; han Bo ; Zhang Ya ; Wang Yanfeng

Abstract

Noisy correspondence that refers to mismatches in cross-modal data pairs, isprevalent on human-annotated or web-crawled datasets. Prior approaches toleverage such data mainly consider the application of uni-modal noisy labellearning without amending the impact on both cross-modal and intra-modalgeometrical structures in multimodal learning. Actually, we find that bothstructures are effective to discriminate noisy correspondence throughstructural differences when being well-established. Inspired by thisobservation, we introduce a Geometrical Structure Consistency (GSC) method toinfer the true correspondence. Specifically, GSC ensures the preservation ofgeometrical structures within and between modalities, allowing for the accuratediscrimination of noisy samples based on structural differences. Utilizingthese inferred true correspondence labels, GSC refines the learning ofgeometrical structures by filtering out the noisy samples. Experiments acrossfour cross-modal datasets confirm that GSC effectively identifies noisy samplesand significantly outperforms the current leading methods.