Rao Yongming ; Chen Guangyi ; Lu Jiwen ; Zhou Jie

Abstract

Attention mechanism has demonstrated great potential in fine-grained visualrecognition tasks. In this paper, we present a counterfactual attentionlearning method to learn more effective attention based on causal inference.Unlike most existing methods that learn visual attention based on conventionallikelihood, we propose to learn the attention with counterfactual causality,which provides a tool to measure the attention quality and a powerfulsupervisory signal to guide the learning process. Specifically, we analyze theeffect of the learned visual attention on network prediction throughcounterfactual intervention and maximize the effect to encourage the network tolearn more useful attention for fine-grained image recognition. Empirically, weevaluate our method on a wide range of fine-grained recognition tasks whereattention plays a crucial role, including fine-grained image categorization,person re-identification, and vehicle re-identification. The consistentimprovement on all benchmarks demonstrates the effectiveness of our method.Code is available at https://github.com/raoyongming/CAL