Wang Jiahao ; Chen Guo ; Huang Yifei ; Wang Limin ; Lu Tong

Abstract

Most existing forecasting systems are memory-based methods, which attempt tomimic human forecasting ability by employing various memory mechanisms and haveprogressed in temporal modeling for memory dependency. Nevertheless, an obviousweakness of this paradigm is that it can only model limited historicaldependence and can not transcend the past. In this paper, we rethink thetemporal dependence of event evolution and propose a novelmemory-anticipation-based paradigm to model an entire temporal structure,including the past, present, and future. Based on this idea, we presentMemory-and-Anticipation Transformer (MAT), a memory-anticipation-basedapproach, to address the online action detection and anticipation tasks. Inaddition, owing to the inherent superiority of MAT, it can process onlineaction detection and anticipation tasks in a unified manner. The proposed MATmodel is tested on four challenging benchmarks TVSeries, THUMOS'14, HDD, andEPIC-Kitchens-100, for online action detection and anticipation tasks, and itsignificantly outperforms all existing methods. Code is available athttps://github.com/Echo0125/Memory-and-Anticipation-Transformer.