Tse Tze Ho Elden ; Kim Kwang In ; Leonardis Ales ; Chang Hyung Jin

Abstract

Estimating the pose and shape of hands and objects under interaction findsnumerous applications including augmented and virtual reality. Existingapproaches for hand and object reconstruction require explicitly definedphysical constraints and known objects, which limits its application domains.Our algorithm is agnostic to object models, and it learns the physical rulesgoverning hand-object interaction. This requires automatically inferring theshapes and physical interaction of hands and (potentially unknown) objects. Weseek to approach this challenging problem by proposing a collaborative learningstrategy where two-branches of deep networks are learning from each other.Specifically, we transfer hand mesh information to the object branch and viceversa for the hand branch. The resulting optimisation (training) problem can beunstable, and we address this via two strategies: (i) attention-guided graphconvolution which helps identify and focus on mutual occlusion and (ii)unsupervised associative loss which facilitates the transfer of informationbetween the branches. Experiments using four widely-used benchmarks show thatour framework achieves beyond state-of-the-art accuracy in 3D pose estimation,as well as recovers dense 3D hand and object shapes. Each technical componentabove contributes meaningfully in the ablation study.