Hasson Yana ; Varol Gül ; Laptev Ivan ; Schmid Cordelia

Abstract

Our work aims to obtain 3D reconstruction of hands and manipulated objectsfrom monocular videos. Reconstructing hand-object manipulations holds a greatpotential for robotics and learning from human demonstrations. The supervisedlearning approach to this problem, however, requires 3D supervision and remainslimited to constrained laboratory settings and simulators for which 3D groundtruth is available. In this paper we first propose a learning-free fittingapproach for hand-object reconstruction which can seamlessly handle two-handobject interactions. Our method relies on cues obtained with common methods forobject detection, hand pose estimation and instance segmentation. Wequantitatively evaluate our approach and show that it can be applied todatasets with varying levels of difficulty for which training data isunavailable.