Li Heyuan ; Wang Bo ; Cheng Yu ; Kankanhalli Mohan ; Tan Robby T.

Abstract

Sensitivity to severe occlusion and large view angles limits the usagescenarios of the existing monocular 3D dense face alignment methods. Thestate-of-the-art 3DMM-based method, directly regresses the model'scoefficients, underutilizing the low-level 2D spatial and semantic information,which can actually offer cues for face shape and orientation. In this work, wedemonstrate how modeling 3D facial geometry in image and model space jointlycan solve the occlusion and view angle problems. Instead of predicting thewhole face directly, we regress image space features in the visible facialregion by dense prediction first. Subsequently, we predict our model'scoefficients based on the regressed feature of the visible regions, leveragingthe prior knowledge of whole face geometry from the morphable models tocomplete the invisible regions. We further propose a fusion network thatcombines the advantages of both the image and model space predictions toachieve high robustness and accuracy in unconstrained scenarios. Thanks to theproposed fusion module, our method is robust not only to occlusion and largepitch and roll view angles, which is the benefit of our image space approach,but also to noise and large yaw angles, which is the benefit of our model spacemethod. Comprehensive evaluations demonstrate the superior performance of ourmethod compared with the state-of-the-art methods. On the 3D dense facealignment task, we achieve 3.80% NME on the AFLW2000-3D dataset, whichoutperforms the state-of-the-art method by 5.5%. Code is available athttps://github.com/lhyfst/DSFNet.