You Yurong ; Wang Yan ; Chao Wei-Lun ; Garg Divyansh ; Pleiss Geoff ; Hariharan Bharath ; Campbell Mark ; Weinberger Kilian Q.

Abstract

Detecting objects such as cars and pedestrians in 3D plays an indispensablerole in autonomous driving. Existing approaches largely rely on expensive LiDARsensors for accurate depth information. While recently pseudo-LiDAR has beenintroduced as a promising alternative, at a much lower cost based solely onstereo images, there is still a notable performance gap. In this paper weprovide substantial advances to the pseudo-LiDAR framework through improvementsin stereo depth estimation. Concretely, we adapt the stereo networkarchitecture and loss function to be more aligned with accurate depthestimation of faraway objects --- currently the primary weakness ofpseudo-LiDAR. Further, we explore the idea to leverage cheaper but extremelysparse LiDAR sensors, which alone provide insufficient information for 3Ddetection, to de-bias our depth estimation. We propose a depth-propagationalgorithm, guided by the initial depth estimates, to diffuse these few exactmeasurements across the entire depth map. We show on the KITTI object detectionbenchmark that our combined approach yields substantial improvements in depthestimation and stereo-based 3D object detection --- outperforming the previousstate-of-the-art detection accuracy for faraway objects by 40%. Our code isavailable at https://github.com/mileyan/Pseudo_Lidar_V2.