Qi Charles R. ; Litany Or ; He Kaiming ; Guibas Leonidas J.

Abstract

Current 3D object detection methods are heavily influenced by 2D detectors.In order to leverage architectures in 2D detectors, they often convert 3D pointclouds to regular grids (i.e., to voxel grids or to bird's eye view images), orrely on detection in 2D images to propose 3D boxes. Few works have attempted todirectly detect objects in point clouds. In this work, we return to firstprinciples to construct a 3D detection pipeline for point cloud data and asgeneric as possible. However, due to the sparse nature of the data -- samplesfrom 2D manifolds in 3D space -- we face a major challenge when directlypredicting bounding box parameters from scene points: a 3D object centroid canbe far from any surface point thus hard to regress accurately in one step. Toaddress the challenge, we propose VoteNet, an end-to-end 3D object detectionnetwork based on a synergy of deep point set networks and Hough voting. Ourmodel achieves state-of-the-art 3D detection on two large datasets of real 3Dscans, ScanNet and SUN RGB-D with a simple design, compact model size and highefficiency. Remarkably, VoteNet outperforms previous methods by using purelygeometric information without relying on color images.