Fan Hehe ; Yu Xin ; Ding Yuhang ; Yang Yi ; Kankanhalli Mohan

Abstract

Point cloud sequences are irregular and unordered in the spatial dimensionwhile exhibiting regularities and order in the temporal dimension. Therefore,existing grid based convolutions for conventional video processing cannot bedirectly applied to spatio-temporal modeling of raw point cloud sequences. Inthis paper, we propose a point spatio-temporal (PST) convolution to achieveinformative representations of point cloud sequences. The proposed PSTconvolution first disentangles space and time in point cloud sequences. Then, aspatial convolution is employed to capture the local structure of points in the3D space, and a temporal convolution is used to model the dynamics of thespatial regions along the time dimension. Furthermore, we incorporate theproposed PST convolution into a deep network, namely PSTNet, to extractfeatures of point cloud sequences in a hierarchical manner. Extensiveexperiments on widely-used 3D action recognition and 4D semantic segmentationdatasets demonstrate the effectiveness of PSTNet to model point cloudsequences.