We present a simple and general framework for feature learning from pointclouds. The key to the success of CNNs is the convolution operator that iscapable of leveraging spatially-local correlation in data represented denselyin grids (e.g. images). However, point clouds are irregular and unordered, thusdirectly convolving kernels against features associated with the points, willresult in desertion of shape information and variance to point ordering. Toaddress these problems, we propose to learn an $\mathcal{X}$X-transformationfrom the input points, to simultaneously promote two causes. The first is theweighting of the input features associated with the points, and the second isthe permutation of the points into a latent and potentially canonical order.Element-wise product and sum operations of the typical convolution operator aresubsequently applied on the $\mathcal{X}$X-transformed features. The proposedmethod is a generalization of typical CNNs to feature learning from pointclouds, thus we call it PointCNN. Experiments show that PointCNN achieves onpar or better performance than state-of-the-art methods on multiple challengingbenchmark datasets and tasks.