Zhang Renrui ; Guo Ziyu ; Fang Rongyao ; Zhao Bin ; Wang Dong ; Qiao Yu ; Li Hongsheng ; Gao Peng

Abstract

Masked Autoencoders (MAE) have shown great potentials in self-supervisedpre-training for language and 2D image transformers. However, it still remainsan open question on how to exploit masked autoencoding for learning 3Drepresentations of irregular point clouds. In this paper, we proposePoint-M2AE, a strong Multi-scale MAE pre-training framework for hierarchicalself-supervised learning of 3D point clouds. Unlike the standard transformer inMAE, we modify the encoder and decoder into pyramid architectures toprogressively model spatial geometries and capture both fine-grained andhigh-level semantics of 3D shapes. For the encoder that downsamples pointtokens by stages, we design a multi-scale masking strategy to generateconsistent visible regions across scales, and adopt a local spatialself-attention mechanism during fine-tuning to focus on neighboring patterns.By multi-scale token propagation, the lightweight decoder gradually upsamplespoint tokens with complementary skip connections from the encoder, whichfurther promotes the reconstruction from a global-to-local perspective.Extensive experiments demonstrate the state-of-the-art performance ofPoint-M2AE for 3D representation learning. With a frozen encoder afterpre-training, Point-M2AE achieves 92.9% accuracy for linear SVM on ModelNet40,even surpassing some fully trained methods. By fine-tuning on downstream tasks,Point-M2AE achieves 86.43% accuracy on ScanObjectNN, +3.36% to the second-best,and largely benefits the few-shot classification, part segmentation and 3Dobject detection with the hierarchical pre-training scheme. Code is availableat https://github.com/ZrrSkywalker/Point-M2AE.