Abavisani Mahdi ; Patel Vishal M.

Abstract

We present convolutional neural network (CNN) based approaches forunsupervised multimodal subspace clustering. The proposed framework consists ofthree main stages - multimodal encoder, self-expressive layer, and multimodaldecoder. The encoder takes multimodal data as input and fuses them to a latentspace representation. The self-expressive layer is responsible for enforcingthe self-expressiveness property and acquiring an affinity matrix correspondingto the data points. The decoder reconstructs the original input data. Thenetwork uses the distance between the decoder's reconstruction and the originalinput in its training. We investigate early, late and intermediate fusiontechniques and propose three different encoders corresponding to them forspatial fusion. The self-expressive layers and multimodal decoders areessentially the same for different spatial fusion-based approaches. In additionto various spatial fusion-based methods, an affinity fusion-based network isalso proposed in which the self-expressive layer corresponding to differentmodalities is enforced to be the same. Extensive experiments on three datasetsshow that the proposed methods significantly outperform the state-of-the-artmultimodal subspace clustering methods.