Deforming Autoencoders: Unsupervised Disentangling of Shape and Appearance

Zhixin Shu (1) , Mihir Sahasrabudhe (2, 3) , Riza Alp Güler (2, 3) , Dimitris Samaras (1) , Nikos Paragios (2, 3) , Iasonas Kokkinos (4)

1 Department of Computer Science [Stonybrook - NY]
2 CVN - Centre de vision numérique
3 GALEN-POST - Organ Modeling through Extraction, Representation and Understanding of Medical Image Content
4 FAIR - Facebook AI Research [Paris]

In this work we introduce Deforming Autoencoders, a generative model for images that disentangles shape from appearance in an unsupervised manner. As in the deformable template paradigm, shape is represented as a deformation between a canonical coordinate system ('template') and an observed image, while appearance is modeled in 'canonical', template, coordinates, thus discarding variability due to deformations. We introduce novel techniques that allow this approach to be deployed in the setting of autoencoders and show that this method can be used for unsupervised group-wise image alignment. We show experiments with expression morphing in humans, hands, and digits, face manipulation, such as shape and appearance interpolation, as well as unsupervised landmark local-ization. A more powerful form of unsupervised disentangling becomes possible in template coordinates, allowing us to successfully decompose face images into shading and albedo, and further manipulate face images. Latent Representation Input Image Generated Deformation Generated Texture Decoder Decoder Spatial Warping Reconstructed Image Encoder Fig. 1. Deforming Autoencoders follow the deformable template paradigm and model image generation through a cascade of appearance (or, 'texture') synthesis in a canonical coordinate system and a spatial deformation that warps the texture to the observed image coordinates. By keeping the latent vector for texture short the network is forced to model shape variability through the deformation branch, so as to minimize a reconstruction loss. This allows us to train a deep gen-erative image model that disentangles shape and appearance in an entirely unsupervised manner.

Domaines: Vision par ordinateur et reconnaissance de formes [cs.CV]; Imagerie médicale; Apprentissage [cs.LG]; Traitement des images [eess.IV]; Traitement du signal et de l'image [eess.SP]; Intelligence artificielle [cs.AI]

Zhixin Shu, Mihir Sahasrabudhe, Riza Alp Güler, Dimitris Samaras, Nikos Paragios, et al.. Deforming Autoencoders: Unsupervised Disentangling of Shape and Appearance. The European Conference on Computer Vision, Sep 2018, Munich, Germany. ⟨hal-01935596⟩