Learning Local Descriptor for Comparing Renders with Real Images

Pamir Ghimire (1) , Igor Jovančević (1) , Jean-José Orteu (2)

1 DIOTA
2 ICA - Institut Clément Ader

We present a method to train a deep-network-based feature descriptor to calculate discriminative local descriptions from renders and corresponding real images with similar geometry. We are interested in using such descriptors for automatic industrial visual inspection whereby the inspection camera has been coarsely localized with respect to a relatively large mechanical assembly and presence of certain components needs to be checked compared to the reference computer-aided design model (CAD). We aim to perform the task by comparing the real inspection image with the render of textureless 3D CAD using the learned descriptors. The descriptor was trained to capture geometric features while staying invariant to image domain. Patch pairs for training the descriptor were extracted in a semisupervised manner from a small data set of 100 pairs of real images and corresponding renders that were manually finely registered starting from a relatively coarse localization of the inspection camera. Due to the small size of the training data set, the descriptor network was initialized with weights from classification training on ImageNet. A two-step training is proposed for addressing the problem of domain adaptation. The first, “bootstrapping”, is a classification training to obtain good initial weights for second training step, triplet-loss training, that provides weights for extracting the discriminative features comparable using l2 distance. The descriptor was tested for comparing renders and real images through two approaches: finding local correspondences between the images through nearest neighbor matching and transforming the images into Bag of Visual Words (BoVW) histograms. We observed that learning a robust cross-domain descriptor is feasible, even with a small data set, and such features might be of interest for CAD-based inspection of mechanical assemblies, and related applications such as tracking or finely registered augmented reality. To the best of our knowledge, this is the first work that reports learning local descriptors for comparing renders with real inspection images.

Mots clés: deep convolutinal neural networks; learned feature descriptor; 3D CAD; nonphotorealistic rendering

Domaines: Sciences de l'ingénieur [physics]

Pamir Ghimire, Igor Jovančević, Jean-José Orteu. Learning Local Descriptor for Comparing Renders with Real Images. Applied Sciences, 2021, 11 (8), pp.1-15/3301. ⟨10.3390/app11083301⟩. ⟨hal-03197287⟩