Model-based Clustering with Missing Not At Random Data

Aude Sportisse (1, 2) , Christophe Biernacki (3, 4, 5) , Claire Boyer (6, 7, 8) , Julie Josse (9, 10) , Matthieu Marbac Lourdelle (11, 12, 4, 13) , Gilles Celeux (14, 15) , Fabien Laporte (16)

1 UniCA - Université Côte d'Azur
2 MAASAI - Modèles et algorithmes pour l’intelligence artificielle
3 Université Lille Nord (France)
4 CNRS - Centre National de la Recherche Scientifique
5 MODAL - MOdel for Data Analysis and Learning
6 SU - Sorbonne Université
7 LPSM (UMR_8001) - Laboratoire de Probabilités, Statistique et Modélisation
8 MOKAPLAN - Méthodes numériques pour le problème de Monge-Kantorovich et Applications en sciences sociales
9 CRISAM - Inria Sophia Antipolis - Méditerranée
10 IDESP - Institut Desbrest de santé publique
11 UR - Université de Rennes
12 ENSAI - Ecole Nationale de la Statistique et de l'Analyse de l'Information [Bruz]
13 CREST - Centre de Recherche en Economie et Statistique [Bruz]
14 Inria Saclay - Ile de France
15 Université Paris-Saclay
16 UCO - Université Catholique de l'Ouest

In recent decades, technological advances have made it possible to collect large data sets. In this context, the model-based clustering is a very popular, flexible and interpretable methodology for data exploration in a well-defined statistical framework. One of the ironies of the increase of large datasets is that missing values are more frequent. However, traditional ways (as discarding observations with missing values or imputation methods) are not designed for the clustering purpose. In addition, they rarely apply to the general case, though frequent in practice, of Missing Not At Random (MNAR) values, i.e. when the missingness depends on the unobserved data values and possibly on the observed data values. The goal of this paper is to propose a novel approach by embedding MNAR data directly within model-based clustering algorithms. We introduce a selection model for the joint distribution of data and missing-data indicator. It corresponds to a mixture model for the data distribution and a general MNAR model for the missing-data mechanism, which may depend on the underlying classes (unknown) and/or the values of the missing variables themselves. A large set of meaningful MNAR sub-models is derived and the identifiability of the parameters is studied for each of the sub-models, which is usually a key issue for any MNAR proposals. The EM and Stochastic EM algorithms are considered for estimation. Finally, we perform empirical evaluations for the proposed submodels on synthetic data and we illustrate the relevance of our method on a medical register, the TraumaBase ® dataset.

Mots clés: Model-based Clustering; Missing Not At Random (MNAR) Data; Identifiability; EM and Stochastic EM Algorithms; Medical Data

Domaines: Machine Learning [stat.ML]

Aude Sportisse, Christophe Biernacki, Claire Boyer, Julie Josse, Matthieu Marbac Lourdelle, et al.. Model-based Clustering with Missing Not At Random Data. 2021. ⟨hal-03494674v1⟩