Large scale in transit computation of quantiles for ensemble runs

Alejandro Ribes (1) , Théophile Terraz (2) , Bertrand Iooss (3, 4, 5) , Yvan Fournier (1) , Bruno Raffin (6)

1 EDF - EDF
2 DATAMOVE - Data Aware Large Scale Computing
3 EDF R&D PRISME - Performance, Risque Industriel, Surveillance pour la Maintenance et l’Exploitation
4 IMT - Institut de Mathématiques de Toulouse UMR5219
5 GdR MASCOT-NUM - Méthodes d'Analyse Stochastique des Codes et Traitements Numériques
6 UGA [2016-2019] - Université Grenoble Alpes [2016-2019]

The classical approach for quantiles computation requires availability of the full sample before ranking it. In uncertainty quantification of numerical simulation models, this approach is not suitable at exascale as large ensembles of simulation runs would need to gather a prohibitively large amount of data. This problem is solved thanks to an on-the-fly and iterative approach based on the Robbins-Monro algorithm. This approach relies on Melissa, a file avoiding, adaptive, fault-tolerant and elastic framework. On a validation case producing 11 TB of data, which consists in 3000 fluid dynamics parallel simulations on a 6M cell mesh, it allows on-line computation of spatio-temporal maps of percentiles.

Mots clés: In Situ Data Processing; Parametric Studies; Iterative Statistics; Robbins-Monro; Uncertainty Quantification

Domaines: Statistiques [math.ST]

Alejandro Ribes, Théophile Terraz, Bertrand Iooss, Yvan Fournier, Bruno Raffin. Large scale in transit computation of quantiles for ensemble runs. [Research Report] EDF R&D. 2019. ⟨hal-02016828v2⟩