Cache-aware scheduling of scientific workflows in a multisite cloud

Gaëtan Heidsieck (1) , Daniel de Oliveira (2) , Esther Pacitti (1) , Christophe Pradal (1, 3, 4) , Francois Tardieu (5) , Patrick Valduriez (1)

1 ZENITH - Scientific Data Management
2 UFF - Universidade Federal Fluminense [Rio de Janeiro]
3 Cirad - Centre de Coopération Internationale en Recherche Agronomique pour le Développement
4 UMR AGAP - Amélioration génétique et adaptation des plantes méditerranéennes et tropicales
5 LEPSE - Écophysiologie des Plantes sous Stress environnementaux

Many scientific experiments today are performed using scientific workflows, which become more and more data-intensive. We consider the efficient execution of such workflows in a multisite cloud, leveraging heterogeneous resources available at multiple geo-distributed data centers. Since it is common for workflow users to reuse code or data from previous workflows, a promising approach for efficient workflow execution is to cache intermediate data in order to avoid re-executing entire workflows. However, caching intermediate data and scheduling workflows to exploit such caching in a multisite cloud is complex. In particular, workflow scheduling must be cache-aware, in order to decide whether reusing cache data or re-executing workflows entirely. In this paper, we propose a solution for cache-aware scheduling of scientific workflows in a multisite cloud. Our solution includes a distributed and parallel architecture and new algorithms for adaptive caching, cache site selection, and dynamic workflow scheduling. We implemented our solution in the OpenAlea workflow system, together with cache-aware distributed scheduling algorithms. Our experimental evaluation in a three-site cloud with a real application in plant phenotyping shows that our solution can yield major performance gains, reducing total time up to 42% with 60% of the same input data for each new execution.

Mots clés: Multisite cloud; Distributed Caching; Scientific Workflow; Workflow System; Workflow Scheduling

Domaines: Informatique [cs]; Biologie végétale

Gaëtan Heidsieck, Daniel de Oliveira, Esther Pacitti, Christophe Pradal, Francois Tardieu, et al.. Cache-aware scheduling of scientific workflows in a multisite cloud. Future Generation Computer Systems, 2021, 122, pp.172-186. ⟨10.1016/j.future.2021.03.012⟩. ⟨hal-03189130⟩