Data Management for the RedisDG Scientific Workflow Engine

Leila Abidi (1) , Souha Bejaoui (1) , Christophe Cérin (1) , Jonathan Lejeune (2) , Yanik Ngoko (1) , Walid Saad (1)

1 LIPN - Laboratoire d'Informatique de Paris-Nord
2 Regal - Large-Scale Distributed Systems and Applications

—In this paper we investigate the general problem of controlling a scientific workflow service in terms of data management. We focus on the data management problem for the RedisDG scientific workflow engine. RedisDG is based on the Publish/Subscribe paradigm for the interaction between the different components of the system, hence new issues appear for scheduling. Indeed, the Publish/Subscribe paradigm utilization introduces different challenging problems, among them the design of effective solutions for managing data, on the fly, when tasks are published. Our contributions are twofold. First we add new functionalities to the RedisDG workflow engine with scheduling decisions related to the allocation of data intensive jobs to compute units and according to an efficient management of data and second we introduce a large set of experiments to validate our approaches. We analyze our results and we also sketch perspectives and insights. Experiments are conducted on the Grid'5000 testbed and the paper is a step forward to implement a 'Workflow engine as a Service' (WaaS).

Mots clés: Large scale cloud applications including Internet/Web computing; Volunteer computing; Data management; Scientific workflow; Scheduling; Experimental evaluation on large scale systems

Domaines: Informatique [cs]; Calcul parallèle, distribué et partagé [cs.DC]

Leila Abidi, Souha Bejaoui, Christophe Cérin, Jonathan Lejeune, Yanik Ngoko, et al.. Data Management for the RedisDG Scientific Workflow Engine. IEEE International Conference on Computer and Information Technology, Dec 2016, Nadi, Fiji. pp.599 - 606, ⟨10.1109/CIT.2016.55⟩. ⟨hal-01517163⟩