Tuning EASY-Backfilling Queues

Jérôme Lelong (1) , Valentin Reis (1, 2) , Denis Trystram (2, 3)

1 DAO - Données, Apprentissage et Optimisation
2 DATAMOVE - Data Aware Large Scale Computing
3 IUF - Institut universitaire de France

EASY-Backfilling is a popular scheduling heuristic for allocating jobs in large scale High Performance Computing platforms. While its aggressive reservation mechanism is fast and prevents job starvation, it does not try to optimize any scheduling objective per se. We consider in this work the problem of tuning EASY using queue reordering policies. More precisely, we propose to tune the reordering using a simulation-based methodology. For a given system, we choose the policy in order to minimize the average waiting time. This methodology departs from the First-Come, First-Serve rule and introduces a risk on the maximum values of the waiting time, which we control using a queue thresholding mechanism. This new approach is evaluated through a comprehensive experimental campaign on five production logs. In particular, we show that the behavior of the systems under study is stable enough to learn a heuristic that generalizes in a train/test fashion. Indeed, the average waiting time can be reduced consistently (between 11% to 42% for the logs used) compared to EASY, with almost no increase in maximum waiting times. This work departs from previous learning-based approaches and shows that scheduling heuristics for HPC can be learned directly in a policy space.

Domaines: Calcul parallèle, distribué et partagé [cs.DC]

Jérôme Lelong, Valentin Reis, Denis Trystram. Tuning EASY-Backfilling Queues. 21st Workshop on Job Scheduling Strategies for Parallel Processing, May 2017, Orlando, United States. pp.43-61, ⟨10.1007/978-3-319-77398-8_3⟩. ⟨hal-01522459⟩