Pipelined Model Parallelism: Complexity Results and Memory Considerations

Olivier Beaumont (1, 2) , Lionel Eyraud-Dubois (1, 2) , Alena Shilova (1, 2)

1 HiePACS - High-End Parallel Algorithms for Challenging Numerical Simulations
2 UB - Université de Bordeaux

The training phase in Deep Neural Networks has become an important source of computing resource usage and the resulting volume of computation makes it crucial to perform efficiently on parallel architectures. Data parallelism is the most widely used method, but it requires to replicate the network weights on all processors, and to perform collective communications of the network weights. In this context, model parallelism is an attractive alternative, in which the different layers of the network are distributed over the computing processors. Indeed, it is expected to better distribute weights (to cope with memory problems) and it eliminates the need for large collective communications since only forward activations are communicated. However, to be efficient, it must be combined with a pipelined approach, which in turn induces new memory costs. In this paper, our goal is to formalize pipelined model parallelism as a scheduling problem, to establish its complexity, and to analyze the consequences of the assumptions that are typically performed in practical solutions such as Pipedream.

Domaines: Calcul parallèle, distribué et partagé [cs.DC]; Réseau de neurones [cs.NE]

Olivier Beaumont, Lionel Eyraud-Dubois, Alena Shilova. Pipelined Model Parallelism: Complexity Results and Memory Considerations. Europar 2021, Aug 2021, Lisbon, Portugal. ⟨hal-02968802v3⟩