Hugging Face's logo

Spaces:
sail
/
pipeline-parallelism-with-controllable-memory
Running

App Files Community
pipeline-parallelism-with-controllable-memory / description1.md
Wan Xinyi
Render schedule
4835f75
preview code
|
raw
history blame
583 Bytes

Zero Bubble Pipeline Parallelism

Zero Bubble Pipeline Parallelism is a novel pipeline parallelism algorithm able to reduce the bubble of pipeline parallelism to almost zero while preserving synchronous semantics.

Our paper is coming soon.

Try out our implementation based on Megatron on https://github.com/sail-sg/zero-bubble-pipeline-parallelism

Experiments shows zero bubble pipeline parallelism can accelerate training up to 30% with a similar memory comsumption. A detailed table of experiments is coming soon.