nvmdava commited on
Commit
ec79aec
1 Parent(s): a9bdfe4

Update description2.md

Browse files
Files changed (1) hide show
  1. description2.md +10 -2
description2.md CHANGED
@@ -1,5 +1,7 @@
1
  ## Building a pipeline
2
 
 
 
3
  ### Building Block
4
  It starts by laying out the passes for a single microbatch, which we call a building
5
  block. For example, the building block of 1F1B schedule is made of a sequence of forward passes
@@ -7,12 +9,14 @@ followed by backward passes in the reverse order.
7
 
8
  ### Repeating
9
  More microbatches are then introduced. The building blocks are repeated and woven
10
- together to form a pipeline. Notably, legit building blocks are required to repeat without a collision, namely, the
 
11
  passes from two building blocks should not overlap with each other.
12
 
13
  ### Squeezing
14
  Depending on the building block, there may be redundant bubbles in the pipeline, which
15
- can be simply removed by squeezing without changing the order of the passes.
 
16
 
17
  ### Reordering (optional)
18
  We can reorder the passes in the warm-up and cool-down phase to further
@@ -20,6 +24,10 @@ improve the computation throughput. Intuitively, the peak of memory happens in t
20
  the pipeline, while in the warm-up and cool-down phases the RAM is under utilized, leaving some
21
  space for improving the computation throughput without changing peak memory.
22
 
 
 
 
 
23
  ## Alternative schedules
24
 
25
  By utilizing the building block, we can search for different types of schedules depending on the need. We illustrate few of them here below:
 
1
  ## Building a pipeline
2
 
3
+ Most of existing pipeline schedules can be explained under the following 4 step framework. In the example, we illustrate the construction of 1F1B and Eager 1F1B.
4
+
5
  ### Building Block
6
  It starts by laying out the passes for a single microbatch, which we call a building
7
  block. For example, the building block of 1F1B schedule is made of a sequence of forward passes
 
9
 
10
  ### Repeating
11
  More microbatches are then introduced. The building blocks are repeated and woven
12
+ together to form a pipeline. In the figure below, the repeating building blocks are shown in different shades of grey.
13
+ Notably, legit building blocks are required to repeat without a collision, namely, the
14
  passes from two building blocks should not overlap with each other.
15
 
16
  ### Squeezing
17
  Depending on the building block, there may be redundant bubbles in the pipeline, which
18
+ can be simply removed by squeezing without changing the order of the passes. For example,
19
+ Eager 1F1B shows a case where squeezing produces more efficient pipeline.
20
 
21
  ### Reordering (optional)
22
  We can reorder the passes in the warm-up and cool-down phase to further
 
24
  the pipeline, while in the warm-up and cool-down phases the RAM is under utilized, leaving some
25
  space for improving the computation throughput without changing peak memory.
26
 
27
+ | 1F1B | Eager 1F1B |
28
+ |-|-|
29
+ | <img src="https://cdn-uploads.huggingface.co/production/uploads/646968e05d7015663950e95b/olufjIyulR25CvAY6V2Hi.jpeg"/> | <img src="https://cdn-uploads.huggingface.co/production/uploads/646968e05d7015663950e95b/MbxeLnxyCGXNa6HVTt3sx.jpeg"/> |
30
+
31
  ## Alternative schedules
32
 
33
  By utilizing the building block, we can search for different types of schedules depending on the need. We illustrate few of them here below: