We use computation graph to explain why we have forward pass (forward propagation) follow by a backward pass (back-propagation)
Example: J(a, b, c) = 3(a + bc) where a=5, b=3, and c=2
To calculate J, we actually have to do 3 steps:
Step 1: calculate u = bc = 3x2 = 6
Step 2: calculate v = a + u = 5 + 6 = 11
Step 3 calculate J = 3(a + bc) = 3(a+u) = 3v = 3 x 11 = 33
This left to right computation is the forward pass.
When computing derivative, we do backward pass from right to left
If v = 11.001, then J = 33.003, so ∂J∂v=33.003−3311.001−11=3
If a = 5.001, then v = 5.001 + 6 = 11.001, then J = 33.001,
so ∂J∂a=33.003−3311.001−11=3=3∗1=∂J∂v∗∂v∂a . This is chain rule of calculus
Similarly, ∂J∂u=∂J∂v∗∂v∂u=3∗1=3
If b = 3.001, then u = 3.001 x 2 = 6.002, so ∂u∂b=6.002−63.001−3=2. Therefore,
∂J∂b=∂J∂u∗∂u∂b=3∗2=6=∂J∂v∗∂v∂u∗∂u∂b=3∗1∗2
∂J∂c=∂J∂v∗∂v∂u∗∂u∂c=3∗1∗3=9
Example: J(a, b, c) = 3(a + bc) where a=5, b=3, and c=2
To calculate J, we actually have to do 3 steps:
Step 1: calculate u = bc = 3x2 = 6
Step 2: calculate v = a + u = 5 + 6 = 11
Step 3 calculate J = 3(a + bc) = 3(a+u) = 3v = 3 x 11 = 33
This left to right computation is the forward pass.
When computing derivative, we do backward pass from right to left
If v = 11.001, then J = 33.003, so ∂J∂v=33.003−3311.001−11=3
If a = 5.001, then v = 5.001 + 6 = 11.001, then J = 33.001,
so ∂J∂a=33.003−3311.001−11=3=3∗1=∂J∂v∗∂v∂a . This is chain rule of calculus
Similarly, ∂J∂u=∂J∂v∗∂v∂u=3∗1=3
If b = 3.001, then u = 3.001 x 2 = 6.002, so ∂u∂b=6.002−63.001−3=2. Therefore,
∂J∂b=∂J∂u∗∂u∂b=3∗2=6=∂J∂v∗∂v∂u∗∂u∂b=3∗1∗2
∂J∂c=∂J∂v∗∂v∂u∗∂u∂c=3∗1∗3=9
Comments
Post a Comment