Backpropagation Through Time (BPTT) - Complete Walkthrough
Given Values
Video companion: This verified YouTube video gives a second pass on the same prep area.
Video: Tutorial 30- Recurrent Neural Network Forward Propogation With Time
Inputs
x₁ = 1.0
x₂ = 2.0
Initial hidden: h₀ = 0
Weights
Wₓ = 0.5
Wₕ = 0.1
b = 0.0
W_y = 2.0
Target Outputs
y₁^target = 1.0
y₂^target = 2.0
Loss Function
L = 1 2 ( y 1 − 1 ) 2 + 1 2 ( y 2 − 2 ) 2 L = \frac{1}{2}(y_1 - 1)^2 + \frac{1}{2}(y_2 - 2)^2 L = 2 1 ( y 1 − 1 ) 2 + 2 1 ( y 2 − 2 ) 2
🟦 1. FORWARD PASS
Step 1: t = 1
h 1 = tanh ( W x x 1 + W h h 0 ) h_1 = \tanh(W_x x_1 + W_h h_0) h 1 = tanh ( W x x 1 + W h h 0 ) = tanh ( 0.5 ⋅ 1 + 0.1 ⋅ 0 ) = tanh ( 0.5 ) ≈ 0.4621 = \tanh(0.5 \cdot 1 + 0.1 \cdot 0) = \tanh(0.5) \approx 0.4621 = tanh ( 0.5 ⋅ 1 + 0.1 ⋅ 0 ) = tanh ( 0.5 ) ≈ 0.4621
y 1 = W y h 1 = 2.0 ⋅ 0.4621 = 0.9242 y_1 = W_y h_1 = 2.0 \cdot 0.4621 = 0.9242 y 1 = W y h 1 = 2.0 ⋅ 0.4621 = 0.9242
Step 2: t = 2
h 2 = tanh ( W x x 2 + W h h 1 ) h_2 = \tanh(W_x x_2 + W_h h_1) h 2 = tanh ( W x x 2 + W h h 1 ) = tanh ( 0.5 ⋅ 2 + 0.1 ⋅ 0.4621 ) = \tanh(0.5 \cdot 2 + 0.1 \cdot 0.4621) = tanh ( 0.5 ⋅ 2 + 0.1 ⋅ 0.4621 ) = tanh ( 1.0 + 0.04621 ) = tanh ( 1.04621 ) ≈ 0.7790 = \tanh(1.0 + 0.04621) = \tanh(1.04621) \approx 0.7790 = tanh ( 1.0 + 0.04621 ) = tanh ( 1.04621 ) ≈ 0.7790
y 2 = W y h 2 = 2.0 ⋅ 0.7790 = 1.558 y_2 = W_y h_2 = 2.0 \cdot 0.7790 = 1.558 y 2 = W y h 2 = 2.0 ⋅ 0.7790 = 1.558
Loss Calculation
L 1 = 1 2 ( 0.9242 − 1 ) 2 = 0.00288 L_1 = \frac{1}{2}(0.9242 - 1)^2 = 0.00288 L 1 = 2 1 ( 0.9242 − 1 ) 2 = 0.00288 L 2 = 1 2 ( 1.558 − 2 ) 2 = 0.0978 L_2 = \frac{1}{2}(1.558 - 2)^2 = 0.0978 L 2 = 2 1 ( 1.558 − 2 ) 2 = 0.0978 L = 0.1007 L = 0.1007 L = 0.1007
🟥 2. BACKWARD PASS (BPTT)
We compute gradients from t=2 backward to t=1.
⚠️ Remember: The gradient for h₁ has TWO paths:
Loss at t=1
Through h₂ (because h₁ affects h₂)
🔻 Step 1: Gradients at t = 2
dL/dy₂
∂ L ∂ y 2 = y 2 − y 2 t a r g e t = 1.558 − 2 = − 0.442 \frac{\partial L}{\partial y_2} = y_2 - y_2^{target} = 1.558 - 2 = -0.442 ∂ y 2 ∂ L = y 2 − y 2 t a r g e t = 1.558 − 2 = − 0.442
Since ∂y₂/∂h₂ = W_y = 2:
∂ L ∂ h 2 = − 0.442 ⋅ 2 = − 0.884 \frac{\partial L}{\partial h_2} = -0.442 \cdot 2 = -0.884 ∂ h 2 ∂ L = − 0.442 ⋅ 2 = − 0.884
Backprop through tanh
tanh ′ ( a 2 ) = 1 − h 2 2 = 1 − ( 0.7790 ) 2 = 1 − 0.6069 = 0.3931 \tanh'(a_2) = 1 - h_2^2 = 1 - (0.7790)^2 = 1 - 0.6069 = 0.3931 tanh ′ ( a 2 ) = 1 − h 2 2 = 1 − ( 0.7790 ) 2 = 1 − 0.6069 = 0.3931
So:
∂ L ∂ a 2 = ∂ L ∂ h 2 ⋅ tanh ′ ( a 2 ) = − 0.884 ⋅ 0.3931 = − 0.3475 \frac{\partial L}{\partial a_2} = \frac{\partial L}{\partial h_2} \cdot \tanh'(a_2) = -0.884 \cdot 0.3931 = -0.3475 ∂ a 2 ∂ L = ∂ h 2 ∂ L ⋅ tanh ′ ( a 2 ) = − 0.884 ⋅ 0.3931 = − 0.3475
Where a₂ = Wₓ·x₂ + Wₕ·h₁.
Gradient wrt W_x
∂ L ∂ W x ∣ t = 2 = ∂ L ∂ a 2 ⋅ x 2 = − 0.3475 ⋅ 2 = − 0.695 \frac{\partial L}{\partial W_x}\bigg|_{t=2} = \frac{\partial L}{\partial a_2} \cdot x_2 = -0.3475 \cdot 2 = -0.695 ∂ W x ∂ L t = 2 = ∂ a 2 ∂ L ⋅ x 2 = − 0.3475 ⋅ 2 = − 0.695
Gradient wrt W_h
∂ L ∂ W h ∣ t = 2 = ∂ L ∂ a 2 ⋅ h 1 = − 0.3475 ⋅ 0.4621 = − 0.1606 \frac{\partial L}{\partial W_h}\bigg|_{t=2} = \frac{\partial L}{\partial a_2} \cdot h_1 = -0.3475 \cdot 0.4621 = -0.1606 ∂ W h ∂ L t = 2 = ∂ a 2 ∂ L ⋅ h 1 = − 0.3475 ⋅ 0.4621 = − 0.1606
🔻 Step 2: Gradient flows back to h₁
Because h₁ affects:
Loss at t=1
h₂
From t = 2:
∂ L ∂ h 1 ∣ via t = 2 = ∂ L ∂ a 2 ⋅ W h = − 0.3475 ⋅ 0.1 = − 0.03475 \frac{\partial L}{\partial h_1}\bigg|_{\text{via } t=2} = \frac{\partial L}{\partial a_2} \cdot W_h = -0.3475 \cdot 0.1 = -0.03475 ∂ h 1 ∂ L via t = 2 = ∂ a 2 ∂ L ⋅ W h = − 0.3475 ⋅ 0.1 = − 0.03475
🔻 Step 3: Gradients at t = 1
dL/dy₁
∂ L ∂ y 1 = y 1 − 1 = 0.9242 − 1 = − 0.0758 \frac{\partial L}{\partial y_1} = y_1 - 1 = 0.9242 - 1 = -0.0758 ∂ y 1 ∂ L = y 1 − 1 = 0.9242 − 1 = − 0.0758
∂ L ∂ h 1 ∣ via y 1 = − 0.0758 ⋅ 2 = − 0.1516 \frac{\partial L}{\partial h_1}\bigg|_{\text{via } y_1} = -0.0758 \cdot 2 = -0.1516 ∂ h 1 ∂ L via y 1 = − 0.0758 ⋅ 2 = − 0.1516
TOTAL gradient wrt h₁
∂ L ∂ h 1 = − 0.1516 + ( − 0.03475 ) = − 0.18635 \frac{\partial L}{\partial h_1} = -0.1516 + (-0.03475) = -0.18635 ∂ h 1 ∂ L = − 0.1516 + ( − 0.03475 ) = − 0.18635
Backprop through tanh for t=1
tanh ′ ( a 1 ) = 1 − h 1 2 = 1 − ( 0.4621 ) 2 = 1 − 0.2135 = 0.7865 \tanh'(a_1) = 1 - h_1^2 = 1 - (0.4621)^2 = 1 - 0.2135 = 0.7865 tanh ′ ( a 1 ) = 1 − h 1 2 = 1 − ( 0.4621 ) 2 = 1 − 0.2135 = 0.7865
∂ L ∂ a 1 = − 0.18635 ⋅ 0.7865 = − 0.1466 \frac{\partial L}{\partial a_1} = -0.18635 \cdot 0.7865 = -0.1466 ∂ a 1 ∂ L = − 0.18635 ⋅ 0.7865 = − 0.1466
Gradients for W_x at t=1
∂ L ∂ W x ∣ t = 1 = − 0.1466 ⋅ x 1 = − 0.1466 \frac{\partial L}{\partial W_x}\bigg|_{t=1} = -0.1466 \cdot x_1 = -0.1466 ∂ W x ∂ L t = 1 = − 0.1466 ⋅ x 1 = − 0.1466
Gradients for W_h at t=1
∂ L ∂ W h ∣ t = 1 = − 0.1466 ⋅ h 0 = 0 \frac{\partial L}{\partial W_h}\bigg|_{t=1} = -0.1466 \cdot h_0 = 0 ∂ W h ∂ L t = 1 = − 0.1466 ⋅ h 0 = 0
(because h₀ = 0)
🟩 3. Combine Gradients Across Timesteps
∂ L ∂ W x = − 0.1466 + ( − 0.695 ) = − 0.8416 \frac{\partial L}{\partial W_x} = -0.1466 + (-0.695) = -0.8416 ∂ W x ∂ L = − 0.1466 + ( − 0.695 ) = − 0.8416
∂ L ∂ W h = − 0.1606 + 0 = − 0.1606 \frac{\partial L}{\partial W_h} = -0.1606 + 0 = -0.1606 ∂ W h ∂ L = − 0.1606 + 0 = − 0.1606
∂ L ∂ W y = h 1 ⋅ d L d y 1 + h 2 ⋅ d L d y 2 \frac{\partial L}{\partial W_y} = h_1 \cdot \frac{dL}{dy_1} + h_2 \cdot \frac{dL}{dy_2} ∂ W y ∂ L = h 1 ⋅ d y 1 d L + h 2 ⋅ d y 2 d L = 0.4621 ( − 0.0758 ) + 0.779 ( − 0.442 ) = 0.4621(-0.0758) + 0.779(-0.442) = 0.4621 ( − 0.0758 ) + 0.779 ( − 0.442 ) = − 0.0350 − 0.344 = − 0.379 = -0.0350 - 0.344 = -0.379 = − 0.0350 − 0.344 = − 0.379
🎯 Final Gradients
Parameter Gradient dL/dWₓ -0.8416 dL/dWₕ -0.1606 dL/dW_y -0.379
These would be used to update weights:
W ← W − η ∂ L ∂ W W \leftarrow W - \eta \frac{\partial L}{\partial W} W ← W − η ∂ W ∂ L
where η is the learning rate.
How to Use This Page as a Prep Plan
Do not treat this as passive reading. Convert the ideas in this page into a short weekly loop: learn one idea, practice it under interview conditions, then write down what changed. That is the fastest way to turn advice into visible interview behavior.
Prep area What you need to prove Practice artifact Understand Turn the prompt into a concrete goal. Clarifying questions and success criteria. Practice Use realistic constraints and timed reps. Worked examples with edge cases. Explain Make reasoning visible. Tradeoffs, assumptions, and test strategy. Improve Review misses quickly. A short feedback log and next action.
For Detailed RNN Backpropagation Walkthrough, the strongest candidates usually do three things well: they make their assumptions explicit, they use concrete examples instead of vague claims, and they review mistakes quickly enough that the next practice rep is better than the last one.
FAQ
How should I use this guide?
Read it once for the structure, then turn each section into a practice task with a visible artifact.
What should I do if I am short on time?
Prioritize the skills most likely to be tested, then do one mock or timed drill to expose the largest gap.
How do I know I am ready?
You can explain your approach clearly, recover from hints, and name tradeoffs without relying on memorized wording.