LLMs – Activation Functions (3)

Feed-Forward Networks (FFN) in Transformers: From Linear Blocks to Gated Designs

In Transformer models, attention usually gets all the credit. But in practice, the Feed-Forward Network (FFN) block is where most parameters live and where a large portion of a model’s expressive power comes from. If attention decides what information to mix, FFNs decide how to transform it.

This post is meant to be a learning resource, helping you understand FFN computation formulas, activation choices, and why modern LLMs moved from simple FFNs to gated variants. More importantly, it connects these ideas to broader model design principles you’ll see far beyond Transformers.

---

1. What is the computation formula of the FFN block?

2. What is the computation formula of GeLU?

3. What is the computation formula of Swish? Here’s what matters conceptually:

Uses 2 trainable weight matrices Hidden dimension is 4h

4. What is the FFN computation formula using GLU (Gated Linear Unit)?

5. What is the computation formula of GLU using GeLU?

6. What is the computation formula of GLU using Swish?

Note: Uses 3 trainable weight matrices Hidden dimension is $\frac{4h \times 2}{3}$

Which activation functions are used by different LLMs?

• The FFN uses two trainable weight matrices
• The hidden dimension is typically 4× the model dimension
• The same FFN is applied to every token position

For a model with hidden size $h = 4096$, the FFN expands activations to:

$$4h = 4 \times 4096 = 16384$$

This expansion is not arbitrary. It gives the model room to learn richer, nonlinear transformations before projecting back down.

This design pattern - expand → nonlinearity → compress - appears everywhere in deep learning, from ResNets to diffusion models.

---

2. Activation Functions: Why Nonlinearity Matters

Without a nonlinearity, the FFN would collapse into a single linear transformation. Activation functions determine how expressive and trainable the block becomes.

GeLU (Gaussian Error Linear Unit)

GeLU smoothly weights inputs by how likely they are under a Gaussian distribution. Its commonly used approximation is:

[ \text{GeLU}(x) \a

How to Use This Page as a Prep Plan

Do not treat this as passive reading. Convert the ideas in this page into a short weekly loop: learn one idea, practice it under interview conditions, then write down what changed. That is the fastest way to turn advice into visible interview behavior.

Prep area	What you need to prove	Practice artifact
Understand	Turn the prompt into a concrete goal.	Clarifying questions and success criteria.
Practice	Use realistic constraints and timed reps.	Worked examples with edge cases.
Explain	Make reasoning visible.	Tradeoffs, assumptions, and test strategy.
Improve	Review misses quickly.	A short feedback log and next action.

For LLMs – Activation Functions (3), the strongest candidates usually do three things well: they make their assumptions explicit, they use concrete examples instead of vague claims, and they review mistakes quickly enough that the next practice rep is better than the last one.

Video Walkthrough

This verified YouTube video gives a second pass on the same preparation area. Use it after reading the guide, then come back and turn the advice into a practice artifact.

Video: Activation Functions - EXPLAINED!

FAQ

How should I use this guide?

Read it once for the structure, then turn each section into a practice task with a visible artifact.

What should I do if I am short on time?

Prioritize the skills most likely to be tested, then do one mock or timed drill to expose the largest gap.

How do I know I am ready?

You can explain your approach clearly, recover from hints, and name tradeoffs without relying on memorized wording.