LLMs – Loss Functions (6)

Understanding KL Divergence, Cross-Entropy, and Classification Losses

A concept-first learning resource for building correct intuition about probabilistic losses

---

Why these concepts matter together

KL Divergence, Cross-Entropy, Softmax, and Information Gain all come from the same intellectual root: information theory. They are not arbitrary loss functions invented for deep learning - they formalize how much information is lost, gained, or misrepresented when a model makes predictions.

If you understand these ideas as measures of information mismatch, many design decisions in machine learning stop feeling magical and start feeling inevitable.

---

1. What is KL Divergence?

KL (Kullback–Leibler) Divergence measures how different two probability distributions are. More precisely, it measures how much extra information is required when you use one distribution $Q$ to represent data that actually follows another distribution $P$.

This quantity is asymmetric. Using $Q$ to approximate $P$ is not the same as using $P$ to approximate $Q$. That asymmetry matters: KL is not a distance, but a penalty for wrong assumptions.

From an information perspective, KL Divergence answers the question:

How much information do I lose by believing the world looks like Q when it actually looks like P?

---

2. Cross-Entropy Loss and its physical meaning

Cross-Entropy measures how well one probability distribution predicts another. In supervised learning, the “true” distribution comes from labels, and the predicted distribution comes from the model.

For binary classification, the loss is:

Here, the labels are treated as a probability distribution - even if they are one-hot - and the model output is explicitly a probability.

The physical meaning is straightforward:

• If the predicted distribution perfectly matches the true one, the loss is zero.
• As predictions become more confident and wrong, the loss grows rapidly.
• The loss reflects how many extra bits of information are needed to encode the truth using the model’s beliefs.

Cross-Entropy does not measure numeric error. It measures belief mismatch.

---

3. KL Divergence vs Cross-Entropy

The relationship between these two is foundational:

$$\text{CrossEntropy}(P, Q) = H(P) + \mathrm{KL}(P \,\|\, Q)$$

The entropy of the true distribution $H(P)$ is fixed - it does not depend on the model. That means minimizing cross-entropy is equivalent to minimizing KL divergence.

This leads to an important practical insight:

• KL Divergence is a general theoretical measure.
• Cross-Entropy is the trainable objective we can actually optimize.

In classification, we almost never compute KL explicitly because cross-entropy already contains the same signal, minus an irrelevant constant.

---

4. Loss imbalance in multi-task learning

In multi-task learning, different tasks often produce losses on very different scales. If left untreated, one task dominates training while others stagnate.

Common strategies include:

• Assigning explicit weights to each task’s loss.
• Learning those weights dynamically based on gradient magnitudes or uncertainty.
• Normalizing losses so they are comparable.
• Structuring the architecture so tasks share only appropriate representations.

The underlying principle is not mathematical elegance - it is optimization fairness. Every task should meaningfully influence parameter updates.

---

5. Why classification uses Cross-Entropy instead of MSE

Mean Squared Error measures numeric distance. Classification is not a numeric prediction problem - it is a probabilistic decision problem.

Cross-Entropy is preferred because:

• It operates directly on probabilities.
• It strongly penalizes confident wrong predictions.
• Its gradients remain informative even when predictions are very wrong.
• It aligns perfectly with maximum likelihood estimation.

MSE treats probabilities like arbitrary numbers. Cross-Entropy treats them like beliefs.

---

6. What is Information Gain?

Information Gain measures how much uncertainty is reduced after observing a feature. It i

How to Use This Page as a Prep Plan

Do not treat this as passive reading. Convert the ideas in this page into a short weekly loop: learn one idea, practice it under interview conditions, then write down what changed. That is the fastest way to turn advice into visible interview behavior.

Prep area	What you need to prove	Practice artifact
Understand	Turn the prompt into a concrete goal.	Clarifying questions and success criteria.
Practice	Use realistic constraints and timed reps.	Worked examples with edge cases.
Explain	Make reasoning visible.	Tradeoffs, assumptions, and test strategy.
Improve	Review misses quickly.	A short feedback log and next action.

For LLMs – Loss Functions (6), the strongest candidates usually do three things well: they make their assumptions explicit, they use concrete examples instead of vague claims, and they review mistakes quickly enough that the next practice rep is better than the last one.

Video Walkthrough

This verified YouTube video gives a second pass on the same preparation area. Use it after reading the guide, then come back and turn the advice into a practice artifact.

Video: Transformers, the tech behind LLMs | Deep Learning Chapter 5

FAQ

How should I use this guide?

Read it once for the structure, then turn each section into a practice task with a visible artifact.

What should I do if I am short on time?

Prioritize the skills most likely to be tested, then do one mock or timed drill to expose the largest gap.

How do I know I am ready?

You can explain your approach clearly, recover from hints, and name tradeoffs without relying on memorized wording.