Articles

Large Language Models (LLMs): Fundamentals (1)

This guide covers fundamentals of large language models, including architectural choices, training objectives, scaling behavior, capability…

Large Language Models (LLMs): Fundamentals - A Learning-Oriented Resource

Large Language Models (LLMs) are no longer just an NLP topic; they are becoming a general-purpose interface layer between humans and computation. To understand where they are useful - and where they are not - it’s important to go beyond surface-level definitions and understand how architectural choices, training objectives, and scaling behavior shape what these models can do in practice.

This post is designed as a learning resource, not a quick summary. It aims to help candidates build mental models they can reuse across research, engineering, product design, and system thinking.

1. Mainstream Open-Source LLM Architectures: What Problem Each One Solves

Most modern open-source LLMs fall into three architectural families. These architectures differ less in what they compute and more in how information flows during training and generation.

Encoder–Decoder architectures were originally built for tasks where understanding comes before generation. The encoder reads the entire input with bidirectional attention, forming a rich internal representation. The decoder then generates output token by token. This structure is naturally suited for translation, summarization, and tasks where the full input must be deeply understood before producing an answer. However, this separation introduces overhead. Long-form generation is inefficient, and performance often depends on supervised fine-tuning rather than pure pretraining.

Causal Decoder architectures remove this separation entirely. The model reads and generates text in a strictly left-to-right fashion. This design aligns training and inference perfectly: the model is always doing the same task - predicting the next token. This alignment explains why causal decoders scale so well with massive unlabeled data and why they dominate modern LLM development. Their simplicity is not a limitation; it is their main advantage.

Prefix Decoder architectures sit between the two. They allow a prefix of tokens to attend bidirectionally, while enforcing autoregressive generation for the output. Conceptually, this attempts to combine strong input understanding with flexible generation. In practice, the added complexity reduces training efficiency, which limits large-scale adoption.

The key insight: architecture reflects assumptions about how much understanding must precede generation.

2. Attention Masks: The Real Architectural Difference

At a technical level, these architectures mainly differ in their attention mask design - that is, which tokens are allowed to attend to which other tokens.

Encoder–Decoder models allow full bidirectional attention on the input side, which improves comprehension but increases cost. Causal decoders strictly forbid access to future tokens, enforcing a clean probabilistic structure. Prefix decoders selectively relax this restriction.

Understanding attention masks helps explain:

why some models excel at comprehension-heavy tasks
why long-context generation remains difficult
why decoder-only models scale more efficiently

This perspective generalizes beyond NLP and applies to any sequence modeling problem.

3. Training Objectives: Why “Next Token Prediction” Is Enough

Despite their complexity, most LLMs are trained with surprisingly simple objectives. The dominant one is the language modeling objective: given previous tokens, predict the next token and maximize log-likelihood.

This objective has two important properties:

It requires no labeled data.
It scales smoothly with more data and parameters.

Causal decoders compute loss on every token, making training highly efficient. Prefix decoders compute loss only on output tokens, which partially explains their lower efficiency.

Some models - especially encoder–decoder ones like T5 - use a denoising autoencoder objective, where corrupted text must be reconstructed. This encourages strong internal representations but adds implementation and computational complexity.

The broader lesson: emergent intelligence comes from scale and consistency, not from exotic objectives.

4. Emergent Abilities Are Usually Measurement Artifacts

LLMs often appear to develop abilities “suddenly” once they reach a certain size. Research suggests this is largely an illusion caused by how we evaluate models.

Complex tasks are composed of many sub-skills. Each sub-skill improves gradually as models scale, but the final task metric may only improve once all sub-skills cross a threshold. This creates the appearance of sudden emergence, even though underlying capabilities grew smoothly.

This insight is useful beyond LLMs:

it applies to system performance metrics
it explains misleading benchmark jumps
it cautions against over-interpreting sharp inflection points

5. Why Decoder-Only Models Dominate Modern LLMs

Decoder-only architectures dominate not because alternatives are flawed, but because they are better aligned with today’s constraints.

They perform well in zero-shot settings, require no labeled data for pretraining, and are simpler to engineer at scale. Encoder–Decoder models often outperform them only because they use more parameters, not because they are fundamentally superior.

From a systems perspective, decoder-only models offer:

higher training efficiency
lower engineering complexity
better leverage of massive unlabeled corpora

Under equal parameter and inference budgets, they are usually the optimal choice.

6. What “Large” Means in Large Language Models

In practice, “large” typically means over one billion parameters, though this threshold keeps moving.

Parameter count (e.g., 7B, 60B, 175B) refers to the number of learned weights in the model. More parameters increase representational capacity, but also raise costs in compute, memory, and energy.

Understanding parameter scaling helps when:

choosing models for deployment
estimating inference costs
interpreting benchmark comparisons

7. Strengths and Limits: A Balanced View

LLMs are powerful because they can be pretrained on massive unlabeled data, then adapted cheaply to many tasks. They generate text, images, and structured outputs, and demonstrate reasoning behaviors once thought unreachable by statistical models.

At the same time, they impose real constraints:

high training and inference costs
environmental impact
data bias and safety risks
limited interpretability and controllability

These are not edge cases; they are central challenges that shape how LLMs should be deployed responsibly.

8. Why This Knowledge Transfers Beyond NLP

Understanding LLM fundamentals is not just for ML engineers.

It informs:

product decisions (generation vs understanding)
system design (cost–capability trade-offs)
evaluation thinking (avoiding misleading metrics)
career growth (reading papers and model cards critically)

Once you understand why these models work, you can reason about new architectures, new benchmarks, and new applications without starting from scratch.

Final Takeaway

Large Language Models are not mysterious because they are complex. They are powerful because simple ideas - attention, next-token prediction, scale - were executed relentlessly.

That principle will outlast any specific model name.

How to Use This Page as a Prep Plan

Do not treat this as passive reading. Convert the ideas in this page into a short weekly loop: learn one idea, practice it under interview conditions, then write down what changed. That is the fastest way to turn advice into visible interview behavior.

Prep area	What you need to prove	Practice artifact
Understand	Turn the prompt into a concrete goal.	Clarifying questions and success criteria.
Practice	Use realistic constraints and timed reps.	Worked examples with edge cases.
Explain	Make reasoning visible.	Tradeoffs, assumptions, and test strategy.
Improve	Review misses quickly.	A short feedback log and next action.

For Large Language Models (LLMs): Fundamentals (1), the strongest candidates usually do three things well: they make their assumptions explicit, they use concrete examples instead of vague claims, and they review mistakes quickly enough that the next practice rep is better than the last one.

Video Walkthrough

This verified YouTube video gives a second pass on the same preparation area. Use it after reading the guide, then come back and turn the advice into a practice artifact.

Video: Large Language Models explained briefly

FAQ

How should I use this guide?

Read it once for the structure, then turn each section into a practice task with a visible artifact.

What should I do if I am short on time?

Prioritize the skills most likely to be tested, then do one mock or timed drill to expose the largest gap.

How do I know I am ready?

You can explain your approach clearly, recover from hints, and name tradeoffs without relying on memorized wording.