Articles
LLM Tokenization Interview Questions: BPE, Special Tokens, and Cost Trade-Offs
Prepare for LLM tokenization interviews with BPE traces, special-token pitfalls, multilingual examples, and practical token budget and cost calculations.
A tokenizer is part of an LLM's input contract, not a cosmetic text splitter. Changing it can change token IDs, context consumption, special-token behavior, and the meaning of a checkpoint's embedding rows. A useful interview answer connects those consequences to one small example.
The direct answer: BPE learns a ranked sequence of frequent adjacent-symbol merges. At inference, the tokenizer applies its saved rules and preprocessing; it does not relearn merges from your prompt. Special tokens carry model-specific structure, while cost depends on the complete serialized request and the applicable billing rules.
The exercises below are original teaching examples, not claims about questions asked by particular employers or measured outputs from a named tokenizer.
What exactly does a tokenizer return?
At its simplest, encoding maps input text to an ordered sequence of integer IDs. The model uses those IDs to look up embeddings. Depending on the interface, the tokenizer can also return attention masks, offsets, special-token masks, or overflow information. An ID is an index, not a semantic score: token 400 is not necessarily more similar to token 401 than to token 900.
Ask which artifact defines the mapping. A reproducible answer names the tokenizer revision, vocabulary, preprocessing, added tokens, and chat template, alongside the model checkpoint. A vocabulary file alone may not describe the complete pipeline.
Consider a service that accidentally deploys a different tokenizer with unchanged weights. If the same integer now denotes different text, the embedding lookup remains numerically valid but semantically wrong. Checking tensor shapes will miss the failure. Compare a small set of encoded requests against saved expected IDs before accepting a deployment.
Can you perform one BPE training step by hand?
Hugging Face's BPE course describes training as repeatedly merging the most frequent adjacent pair. Here is a deliberately small corpus with no normalization, no end marker, and no merges across word boundaries:
| Word | Frequency | Initial symbols |
|---|---|---|
| cat | 5 | c a t |
| car | 3 | c a r |
| dog | 2 | d o g |
The weighted counts are c,a = 8, a,t = 5, a,r = 3, d,o = 2, and o,g = 2. The first merge is therefore c + a -> ca. After that merge, the corpus contains ca t, ca r, and d o g; the next highest pair is ca,t = 5.
Explain why counting unique words alone gives the wrong result. Pair frequencies must reflect corpus multiplicity. Also explain the stopping condition: this toy procedure could stop after a target number of merges or when no useful pairs remain. Tie-breaking must be specified if reproducibility matters.
Why is encoding different from training?
During training, pair frequencies change as merges alter the corpus. During encoding, the learned vocabulary and merge priorities are fixed. The new input does not get to choose its own statistically convenient vocabulary.
Extend the toy vocabulary with both learned merges: first c+a, then ca+t. Encoding cat yields the single token cat; encoding car yields ca,r. A candidate who merges ar because it appears frequently in a new prompt has quietly switched from encoding to training.
Pre-tokenization can impose hard boundaries before the merge algorithm runs. If whitespace separates segments, an otherwise available merge cannot necessarily cross that boundary. This is why "just pick the longest dictionary string" is an incomplete description of an arbitrary BPE implementation.
A good debugging artifact is a trace with raw text, normalized text, pre-tokenized segments, initial symbols, applied merges, and final IDs. That trace tells you which stage introduced a difference.
How do BPE, bytes, and SentencePiece differ?
BPE names a merge algorithm. Byte-level BPE specifies a byte-based starting representation. SentencePiece is a tokenizer framework described by its authors as operating directly on raw sentences; it is not a synonym for one particular segmentation algorithm.
The practical interview distinction is between algorithm, representation, and implementation. A byte-based vocabulary can represent UTF-8 input through its constituent bytes without needing a separate unknown symbol for every unfamiliar character. That does not mean each character becomes exactly one token, or that every token boundary is a Unicode character boundary.
Imagine logging each token as independently decoded text. A token containing only part of a multibyte character may display poorly when decoded alone, even though decoding the complete byte sequence reconstructs the text. Inspect bytes and offsets before concluding that the model corrupted the input.
Sennrich and colleagues' original subword paper motivates reusable subword units for rare words. That motivation does not establish that every modern tokenizer uses BPE or that one vocabulary is best for every workload.
Why can spaces and normalization change the answer?
Token counts belong to an exact string and tokenizer, not to a sentence's abstract meaning. A leading space, newline, punctuation choice, or normalization policy can change segmentation. Some systems preserve whitespace distinctions; others normalize selected characters before encoding.
Suppose a customer identifier contains a visually similar character copied from another application. Aggressive normalization may improve search matching while destroying the identifier's exact representation. Conversely, inconsistent normalization between training and inference can introduce unexpected fragments.
Build a boundary test set containing leading spaces, repeated newlines, tabs, mixed scripts, accented text, emoji sequences, and domain identifiers. For each case, decide whether the requirement is exact byte round-tripping or an explicitly normalized output. Do not declare the tokenizer broken because it implements a documented normalization rule.
The relevant acceptance test is not merely "the decoded sentence looks right." Compare the expected normalized or original string according to the application's actual contract.
How should you compare multilingual fragmentation?
Avoid claiming that one language always costs a fixed multiple of another. Fragmentation depends on the tokenizer's training distribution, vocabulary allocation, writing system, and the particular text. Word counts also become difficult to compare across languages with different word-boundary conventions.
For a controlled exercise, imagine two human-checked translations of the same support instruction. A fictional tokenizer produces 24 tokens for version A and 42 for version B. Version B uses 1.75 times as many input tokens for this pair. That is a measurement design example, not a benchmark for any language or product.
A serious evaluation repeats this comparison across representative tasks and reports distributions, including long-tail documents. Check quality as well as length: a cheaper translated request is not a saving if it changes legal names, loses distinctions, or causes more retries.
For context budgeting, the longer representation leaves fewer tokens for evidence and the answer. That makes fragmentation an application-capacity concern even when monetary billing is irrelevant.
What do special tokens actually control?
A special token is reserved for a structural role in a particular model interface. Examples can include sequence boundaries, padding, message boundaries, and modality markers. Their exact identities and semantics are checkpoint-specific.
Hugging Face's chat-template documentation warns that separately adding special tokens after formatting a chat can duplicate tokens already supplied by the template. A useful interview scenario is a model that suddenly produces malformed turn boundaries after an integration change. Compare the serialized token sequence before tuning generation settings.
Do not assume that typing the visible spelling of a reserved token always produces its reserved ID. Tokenizer configuration determines how that string is interpreted. Likewise, adding a token to the vocabulary does not automatically teach the model its intended behavior.
If a new token expands the vocabulary, inspect the model's embedding compatibility and training requirements. Treat token registration, model resizing where required, and semantic learning as separate operations with separate tests.
How do you calculate an honest token budget?
Use a transparent hypothetical price model instead of memorizing a provider's current rates. Assume ordinary input costs $2 per million tokens and output costs $8 per million tokens, with no caching discount, tools, or other charges.
| Item | Tokens | Hypothetical rate | Cost |
|---|---|---|---|
| Complete input | 1,200 | $2 per million | $0.0024 |
| Generated output | 300 | $8 per million | $0.0024 |
| Total request | 1,500 | Separate rates | $0.0048 |
At 100,000 identical requests, that arithmetic gives $480. Cutting input by 200 tokens saves $0.0004 per request, or $40 across that volume, assuming output and retries remain unchanged.
The qualification matters. Shortening an instruction can increase retries or response length, reversing the apparent saving. Count system content, history, retrieved passages, tool schemas, and formatting overhead where the actual interface includes them. Measure the final request rather than adding isolated string estimates and assuming boundaries are additive.
Does a larger vocabulary always make inference cheaper?
No. A larger vocabulary can shorten sequences for some text while increasing embedding storage and the size of the output projection. The best choice depends on workload and model architecture.
For a concrete storage estimate, suppose the vocabulary has 50,000 rows, each with 4,096 values stored in two bytes. One embedding matrix occupies 409,600,000 bytes, about 409.6 decimal MB. Increasing the vocabulary to 100,000 doubles that matrix's storage. This calculation excludes optimizer state, gradients, alignment, and additional untied output weights.
Shorter sequences may reduce attention and cache work, but the total latency change is empirical. A reduction in token count does not guarantee the same proportional reduction in wall-clock time.
An interview answer should propose an experiment: fix model quality targets, evaluate representative language and domain slices, measure prefill and generation separately, and include peak memory. Compare complete systems rather than celebrating one compression statistic.
How would you debug a context-limit incident?
Start with the exact failed request and the exact tokenizer revision, preserving sensitive text through the team's approved logging controls. Determine whether the limit concerns input alone or the combined budget for input and output.
Suppose a hypothetical service has an 8,000-token combined limit. Instructions consume 600 tokens, history 1,800, retrieved evidence 4,900, and the reserved response 1,000. The total is 8,300, so the request exceeds the limit by 300 tokens. Removing arbitrary characters until it fits risks cutting the evidence that answers the question.
Instead, rank evidence for relevance, remove repeated history, and preserve provenance and document boundaries. Recount after assembling the final request. Leave a margin where wrappers or service-side formatting are not fully observable.
If local estimates disagree with provider usage, compare model selection, template revision, special-token settings, and any separately metered components. An unexplained discrepancy is an investigation result, not permission to invent a constant correction factor.
What makes a strong interview explanation?
A strong response moves between representation, arithmetic, and failure diagnosis. For the BPE exercise, state the corpus assumptions before counting pairs. For multilingual comparison, distinguish a measured sample ratio from a language-wide claim. For costs, label hypothetical rates and preserve separate input and output terms.
Try a final challenge: a release lowers average input tokens by 12 percent but raises total spend. Offer three testable explanations. Longer answers, more retries, or a shift toward more expensive request categories could each produce that result. Ask for the relevant breakdown instead of blaming the tokenizer without evidence.
For practice, explain the toy merge trace aloud, calculate the storage example without a calculator, and then diagnose the 8,300-token request. Repeat with one changed assumption, such as a different template or output reservation. You are ready when your conclusion changes for the right reason and you can identify the observation that would disprove it.
Sources and Further Reading
Research checked September 24, 2026. Numeric workloads, prices, and corpus examples above are illustrative, not vendor measurements.