Positional Encoding Interview Questions: RoPE, ALiBi, and Long-Context Limits

Positional encoding tells a transformer how token order should influence its computations. In an interview, naming sinusoidal embeddings, RoPE, and ALiBi is only the beginning. The harder question is what changes when the same content moves, and what breaks when the sequence becomes longer.

The direct answer: RoPE rotates query and key components using their positions, making their dot products depend on relative rotation. ALiBi adds a distance-dependent bias to attention scores. Neither mechanism, by itself, proves that a model can reliably reason over arbitrarily long inputs.

The exercises here use simplified attention heads and hypothetical results; they are not benchmark claims about a commercial model.

Why does attention need position information?

Without position information, unconstrained self-attention treats a permutation of the input tokens as a corresponding permutation of the outputs. Content similarities alone do not distinguish all ordering relationships. "Alex followed Sam" and "Sam followed Alex" contain the same words but assign different roles.

The original Transformer paper adds positional information to token embeddings. Its sinusoidal construction uses multiple frequencies, allowing positions to have distinct patterns without a learned lookup row for every possible index.

For an interview, separate this argument from causal masking. A causal mask controls which tokens are visible at each step and therefore introduces an ordering constraint. It does not replace every useful representation of distance or location. Saying that a decoder has "no order information at all" without explicit position embeddings is too broad.

Ask whether the problem is bidirectional encoding, autoregressive decoding, image patches, or another setting. The relevant notion of position and the visibility rules may differ.

How do absolute and relative approaches differ?

An absolute scheme associates information with a token's position in the sequence. A relative mechanism directly influences interactions using relationships such as the distance between a query and key. Real architectures can combine ideas, so these labels are explanatory categories rather than mutually exclusive implementation laws.

Mechanism	Where position enters	A useful interview concern
Learned absolute embeddings	Added position vectors	Untrained or unavailable position rows
Sinusoidal embeddings	Fixed position-dependent vectors	Computable positions versus learned generalization
RoPE	Rotations of query and key components	Frequency behavior outside the training regime
ALiBi	Distance bias in attention logits	Recency preference and distant evidence

Computing a positional representation at a new index is not equivalent to having learned how to use it. A sinusoid can be evaluated at position one million, but that mathematical fact says little about a particular model's accuracy there.

For a concrete design choice, suppose a model must process forms whose key fields appear near the beginning. Absolute location may be a useful signal. If the same field moves between templates, evaluation should test whether that positional shortcut has become a liability.

Can you explain RoPE without memorizing the paper?

Take two coordinates of a query vector and rotate them by an angle determined by position. Do the same to the corresponding key coordinates. Across a head, different coordinate pairs can use different frequencies.

Let R(p) represent rotation at position p for one frequency. The rotated dot product is q^T R(m)^T R(n) k. Because rotations compose, the middle term becomes R(n-m). The relative displacement appears naturally in the interaction. The RoFormer paper develops this construction.

Now move two otherwise unchanged vectors from positions 2 and 5 to positions 12 and 15. Their displacement stays 3. Under this simplified fixed-frequency calculation, the relative rotation is unchanged.

Do not extend that statement into "the entire model output is always unchanged." The surrounding context, masks, position-dependent modifications, and earlier hidden states may differ. The invariance belongs to the stated local calculation. Naming that boundary is stronger than making a dramatic but incorrect global claim.

What does an actual rotation do to a score?

Use an intentionally simple two-dimensional example. Let the unrotated query and key both be (1, 0). If their relative angle is zero, their dot product is 1. At a relative angle of 60 degrees, the dot product is 0.5. At 90 degrees, it is zero.

This shows how content alignment and position interact. It does not establish that every real head uses these angles or that distance always decreases every attention score. Rotations are periodic, and actual vectors have many coordinates with learned content.

A useful follow-up is whether RoPE guarantees monotonically decreasing attention with distance. It does not guarantee that for each query-key pair. Content, multiple frequencies, and the rest of the model determine the resulting scores and probabilities.

When presenting a derivation, specify radians or degrees, coordinate pairing, and sign conventions. Two implementations can use different layouts and still be consistent internally, but mixing incompatible conventions between queries and keys can silently damage attention.

How does ALiBi change an attention decision?

For a toy causal head, let the attention score be the scaled content score minus slope * distance. The slope is positive, and distance is how far a visible key lies behind the query. ALiBi uses head-specific slopes rather than adding a position embedding to every input token.

Choose slope 0.5 and three candidate keys:

Distance	Content score	Distance bias	Final logit
0	2.0	0.0	2.0
2	2.2	-1.0	1.2
6	3.0	-3.0	0.0

The farthest key has the largest content score but the smallest final logit. These are values before softmax, not probabilities. With only these three candidates, softmax probabilities are approximately 0.631, 0.283, and 0.085, subject to rounding.

The original ALiBi paper reports length-extrapolation results under its experimental setup. Those results motivate testing the approach; they do not guarantee a particular outcome for every dataset or architecture.

Why can longer contexts still fail?

There are several different limits. A model may reject the input because of a configured cap, run out of memory, process the sequence but misread positional relationships, or fail to retrieve a fact among distractors. These are different incidents requiring different remedies.

Consider a model trained mostly on 2,000-token documents. At 16,000 tokens, evidence may be farther away than typical training dependencies. Its position mechanism can produce valid numbers while the attention patterns and task distribution become unfamiliar.

Another problem is evidence competition. Eight similar policy passages may confuse a model even when the correct passage is well within the nominal window. Increasing the maximum length can admit more distracting content rather than improving the answer.

In a diagnostic conversation, ask whether the failure follows total length, answer location, distance between necessary facts, or the number of competing candidates. A single aggregate long-context score cannot distinguish these explanations.

What does positional interpolation actually change?

The positional-interpolation paper explores mapping longer-context positions into a range associated with the model's shorter training window, alongside adaptation. Conceptually, it compresses the position scale rather than asking the original scheme to operate unchanged at much larger indices.

For a toy scale factor of four, original position 8,000 maps to 2,000. Original adjacent positions now differ by 0.25 on the rescaled axis. This may reduce exposure to unfamiliar large positions, but it also changes how local distances are represented.

That trade-off suggests two tests: evaluate long-distance dependencies after extension, and recheck short sequences for regressions. An extension that helps distant retrieval but damages ordinary local tasks may be unacceptable.

Do not present all methods called "RoPE scaling" as identical. Frequency adjustments, interpolation formulas, fine-tuning recipes, and implementation support vary. Specify the exact method before making claims about quality or cache compatibility.

Is a faster attention kernel a context solution?

A more memory-efficient attention implementation can make longer sequences computationally feasible. That is distinct from teaching the model to use those sequences reliably.

For dense attention, doubling sequence length from 4,000 to 8,000 multiplies the number of query-key pairs by four, from 16 million to 64 million per head before considering masking. Causal visibility reduces the number of permitted pairs but preserves the quadratic scaling trend.

A kernel that avoids materializing the full attention matrix can change memory behavior substantially. It does not automatically remove the underlying dense pairwise computation or repair poor retrieval from distant positions.

Use three separate measurements: whether the request fits, how long it takes, and whether the answer is correct. Calling all three "long-context performance" hides the engineering decision. A system can improve the first two and leave the third unchanged.

What can go wrong with position IDs and cached keys?

During autoregressive decoding, the model commonly reuses previously computed keys and values. New tokens must use positions consistent with the cache and the model's positional conventions.

Imagine a four-token prompt occupying positions 0 through 3. The next generated token ordinarily continues at position 4 under a simple zero-based contiguous scheme. If a wrapper mistakenly resets it to zero while retaining cached keys from the prompt, the intended positional relationships are disrupted.

Padding creates another source of mistakes. Physical array indices and logical sequence positions need not coincide, especially when batching sequences of different lengths. Check attention masks and position IDs together.

A targeted test compares a full forward pass against incremental cached decoding on the same short sequence, allowing for normal numerical tolerances. Add unequal-length batched sequences and padding variants. If the discrepancy appears before any long-context extension, investigate cache bookkeeping first.

How should you evaluate usable context length?

Create a test matrix rather than one oversized prompt. Vary total length, answer location, distractor similarity, and the distance between evidence pieces. Include tasks that require combining two facts rather than merely copying one isolated string.

Suppose hypothetical exact-answer accuracy is 95, 94, and 96 percent for beginning, middle, and end positions in short inputs. For long inputs it becomes 92, 61, and 91 percent. The model accepts the long inputs, but the middle-position deficit is operationally important.

The "Lost in the Middle" paper provides evidence that position can affect retrieval performance in evaluated language models. Use it as motivation for your own controlled test, not as a current scorecard for every model.

Also report sample sizes and uncertainty. If each cell contains only ten prompts, small count changes can swing percentages sharply. Keep task difficulty comparable across positions by moving the same evidence and controlling distractors.

How would you defend a design in an interview?

Start with the workload. A code assistant searching related functions, a document summarizer, and a temporal sensor model do not have identical positional requirements. Name the relevant dependencies before choosing a mechanism.

Then state the mechanism at the correct level: "RoPE changes query-key interactions through rotations" or "ALiBi adds head-specific distance penalties." Work through the corresponding example and identify one failure that the mechanism cannot solve.

Finally, propose a falsifiable acceptance criterion. For an illustrative document assistant, require stable answer accuracy across evidence locations at the supported length, plus a latency and memory budget. Define those targets with the product owner rather than borrowing an unexplained industry threshold.

For practice, shift the RoPE positions together, change the ALiBi slope, and move the evidence in the evaluation matrix. Explain what should remain constant, what should change, and what must be measured. That is the difference between recalling architecture names and reasoning about a deployed model.

Sources and Further Reading

Research checked September 24, 2026. Numerical heads, scaling examples, and evaluation percentages are original illustrations.

• Vaswani and colleagues: Attention Is All You Need
• Su and colleagues: RoFormer
• Press, Smith, and Lewis: Train Short, Test Long
• Chen and colleagues: Extending Context Window via Positional Interpolation
• Liu and colleagues: Lost in the Middle