Mixture-of-Experts Interview Questions: Routing, Load Balance, and Communication

Mixture-of-Experts interview questions expose a common shortcut: sparse activation can reduce computation per token, but routing and communication can still dominate the system. A good answer does not begin with a fashionable tool name. It begins with a contract, draws the state boundaries, and shows how failure changes the design.

This guide gives you a reusable interview method plus concrete traces for mixture of experts interview questions.

Mixture-of-Experts Interview Questions: Quick Answer

Trace one token from router logits to selected experts, capacity admission, all-to-all dispatch, expert compute, and combine. Separate model capacity, active parameters, FLOPs, memory, and interconnect traffic.

Official documentation: Switch Transformer uses top-1 routing and defines expert capacity from tokens per batch, number of experts, and a capacity factor. Expert Choice Routing reverses selection so experts choose tokens, addressing imbalance in token-choice routing. These papers establish mechanisms and experiments; they do not guarantee a production speedup on every cluster.

Candidate reports: We did not find a reliable, same-cycle set of public candidate reports establishing a standard interview format for this topic.

Engineering inference: The interview structure and recommendations in this guide are derived from the documented semantics and common failure modes. They are not an employer-wide hiring rule.

Draw the state boundaries first

Boundary	What it contains	Interview consequence
Router	Scores experts for each token	Top-k selection and gating weights
Capacity	Maximum tokens accepted by an expert	Overflow is dropped, rerouted, or buffered
Dispatch	Groups tokens by destination device	Creates all-to-all communication
Combine	Returns and weights expert outputs	Restores original token order

This table is the first article-specific asset. Point to a row whenever the interviewer changes the scenario. If a value crosses a boundary, say how it is identified, serialized, authorized, versioned, and cleaned up. If it does not cross, say why the process can safely reconstruct it.

A boundary is also where partial failure becomes visible. The caller may know it sent a request while the receiver knows it committed a change; neither observation alone proves the end-to-end outcome. Keep attempted action, durable result, and later verification as separate facts.

Turn the prompt into an explicit contract

Before proposing an implementation, ask what must be correct and what may be approximate, delayed, retried, or discarded. Name the unit of identity, the lifetime of the state, the isolation boundary, and the evidence that counts as success. These questions prevent a polished solution from answering a different problem.

A useful answer also names ownership. Who creates the state? Who may mutate it? Who cleans it up? What happens when the process, request, or connection ends early? Interviewers often add a failure after the happy path; an ownership model lets you extend the design without replacing it.

Explain the invariant before the mechanism

The invariant is the sentence that must remain true after every transition. State it before code. Then walk one normal case, one boundary case, and one failure case. This makes the answer reviewable: the interviewer can see why a line exists and which assumption would invalidate it.

Avoid claiming a mechanism is safe or fast by name alone. A cache can miss, a queue can duplicate, a lock can block, and an async task can fail unobserved. Connect the mechanism to the stated contract and include the metadata or measurement needed to prove it at runtime.

Work through a concrete scenario

With 4,096 tokens, 64 experts, top-1 routing, and capacity factor 1.25, nominal capacity is 80 tokens per expert: 4096 / 64 × 1.25. Uniform routing fits; a hot expert receiving 140 tokens overflows by 60 under a hard cap. The candidate should say what happens to those tokens and what quality or latency cost follows.

Do the arithmetic or state trace aloud. Label every number that is an assumption and every behavior that comes from documentation. If the interviewer changes scale, platform, or failure semantics, update only the affected term. This shows that you understand the model instead of memorizing a configuration.

The supporting visual is the second topic-specific asset. Read it left to right: establish identity and preconditions, perform one bounded transition, collect the result, and verify durable state. When the result is unknown, prefer a read or reconciliation step before repeating a side effect.

Show a small implementation skeleton

logits = router(hidden)                 # [tokens, experts]
expert_id = logits.argmax(dim=-1)       # top-1 route
capacity = ceil(tokens / experts * capacity_factor)
position = cumulative_count(expert_id)
accepted = position < capacity
dispatched = all_to_all(pack(hidden[accepted], expert_id[accepted]))
output = combine(expert_forward(dispatched))

This is deliberately a skeleton, not production-ready code. In an interview, explain which functions are pure, which perform I/O, which values carry identity, and where errors cross the boundary. Mention tests before adding framework details.

The most important code review question is: “What fact makes the next transition safe?” If the answer is only “the previous call returned,” add validation. If the action can be retried, define whether the same identity repeats the same effect, resumes work, or creates a new attempt.

Rehearse one failure before adding complexity

Start with the smallest version of the system that can demonstrate the invariant. Record the state before the action, introduce say sparse means cheap, and observe the concrete failure: total weights still consume memory and routing adds traffic. Apply the narrow repair—separate active flops from total parameters and bytes—then repeat the same input. The before-and-after evidence should differ only at the repaired boundary.

Next inject a second, independent fault: average load looks balanced. The expected failure is one expert can overflow while the mean is fine. Repair it by following this rule: Inspect per-expert histograms and tail load. This two-step exercise shows whether the design composes. If the first repair masks the second failure, add an observation point rather than assuming the second path ran.

In a live interview, narrate the distinction between prevention, detection, and recovery. Prevention rejects an invalid transition before it changes state. Detection proves that an attempted transition produced an unacceptable result. Recovery moves from that observed result to a known state. A single mechanism rarely supplies all three.

Also name what you would leave out of the first implementation. Extra brokers, caches, services, or abstractions create more state transitions and more failure combinations. Add one only when a stated throughput, isolation, ownership, or recovery requirement needs it. That trade-off demonstrates judgment without hiding behind an oversized architecture.

Diagnose the misleading answers

Tempting answer	Failure	Better interview response
Say sparse means cheap	Total weights still consume memory and routing adds traffic	Separate active FLOPs from total parameters and bytes
Average load looks balanced	One expert can overflow while the mean is fine	Inspect per-expert histograms and tail load
Increase capacity indefinitely	Padding and memory grow	Sweep capacity factor against drop rate and throughput
Ignore topology	All-to-all may cross slow links	Place experts with network bandwidth in the model

These counterexamples are more useful than a long list of trivia. For each one, reproduce the smallest failing case, name the violated invariant, repair the narrow boundary, and rerun the check. This approach also keeps you from changing several variables at once.

Measure the behavior you actually promised

Measure per-expert token histograms, coefficient of variation, overflow rate, router entropy, auxiliary loss, all-to-all bytes, exposed communication time, tokens per second, and quality. Compare against a dense baseline at a clearly stated compute or parameter budget.

Do not report a percentage improvement without naming the workload, baseline, sample window, and measurement point. Separate average behavior from tail behavior and separate task failures from infrastructure failures. A design can lower average latency while making recovery or isolation worse.

For an interview experiment, use a deterministic fixture with a fixed seed and record inputs plus versions. Run the baseline and proposed design on the same fixture. Add one fault at a time—timeout, duplicate, stale state, invalid identity, cancellation, or capacity overflow—and assert both the immediate result and the durable state afterward.

Use this answer structure in the interview

1. Restate the contract. Define input, output, identity, lifetime, failure semantics, and scale.
2. Name the invariant. Give one sentence that must survive every transition.
3. Draw boundaries. Separate caller, worker or service, durable state, and external side effects.
4. Walk one example. Use concrete values and show the state before and after each step.
5. Inject failure. Choose timeout, crash, duplicate, stale data, or cancellation and recover without guessing.
6. Measure and test. Name observable counters, a baseline, and an adversarial test.
7. State the trade-off. Explain what your design optimizes and what it deliberately spends.

This sequence is an editorial recommendation based on the mechanics above. It is not an official interview rubric. Its advantage is that every claim can be challenged locally: the interviewer can change a constraint and you can revise the affected boundary instead of rebuilding the answer.

Questions interviewers may ask next

What should I clarify before writing code?

Ask about correctness, ordering, identity, concurrency, platform or provider version, memory and latency limits, cancellation, security scope, and how success is verified. If the prompt omits one, state a reasonable assumption and make it easy to change.

How do I distinguish a fact from an inference?

A fact should point to current official documentation or a reproducible program result. A candidate report describes one person’s prompt or experience. An inference connects those facts to a design recommendation. Label the category when the claim first appears, especially for changing platform behavior.

What makes an answer senior-level?

Senior answers make ownership and failure explicit. They identify the source of truth, define what happens after an unknown result, protect tenant or user boundaries, and propose measurements that could disprove the design. More components do not automatically make the answer stronger.

Should I memorize every API detail?

No. Memorize the state model and a few diagnostic examples. State version-sensitive details carefully, cite current documentation when writing, and say how you would verify an uncertain API during real work.

Final preparation checklist

• Define the unit of identity and the source of truth.
• Separate current observation from durable state.
• State the main invariant before code.
• Trace one normal, boundary, and failure case.
• Explain ownership, cleanup, cancellation, and retries.
• Preserve tenant or user authorization at every boundary.
• Name version-sensitive claims and their official source.
• Measure the promised outcome and its failure surface.
• Avoid turning a candidate-reported prompt into an employer-wide claim.
• End with one clear trade-off and one test that could falsify your design.

Sources and Further Reading

• Switch Transformers, JMLR
• Expert Choice Routing