Knowledge Distillation Interview Questions: Teachers, Losses, and Deployment Trade-Offs

Knowledge distillation is often summarized as “train a small student to copy a large teacher.” That sentence is correct and too weak for an interview. A production answer must define what the student copies, how the teacher's mistakes enter training, and whether the resulting model actually meets a latency, memory, cost, and quality budget on target hardware.

This guide builds a classification loss from hard labels and softened teacher probabilities, then evaluates a synthetic deployment decision. For a broader compression prompt, start with Explain Model Compression Techniques and separate distillation from pruning and quantization.

Evidence boundary: Method claims use the original Hinton, Vinyals, and Dean paper, the DistilBERT paper, and current PyTorch documentation. The latency fixture, acceptance table, and interview wording are original preparation material.

What an interviewer is testing

A complete answer resolves six questions:

1. Why can a teacher distribution carry more information than a one-hot label?
2. Which teacher output, hidden state, or generated sequence will the student imitate?
3. How do temperature and loss weights change the training signal?
4. What happens when the teacher is wrong, biased, or outside the student's capacity?
5. Which baselines and ablations establish that distillation helped?
6. Does the student meet the actual deployment budget?

The article-specific thesis is: distillation changes the student's training supervision, not its serving architecture. A better loss may recover quality for a smaller model, but only target-hardware measurement can establish the deployment gain.

From hard labels to soft targets

In ordinary multiclass training, a sample labeled “cat” may provide a target such as [1, 0, 0] for cat, dog, and truck. A teacher may instead output [0.70, 0.25, 0.05]. The non-target probabilities say that this cat resembles a dog more than a truck. Hinton and colleagues call the relative probabilities among incorrect classes useful “dark knowledge.”

For teacher logits z_t, student logits z_s, and temperature T, define softened distributions:

p_t(T) = softmax(z_t / T) and p_s(T) = softmax(z_s / T).

For T > 1, the distribution is softer, exposing more relationships among classes. A common objective is:

L = alpha * CE(y, p_s(1)) + (1 - alpha) * T^2 * KL(p_t(T) || p_s(T)).

The first term keeps the student tied to observed labels. The second encourages it to match the teacher. The T^2 factor compensates for the way temperature shrinks gradients in the softened term; exact weighting conventions vary, so state the implementation rather than reciting alpha without defining which term it weights.

Teacher inference should normally run with gradients disabled and evaluation-time behavior fixed. Accidentally updating the teacher, leaving dropout active, or applying different preprocessing to teacher and student makes the target move. For numerical stability, use the framework's expected log-probability and probability inputs for KL divergence rather than taking logs of rounded probabilities.

Worked question: choose a student under a latency budget

Suppose an image API has these synthetic acceptance criteria:

• p95 latency at batch size 1 at or below 12 ms;
• resident model memory below 150 MB;
• accuracy at or above 89%;
• no critical-slice recall drop larger than two percentage points from the teacher.

An offline test gives:

Model	Training	Size	p95 latency	Accuracy	Critical-slice recall
Teacher	hard labels	900 MB	45 ms	92.0%	90.0%
Student baseline	hard labels	90 MB	8 ms	86.0%	84.0%
Student KD	hard + soft targets	90 MB	9 ms	89.4%	88.5%

The distilled student passes the stated gates; the baseline does not. The result supports deployment for this fixture. It does not show that distillation always recovers three points, that the student equals the teacher on every slice, or that a 10x smaller file is always 5x faster.

The one-millisecond difference between student runs may come from measurement noise because the architecture is identical. Distillation changes weights and training, so its forward-pass cost is generally unchanged unless the training choice also changes architecture or execution. Benchmark warm and cold behavior, p50 and p95, concurrency, batch sizes, preprocessing, memory, power, and cost on the real device.

Choose the teaching signal

Logit or probability distillation

Matching class distributions is the classic setup. It works naturally when teacher and student share the same output space. Tune temperature and loss weights on validation data, and log both hard-label and distillation losses so one cannot silently dominate.

Feature distillation

A student can match hidden representations, attention maps, or intermediate features. This may transfer structure beyond final logits, but mismatched layer dimensions require projections or alignment choices. Every extra loss adds a hypothesis that needs an ablation.

The PyTorch tutorial demonstrates hidden-representation approaches alongside soft-target training. DistilBERT used a triple objective combining language modeling, distillation, and cosine-distance losses during pretraining. That published result is evidence for that model and setup, not a universal recipe.

Sequence-level distillation

For generation, the teacher can produce sequences or token distributions. Training only on one generated answer can reduce output diversity and propagate teacher artifacts. Tokenization must be compatible or mapped deliberately, and evaluation must include task correctness, safety, calibration, and generation behavior rather than teacher similarity alone.

Teacher ensembles and specialists

The original distillation work includes ensemble and specialist-model ideas. An ensemble can provide richer targets but makes target generation expensive. Cache provenance and logits when policy permits; record teacher versions, sampling settings, preprocessing, and transfer-set identity so the student can be reproduced.

Teacher quality sets the ceiling and the hazards

A high aggregate teacher score is insufficient. If its mistakes concentrate on rare classes, languages, devices, or protected groups, imitation can preserve or amplify those failures. Evaluate the teacher before treating it as supervision.

Ask:

• Is the teacher calibrated on the transfer population?
• Does the transfer set cover the student's production inputs?
• Are teacher outputs available under the relevant privacy and licensing terms?
• Do hard labels correct teacher errors, or are they themselves noisy?
• Can the student represent the distinctions expressed by the teacher?
• Are there adversarial or out-of-distribution cases where imitation is unsafe?

A larger teacher is not automatically better. A teacher fine-tuned for a mismatched domain can provide confident but irrelevant targets. Multiple teachers can disagree. Define a conflict policy: combine distributions, weight validated teachers, fall back to labels, or exclude uncertain cases.

Build the experiment so distillation earns credit

Use controlled baselines:

1. teacher evaluated on the same frozen test set;
2. student architecture trained from scratch with hard labels;
3. the same student with the distillation loss;
4. label-only and teacher-only loss ablations;
5. temperature and weight sweeps chosen without touching the test set;
6. a simpler compression alternative when relevant, such as quantization.

Keep initialization budgets and training data comparable. If KD sees additional unlabeled transfer data, report that difference instead of attributing all improvement to soft targets. Run multiple seeds when training variance is material and report confidence intervals or the distribution of results.

Evaluation must include the task loss and deployment decision. For classification, inspect accuracy or cost-weighted metrics, calibration, confusion matrices, and important slices. For retrieval or generation, use task-specific judgments and failure sets. Also test robustness to drift and inputs on which the teacher is uncertain.

Deployment trade-offs beyond parameter count

Parameter count influences storage and weight memory but does not determine latency alone. Operator choice, sequence length, memory bandwidth, kernel support, quantization, compilation, batching, and host-device transfer can dominate. A smaller architecture with unsupported operations can lose to a larger optimized one.

Measure:

• serialized and loaded memory;
• peak activation and cache memory;
• cold start and steady-state latency;
• throughput at expected concurrency;
• power or battery use when relevant;
• accelerator and CPU utilization;
• error rate, calibration, and slice metrics;
• rollback behavior and model provenance.

Deploy the student behind a controlled rollout, compare online outcomes, and retain a rollback path. Teacher agreement can be monitored as a diagnostic on a sampled stream, but the teacher is not ground truth. If both models fail together, agreement looks excellent.

Include preprocessing in that contract. A student exported with different tokenization, image normalization, label ordering, or maximum sequence length is not the model evaluated in the notebook. Package these assets with the weights, run golden-input parity tests across training and serving runtimes, and include preprocessing time in the end-to-end latency measurement. This catches a common failure in which the neural network is faster but the user-visible request is not.

Common wrong answers and precise repairs

Wrong answer	Why it fails	Better response
“The student learns the teacher's labels.”	Soft targets include relative class probabilities.	Define logits, temperature, and each loss term.
“Higher temperature is always better.”	Too much smoothing can erase useful distinctions.	Tune temperature inside validation and inspect gradients.
“A stronger teacher guarantees a better student.”	Domain mismatch, errors, and capacity limits matter.	Validate teacher, transfer data, and critical slices.
“10x fewer parameters means 10x lower latency.”	Hardware and operators determine execution.	Benchmark the packaged model on target hardware.
“Teacher agreement proves quality.”	Teacher and student may share the same mistake.	Evaluate against labels, task costs, and independent slices.
“KD replaces hard labels.”	A combined loss can correct teacher errors.	Ablate hard-only, soft-only, and combined objectives.

A concise interview answer template

Begin: “I will use distillation only if a smaller serving architecture misses the quality gate and a validated teacher provides useful additional supervision.” Then:

1. State the target hardware, workload, quality metric, and latency/memory budget.
2. Validate teacher and transfer-set coverage.
3. Define the hard-label, soft-target, and optional representation losses.
4. Freeze the teacher and version the target-generation pipeline.
5. Compare the identical student with and without KD across seeds and slices.
6. Benchmark the packaged model under realistic load.
7. Roll out with monitoring, provenance, and rollback.

That response connects the loss function to the only outcome that justifies compression: a verified product trade-off.

Sources and Further Reading

• Hinton, Vinyals, and Dean: Distilling the Knowledge in a Neural Network — original soft-target and ensemble distillation treatment.
• PyTorch: Knowledge Distillation Tutorial — soft-target and hidden-representation implementations.
• Sanh et al.: DistilBERT — a documented triple-loss pretraining case study.
• PyTorch torchtune: Distilling Llama 3.1 8B into Llama 3.2 1B — token-level LLM distillation example and ablations.
• PyTorch: KLDivLoss — current input convention and reduction behavior.

Sources checked September 13, 2026. All model sizes, latency figures, quality thresholds, and scorecard results are synthetic.