Articles
Named Entity Recognition Interview Questions: Labels, Spans, and Evaluation
Prepare for named entity recognition interviews with BIO labels, exact-span scoring, nested entities, subword alignment, and grounded extraction examples.
Named entity recognition looks simple until two annotators disagree about where an organization name ends. A model can identify the right general phrase yet fail an exact-span evaluation, or achieve excellent token accuracy by predicting that almost everything is outside an entity.
The direct answer: NER identifies text spans and assigns entity types according to an annotation schema. BIO labels encode flat token spans; exact entity evaluation requires matching both boundaries and type. Nested entities, subword alignment, and generated answers require additional representation and validation decisions.
The examples below are original exercises with explicit toy schemas, not statements about a particular employer's interview or an out-of-the-box model's predictions.
What is the actual NER output contract?
An entity prediction should identify a span in the source text and a label from the agreed schema. A character-offset representation might contain start, end, and type, where the end offset is exclusive. The predicted string should be recoverable from those offsets.
For Maya joined New York Labs., zero-based character offsets place Maya at [0,4) and New York Labs at [12,25). The period is outside the organization span. Naming the offset convention prevents an easy off-by-one disagreement.
Do not confuse recognition with entity linking. Recognizing Maya as a person does not determine which person in a database she is. Coreference is another separate task: connecting she to Maya is not implied by assigning an entity type.
The contract should also say whether spans can overlap, whether labels are mutually exclusive, and what happens when no supported entity is present. Those decisions shape both annotation and architecture.
How do BIO tags encode a span?
In the BIO scheme used here, B begins an entity, I continues that entity, and O marks a token outside any entity. The type suffix distinguishes people, organizations, locations, and other schema-specific classes.
| Token | Label | Explanation |
|---|---|---|
| Maya | B-PER | Start a person |
| joined | O | Outside |
| New | B-ORG | Start an organization |
| York | I-ORG | Continue the organization |
| Labs | I-ORG | Continue the organization |
| . | O | Outside |
This example assumes the entire company name is one organization and does not separately annotate the location words inside it. A different schema could make a different decision, but a dataset must be consistent.
Ask how invalid sequences are handled. An I-ORG after O does not form a valid continuation under strict BIO2 rules. A decoder may prevent it, or postprocessing may repair it according to a documented policy. Silent repairs can change evaluation, so preserve the raw predictions when diagnosing errors.
Why do annotation rules matter more than label names?
An organization label alone does not settle whether to include a leading "the," a legal suffix, punctuation, or a department name. The rules must show examples and counterexamples.
Consider the Northstar Research Group's office. Does the organization include "the"? Does the possessive ending belong to the span? There is no universal answer independent of the dataset's policy. Mixing decisions teaches the model conflicting targets.
spaCy's EntityRecognizer documentation explicitly notes the importance of boundary agreement for its whole-entity objective. That is a practical warning: more model capacity cannot repair a gold dataset whose boundaries are inconsistently defined.
Before training, ask two annotators to label a representative sample independently. Classify disagreements as boundary, type, missing entity, or schema ambiguity. Update the guide and relabel affected examples rather than averaging incompatible annotations. The result is a more defensible target, even if the first reported score temporarily falls.
How do you calculate exact-span precision and recall?
Suppose the gold entities are Maya: PER, New York Labs: ORG, and Paris: LOC. The model predicts Maya: PER, York Labs: ORG, and Paris: LOC.
Only two predictions match both gold boundaries and labels. Therefore true positives equal 2, false positives equal 1, and false negatives equal 1. The shortened organization prediction is a false positive; the missing complete organization is a false negative.
Precision is 2 / (2 + 1), and recall is also 2 / (2 + 1). Both are approximately 66.7 percent, as is F1. A boundary error can therefore contribute to both error counts without being "double-counted" incorrectly: precision and recall ask different questions.
State the matching rule before presenting the number. A relaxed overlap metric might credit the shortened span, but that answers a different question. For database population or redaction, exact boundaries may matter more than approximate semantic overlap.
Why is token accuracy often misleading?
Imagine 1,000 tokens, of which 950 carry O labels. A system that always predicts O reaches 95 percent token accuracy while finding no entities at all. Its useful entity recall is zero.
Entity-level micro F1 aggregates true positives, false positives, and false negatives across types. Macro averaging gives each type equal weight after computing per-type scores, subject to the evaluator's handling of empty classes. These summaries emphasize different properties.
For an operational report, include entity support counts and per-type performance. A frequent ORG class can dominate micro results while a rare account-number class remains unreliable. If the application depends on that rare class, the aggregate cannot justify deployment by itself.
Keep document-level outcomes too. A form with five required entities is usable only if the necessary fields are correct under the product's rules. Strong average span F1 does not directly reveal the proportion of fully correct forms.
How should subword labels align with words?
A transformer tokenizer may split one annotated word into several subwords. The training pipeline must map word-level annotations to those pieces without accidentally shifting labels.
Hugging Face's token-classification guide demonstrates using word IDs to align labels, ignoring special tokens and repeated subword pieces with a loss-ignore value. Another policy can label every subword, but then continuation tags and evaluation aggregation must be handled consistently.
Take a toy word Northstar split into North and star. If the word begins an ORG entity, assigning B-ORG to both pieces can create two entities when decoded naively. A policy that supervises only the first piece avoids that particular mistake, while an all-piece policy needs an appropriate continuation rule.
Inspect one encoded example manually: original words, subword strings, word IDs, special tokens, labels, and attention mask. That small alignment table often reveals a problem that a long training run would only amplify.
Can a flat tagger represent nested entities?
A single BIO tag per token cannot simultaneously represent arbitrary overlapping spans. Consider a schema that requires University of California as ORG and California inside it as LOC. The word California would need to participate in two entities.
spaCy's standard EntityRecognizer produces non-overlapping spans in Doc.ents. Its SpanCategorizer supports span categorization, including overlapping candidates, through a different representation. Changing the component does not eliminate the need to generate suitable candidate spans and supply appropriate labels.
The first question is whether the task actually requires nesting. For a company-name extraction product, keeping only the outer organization may be correct. For a linguistic corpus, both annotations may be required.
Also distinguish nesting from discontinuity. An entity made of separated text segments is not just an ordinary contiguous span containing another span. It needs an output format that can represent multiple segments and an evaluator that understands them.
When should you use rules, a tagger, or an LLM?
Choose according to the schema, data, and operational constraints rather than treating newer models as automatic replacements.
| Approach | Good starting situation | Main validation concern |
|---|---|---|
| Rules or dictionaries | Stable identifiers and narrow vocabularies | Ambiguity and unseen variants |
| Supervised sequence tagger | Consistent labels and sufficient examples | Domain shift and boundary errors |
| Span-based model | Overlapping or nested targets | Candidate coverage and span complexity |
| Prompted language model | Flexible schema or limited initial labels | Grounding, formatting, latency, and consistency |
A hybrid can be reasonable. A rule identifies a structured ticket ID, while a learned model identifies organizations in unrestricted prose. Define conflict resolution explicitly instead of allowing whichever component runs last to overwrite the other.
Evaluate every approach on the same held-out documents and output contract. A prompted model that returns plausible names without valid offsets is not solving the exact same task as a span extractor until those outputs are grounded and scored.
How do you validate an LLM's extracted entities?
Require outputs that can be checked against the source. If the model returns a string, verify that the string occurs, determine which occurrence it refers to, and reject unsupported or ambiguous grounding under the chosen policy.
Suppose a document says Maya called Maya Chen. Returning only Maya without an occurrence index is ambiguous. Searching for the first matching substring is not a principled solution if the model intended the second occurrence.
Offsets also need validation. Confirm that start is less than end, boundaries lie within the text, and slicing recovers the expected entity. Check whether the model's offset convention matches the application's character indexing, particularly for Unicode text.
A well-formed JSON object can still contain a fabricated organization or the wrong label. Separate syntax validity, source grounding, boundary correctness, and type correctness. Then compare those failure rates with the supervised baseline instead of reporting a single percentage of parseable responses.
What causes an NER model to fail after deployment?
A common cause is domain shift. A model trained on news may encounter product codes, usernames, abbreviated organizations, or unusual casing in support tickets. Another is a preprocessing change that modifies offsets while leaving annotations tied to the original text.
Imagine replacing repeated whitespace before inference but applying returned offsets to the unmodified document. Every entity after a changed region can be misplaced even when the model recognized the correct text. Keep a mapping or ensure extraction and display use the same representation.
Split evaluation data by source and time where appropriate. Repeated templates or duplicated documents across training and test sets can inflate scores. Hold out unfamiliar naming patterns to learn whether the system generalizes beyond memorized strings.
For error review, retain enough context to distinguish a missed name from a schema disagreement. Prioritize errors according to the application: an incomplete organization span may be tolerable for search but unacceptable when the output drives an exact record match.
How would you answer a full interview case?
Start by restating the entity schema and output representation. Ask whether nested entities are required, how offsets are defined, and which mistakes are most costly. These are modeling inputs, not administrative details.
Next, annotate one sentence and walk through its BIO sequence. Show how subwords will align and what the decoder does with invalid transitions. Calculate exact-span precision and recall on a tiny prediction set before discussing a large benchmark.
Then propose an experiment that separates model limitations from annotation and pipeline errors. A reasonable first iteration compares a simple baseline and a learned model on adjudicated data, with per-type scores and a boundary-error audit.
For practice, change the example from New York Labs to a name containing punctuation, then add a nested location and a repeated person name. Explain which changes require a new representation and which merely require better examples. A strong answer preserves a clear contract all the way from raw text to the metric used for a decision.
Sources and Further Reading
Research checked September 24, 2026. Sentences, offsets, and score calculations above are original teaching examples.