Articles
Pandas Categorical Interview Questions: Unobserved Groups, Ordering, and Missing Values
Practice pandas categorical interview questions with tested observed/dropna outputs, category ordering, missing values, unknown labels, and version checks.
A pandas report shows Enterprise customers with zero tickets, omits a ticket whose tier is missing, and counts two Pro tickets but only one measured handling time. Those outputs can all be correct under different contracts. Pandas categorical interview questions ask you to explain the category universe, observed groups, missing keys, and measurement denominator separately.
This article is for data analyst and Python candidates. Official documentation establishes API behavior; the five-ticket fixture and recommendations are original practice. No candidate reports establish employer frequency or an exact interview format. The examples were executed locally with pandas 3.0.5; documentation checked on October 3, 2026 displays version 3.0.6. Those are different verification scopes.
What does a categorical contain beyond its values?
A categorical carries a set of allowed categories and an ordering flag alongside its values. Unused categories can remain part of that schema even when no input row contains them. A missing value is possible without becoming an additional category.
Official API contract: CategoricalDtype requires unique, non-null categories and specifies whether they have an ordered relationship. See the dtype reference. For this exercise, the domain is Basic, Pro, Enterprise, in that business order.
Do not infer the schema independently from each filtered dataset when the report must compare the same tiers across months. A month without Enterprise observations does not necessarily remove Enterprise from the business domain. Conversely, a report requiring only observed tiers should not imply that an empty category had actual customers.
The neighboring pandas analyst guide covers joins and general cleaning. This exercise focuses on the metadata that changes grouping and ordering even when the underlying ticket rows stay identical.
Predict a five-ticket result
Create the original fixture below. Each row is one ticket. Minutes can be missing, but a genuine zero means a measured duration of zero; it is not another spelling of missing.
import pandas as pd
TIER = pd.CategoricalDtype(
["Basic", "Pro", "Enterprise"], ordered=True
)
df = pd.DataFrame({
"ticket_id": [1, 2, 3, 4, 5],
"tier": pd.Series(
["Pro", "Basic", "Pro", None, "Basic"],
dtype=TIER,
),
"minutes": [10, 20, None, 40, 0],
})
report = df.groupby(
"tier", observed=False, dropna=False, sort=True
).agg(
tickets=("minutes", "size"),
known=("minutes", "count"),
total=("minutes", "sum"),
mean=("minutes", "mean"),
)
The tested result is:
| Tier | Tickets | Known times | Total minutes | Mean minutes |
|---|---|---|---|---|
| Basic | 2 | 2 | 20 | 10 |
| Pro | 2 | 1 | 10 | 10 |
| Enterprise | 0 | 0 | 0 | NaN |
| Missing key | 1 | 1 | 40 | 40 |
The printed missing-key index is NaN; “Missing key” is an explanatory label in this table, not a declared category. The ticket counts sum to five, and known-time counts sum to four. Those are different reconciliations.
Basic's zero-minute ticket participates in its mean, so 20 divided by two is 10. Pro's missing measurement does not contribute a known time, so 10 divided by one is also 10. Equal means conceal different missingness. Report the denominator when that distinction matters.
How do observed and dropna differ?
Official behavior: observed controls whether categorical groupers emit only observed category values or the full category set. dropna separately controls exclusion of missing grouping keys. Since pandas 3.0, observed=True is the default. The GroupBy API documents both options and the default change.
With .size() on the same fixture, our four explicit combinations yield:
| observed | dropna | Emitted groups and ticket counts |
|---|---|---|
| True | True | Basic 2; Pro 2 |
| False | True | Basic 2; Pro 2; Enterprise 0 |
| True | False | Basic 2; Pro 2; missing 1 |
| False | False | Basic 2; Pro 2; Enterprise 0; missing 1 |
Setting observed=False does not restore the missing-tier ticket when dropna=True. It restores an unused declared category, which is a different absence. Setting dropna=False does not invent Enterprise observations.
Specify both options in interview code when the output contract depends on them. Saying “the default includes all categories” describes older behavior, not the tested 3.0 environment. A reproducible notebook should record the actual pandas version and keep expected groups in its assertions.
The inclusion rule belongs to the report. An operational summary may need only observed groups; a dashboard with a fixed tier axis may need the full domain. Neither choice should change how many source tickets were genuinely processed.
Does a zero total prove there were observations?
No. In the fixture, Enterprise has zero tickets and a zero default sum, while Basic contains a real zero-minute measurement. These states have different meanings even though a chart might draw the same height.
Official aggregation option: GroupBy.sum accepts min_count; insufficient nonmissing values produce NA instead of a sum. See the sum reference. For a measurement policy requiring at least one known value, compute:
totals = df.groupby(
"tier", observed=False, dropna=False
)["minutes"].sum(min_count=1)
In the tested fixture, Enterprise's total becomes NaN. Basic remains 20, Pro remains 10, and the missing-key group remains 40. This changes the measurement interpretation without changing the grouping universe or source rows.
A further counterexample is a single observed ticket whose minutes are missing. Its group has size one, known count zero, default sum zero, and sum with min_count=1 equal to NaN. Empty group, unknown measurement, and measured zero therefore need separate indicators.
Does ordered=True force GroupBy output order?
An ordered category defines a semantic ranking. It does not override every display-order choice. Official version note: since pandas 2.0, sort=False with an ordered categorical grouper no longer forces category-order sorting. See the sorting parameter documentation.
In our 3.0.5 fixture, grouping with observed=True, dropna=False, sort=False emits Pro, Basic, then the missing key, following encounter order. With sort=True, Basic precedes Pro under the declared category order. These are grouping-output choices, not changes to ticket membership.
Sorting the ticket values uses the categorical ranking; it does not alphabetize labels automatically. That matters if a later report renames tiers without changing their semantic order. A lexically convenient label is not evidence of the desired business ranking.
Reordering and renaming are also different operations. Reordering Basic and Pro changes comparisons and future sorts while retaining the individual labels. Renaming changes the labels. Explain which property must remain stable before choosing a .cat method.
A useful follow-up asks for the maximum tier observed in this fixture. It is Pro, not Enterprise: Enterprise belongs to the schema but has no observed value. A category's presence in metadata is not evidence that a source ticket had that tier.
What happens to an unexpected label?
Consider raw values Basic, VIP, and a genuine missing entry. In local pandas 3.0.5, casting directly to our TIER dtype turns VIP into missing and emits Pandas4Warning. The genuine missing entry is also missing afterward, so the cast loses the distinction unless you retained the raw values.
Official deprecation: pandas 3.0 release notes deprecate constructing or casting a categorical with non-NA values outside the specified category set. Do not teach silent coercion as a future-stable validation strategy. See the 3.0 release notes.
Use an explicit pre-cast policy instead:
raw = pd.Series(["Basic", "VIP", None])
invalid = raw.notna() & ~raw.isin(TIER.categories)
assert invalid.tolist() == [False, True, False]
# Exercise policy: quarantine or flag invalid labels,
# then represent them as missing in the typed output.
typed = raw.mask(invalid).astype(TIER)
The flag makes our chosen policy visible. Another prompt may require rejection or a governed new category. Lowercasing everything, adding every new label, or erasing invalid values without an audit trail are business decisions, not automatic improvements.
Assignment has a separate boundary: directly assigning VIP into an existing TIER column raises TypeError in the tested version. If VIP is approved, expand the category schema before assigning it. If it is a typo, correct it under an explicit mapping rather than expanding the domain accidentally.
Can missing values become an Unknown category?
Yes, if Unknown is a deliberately defined reporting value. Official API behavior: add_categories extends the category set without rewriting existing values. See adding categories.
labelled = df["tier"].cat.add_categories(["Unknown"])
labelled = labelled.fillna("Unknown")
Ticket 4 now belongs to an observed Unknown category. That differs from retaining an NA grouping key with dropna=False. Appending Unknown also gives it a position in an ordered schema; choose that position deliberately before using comparisons or presenting a ranked chart.
Official API behavior: remove_unused_categories removes levels absent from values. See removing unused categories. It removes Enterprise from our baseline schema, so a later full-domain report cannot recover Enterprise merely by choosing observed=False.
Removing a used category is more consequential: the tested removal of Pro turns tickets 1 and 3 into missing values. The missing-key group grows from one ticket to three. That changes the data interpretation, not just storage. Keep schema cleanup distinct from deleting or recoding observed labels.
Why are category codes unsafe as durable identifiers?
The fixture's category codes are [1, 0, 1, -1, 0]. Official representation: missing values use code -1 and do not belong in the category list. The categorical guide explains this distinction.
Codes depend on the associated category mapping. Our local check constructs one batch with categories Basic, Pro and another with Pro, Basic. Code zero means Basic in the first and Pro in the second. Joining batches on raw codes would merge different tiers.
For model inputs, share the fitted vocabulary and its unknown-value policy across training and inference. For business joins, prefer governed labels or stable domain IDs. Converting a column to categorical does not automatically implement statistical encoding or prevent leakage.
The same schema issue appears during serialization. A CSV contains values rather than the complete categorical dtype contract; the official guide discusses reconstructing categories and ordering after reading it. Our recommendation is to save the governed schema alongside exported data when downstream calculations depend on it.
What should you test beyond the single-column example?
Add a categorical region with East and West. In our local fixture, observed combinations are Pro/East and Basic/West. Grouping both columns with observed=False, dropna=True produces six category combinations, four with zero tickets. With observed=True, it produces only the two observed combinations.
This is useful for a complete reporting grid, but it can expand output substantially as domains multiply. Estimate the intended domain product before materializing it. A missing or impossible combination is not automatically an observed zero outcome.
Our checks cover the four grouping settings, denominator and total reconciliation, missing-preserving sums, display order, codes, explicit invalid-label flags, new-label assignment, category addition/removal, and two-column expansion. They execute the original fixture with pandas 3.0.5; they are not a memory benchmark or a claim that every reducer or older version behaves identically.
Sources and Further Reading
- pandas: CategoricalDtype — category schema and ordering flag.
- pandas: DataFrame.groupby — observed, dropna, sort, and version notes.
- pandas: categorical data guide — codes, missing values, and serialization.
- pandas: GroupBy.sum — min_count semantics.
- pandas: 3.0 release notes — unsupported-label construction/casting deprecation.
- pandas: add_categories — schema expansion before assignment or filling.
- pandas: remove_unused_categories — removing unobserved levels.
Evidence checked October 3, 2026. Documented current behavior and local 3.0.5 results are labeled separately; no employer-specific candidate evidence is claimed.