Articles
LightGBM vs CatBoost in ML Interviews: Categorical Features, Leakage, and Fair Benchmarks
Prepare for LightGBM vs CatBoost interviews with a four-row target-leakage case, category semantics, unseen-category checks, and a fair benchmark ledger.
“CatBoost handles categories, so it wins on categorical data” is not a fair model comparison. The real interview problem is to define what information was available at prediction time, represent categories without target leakage, and give both libraries a comparable search and evaluation budget. LightGBM and CatBoost make different engineering choices, but neither rescues an invalid split. This guide uses a four-row leakage example and a benchmark ledger to show what a defensible answer looks like.
Evidence boundary: Library behavior below is attributed to LightGBM's official advanced topics, CatBoost's categorical-feature documentation and CatBoost's FAQ. The fixture and benchmark design are original editorial exercises, not candidate reports or measured win rates.
Start with the prediction contract
Question: Which model should you choose for a table with categorical columns? First ask what the target is, what a row represents, how features arrive at serving time, and which outcome metric matters. A customer-risk dataset with repeated customers, delayed labels and a chronological launch needs a different split from an independent, identically distributed benchmark table. Without that contract, “which library is better?” has no stable meaning.
The first technical distinction is categorical handling. LightGBM documents native categorical feature support when categorical columns are identified appropriately; it need not receive a giant one-hot matrix by default. CatBoost documents categorical features and explicitly advises against user-applied one-hot preprocessing as a general recipe because it changes its own categorical processing. Both support categories, but the encodings, split construction and hyperparameters differ. Avoid the false comparison “LightGBM only accepts numbers and CatBoost alone accepts categories.”
The second distinction is evaluation. A defensible comparison holds the rows, labels, cutoffs, metrics and feature availability fixed. It lets each library use a valid representation according to its documentation, then spends a comparable tuning budget. Forcing identical parameter names and values is not fairness when those parameters have different meanings.
Work a target-leakage example before naming a library
Original four-row exercise: Predict whether an event becomes positive. A categorical feature is merchant, and the prediction must be made when the event occurs. The chronological data are:
| Time | Merchant | Label when eventually known | Use |
|---|---|---|---|
| t1 | A | 0 | Training |
| t2 | A | 1 | Training |
| t3 | B | 0 | Training |
| t4 | A | 1 | Future validation |
The training-only target mean for merchant A is (0+1)/2=0.5. If a careless preprocessing step computes the merchant mean on all four rows before splitting, A becomes (0+1+1)/3≈0.667. That feature now contains the t4 validation label. A high validation score after this transformation is contaminated regardless of whether the downstream estimator is LightGBM, CatBoost or logistic regression. The calculation is a small editorial fixture, not a claim about either library's default behavior.
Even calculating a target mean on the training partition can leak each training row's own label into its feature. A training row with a rare merchant may effectively receive its own outcome as a feature. Safer external target encoding uses out-of-fold or ordered historical statistics within the training data, and applies a mapping learned from permissible training history to validation and serving rows. Which scheme is valid depends on whether the task has a time ordering and when labels become available.
CatBoost's FAQ on ordered categorical statistics explains how its training construction uses preceding examples in a permutation to reduce target leakage and prediction shift. That is a useful mechanism to discuss. It does not mean an arbitrary train/validation split or a future feature derived from unavailable labels becomes safe. CatBoost's has_time parameter governs use of input order during relevant training stages, but your outer temporal split, feature cutoff, and delayed-label rules remain your responsibility.
Explain categorical features without stereotypes
Question: How does LightGBM use a categorical variable? The official LightGBM guide says categorical feature values should be non-negative integers within its supported range and describes a partition-based split algorithm. It also notes that negative values are treated as missing and that high-cardinality categories may work better under other representations, depending on the problem. A data frame's category dtype or explicit categorical feature declaration matters; a numerical integer code silently treated as continuous is a different experiment.
Question: How does CatBoost use one? CatBoost can use one-hot handling for categories under particular conditions and create numerical statistics for others, as its categorical-feature documentation explains. The defaults vary by task and hardware, so a statement such as “CatBoost always target-encodes every category” is wrong. Record categorical columns and the relevant configuration rather than relying on a slogan.
Both methods still need a stable representation between training and serving. Decide how missing categories, changed spelling, new merchant IDs and rare values are passed to each library. If you pre-encode for one library, record who fitted the encoder and which rows influenced it. A high-cardinality ID can be genuinely predictive but also serve as a proxy for repeated entities leaking across a random split. Check performance for unseen entities separately from seen ones.
Design a fair benchmark ledger
The following table is a plan for an interview or real experiment, not a reported benchmark result. It makes the comparison auditable.
| Contract | Shared requirement | Library-specific implementation allowed |
|---|---|---|
| Data | Same raw rows, target definition, exclusions and temporal or group split | Supported categorical input format for each library |
| Features | Same information cutoff and feature list | Internal categorical treatment and missing-value handling |
| Objective | Same prediction task and primary metric | Equivalent library objective syntax |
| Search | Comparable number of trials or wall-clock budget, same validation policy | Different parameter spaces and sensible ranges |
| Early stopping | Same validation window, metric and patience rationale | Library-specific callback syntax |
| Reporting | Test metric, uncertainty, latency, memory and serving constraints | Distinct model sizes and training implementation |
Do not compare the best CatBoost run after 100 trials with a LightGBM default fit. Do not tune either model on the held-out test set. If one model's native categorical handling reduces preprocessing, that is a legitimate engineering advantage to report, but it should not be smuggled in as a different feature set. If a custom encoding is used for LightGBM, include its fitting cost and leakage controls in the benchmark.
The word “fair” is contextual. Equal trial counts and equal wall-clock time are different constraints. A production team with a fixed training window may prefer equal time; a research comparison may report both time and quality across a budget curve. Random seeds, hardware and thread counts can affect timing. A single run does not establish a universal speed or accuracy ranking.
Choose metrics and split units that match deployment
Question: Which score decides the winner? For a rare-event classifier, PR-AUC, recall at a review-capacity limit, expected cost or calibrated risk may be more informative than raw accuracy. For ranking, use an appropriate ranking metric and group construction. For regression, MAE versus RMSE changes which errors dominate. Define the primary metric before tuning and report secondary operational metrics separately.
In the merchant fixture, a random row split allows merchant A in both training and validation. If the business wants predictions for new merchants, that split overstates the capability of any model that memorizes merchant history. Hold out merchants for a new-entity evaluation. If the business wants future events from known merchants, use a temporal split and ensure only historically available labels and aggregates enter each row. A single validation design may not answer both questions; report the two scenarios explicitly.
Review delayed labels. A transaction at t4 might not have a confirmed fraud outcome until t5. An aggregate computed at t4 cannot legally include that outcome just because it appears in a retrospective warehouse table. Write the availability time for each feature, not only its event timestamp. This distinction matters more than the choice between two boosting libraries.
Diagnose a surprising benchmark result
Suppose CatBoost wins validation by a large margin but loses on a fresh month. First inspect the split and all target-derived encodings. Look for duplicate customers, merchants or sessions crossing the boundary; features created from the eventual outcome; and early stopping evaluated on the wrong window. Then compare categorical cardinality, unseen-category rate and label prevalence by month. Only after the data contract passes should you attribute the gap to model behavior.
Suppose LightGBM trains much faster in one run. Record row count, feature count, threads, CPU/GPU mode, preprocessing, trial budget and early-stopping rounds. CatBoost may be running a different mode or doing more categorical work. Speed is an observed measurement under a defined setup, not a library identity. Conversely, if CatBoost is faster to ship because it removes a fragile external encoding step, distinguish engineering time from fit time.
If either model's training score is far better than validation, inspect tree complexity, category memorization and the chosen split. LightGBM's leaf-wise growth and CatBoost's default tree structure are relevant mechanisms, but “one always overfits more” is not an established conclusion. Tune each responsibly under the same evaluation contract.
An interview-ready comparison answer
A strong answer might be: “I would start with a chronological holdout because the model predicts future merchant events. I would list feature availability times and keep outcome-derived aggregates within training history. I would pass categories in each library's documented format, use the same raw features and primary metric, give both a comparable wall-clock or trial budget, and report test quality with latency and memory. CatBoost's ordered statistics can help with categorical target leakage inside training, but do not replace the outer time split. LightGBM also supports native categorical features, so I would benchmark both rather than assume one wins.”
That answer is defensible because the four-row example exposes a concrete invalid feature and the ledger exposes an unfair comparison. If asked for a winner without data, say what evidence would make you choose. A candidate who names the test before the library is showing better judgment than one who lists architecture trivia.