Trust and Safety Data Scientist Interview Case: Estimate Harmful-Content Prevalence

An interviewer says, “Our moderation team removed more harmful posts this month. Did users see less harmful content?” A count of removals cannot answer that question. It measures what the system found and acted on, while the interviewer is asking what people encountered. The central skill in this trust and safety data scientist case is to define that gap before proposing a model or a dashboard.

This is a worked interview case, not a report of one company's actual interview prompt. It uses Meta's published prevalence explainer and an August 2026 Pinterest-affiliated research preprint as methodological evidence. Both describe view-based measurement, but their implementations are specific to their platforms. For a related hands-on SQL exercise, Compute Moderation View Metrics asks you to avoid counting the same views twice.

Start by naming the estimand

Ask what “prevalence” should mean for this decision. A useful candidate definition is the fraction of eligible content impressions in a specified period that went to policy-violating content. An impression is a view event under a documented product definition. The numerator counts impressions on content judged to violate a named policy; the denominator counts all eligible impressions in the same population and window.

Meta explains why this choice differs from counting violating posts: a widely seen post creates more user exposure than an unseen one. The Pinterest-affiliated preprint formalizes the same idea as the sum of impressions on violating content divided by all impressions. These are published methods, not proof that every service must use exactly this definition. If the product question is “How many distinct users encountered any harmful item?” a user-reach metric is different and may require a different sample or linkage.

Before calculating anything, specify the policy category, surfaces, geography, age restrictions, calendar window, repeated-view rule, and whether impressions removed before rendering belong in the frame. You may need both attempted distribution and observed exposure, but they answer different questions. If a policy changed midweek, a single “violating” label may not be comparable across days; either hold the policy version fixed for analysis or label each estimate with the policy in force.

Consider an original three-item example. Item A is violating and receives 800 impressions. Items B and C are safe and receive 150 and 50. One of three content items violates policy, so item prevalence is 33.3%. Yet 800 of 1,000 impressions went to A, so view prevalence is 80%. The disagreement is not an arithmetic error; it is why the denominator must match the user's experience question. It also explains why sampling uniformly over content items without accounting for exposure is insufficient for an impression-weighted estimate.

Why the review queue is the wrong sample

The content in a review queue is intentionally selected: model flags, user reports, escalations, and viral incidents send riskier items to reviewers. The share of reviewed items confirmed harmful can describe queue precision or reviewer workload. It does not estimate platform prevalence by itself because unreported and unflagged content had little or no chance of entering that queue. Meta's explainer explicitly notes several ways harmful content can be seen before action or without detection.

Similarly, removal volume can increase when detection improves even if underlying harmful exposure declines. It can also fall when staffing drops while exposure rises. Reports per thousand views depend on who chooses to report, how easy reporting is, and whether a category is recognizable to viewers. Treat these as useful operational signals, not interchangeable prevalence denominators. An interview answer should identify the selection mechanism behind each proposed data source.

This is also a measurement warning beyond any single platform. Ofcom's paper on evaluating online safety measures notes that reporting-based metrics can shift with users' awareness of reporting tools or willingness to report. A product intervention could therefore change the report rate even if exposure does not move in the same direction.

If the interviewer offers only an enforcement table, say what it can and cannot support. You can analyze trends in decisions, appeal reversals, latency, and review throughput. To estimate unseen harm, request a measurement frame drawn from the content people actually saw, including items never reported. If that frame is unavailable, present an explicitly bounded proxy rather than renaming enforcement rate “prevalence.”

Design a sample under a limited review budget

The clean baseline is a probability sample of eligible impressions, or of content units with known exposure counts and inclusion probabilities. Send sampled units for policy labeling. Every in-frame unit must have a known nonzero chance of selection if you intend to generalize to the whole frame. Randomly sampling impressions directly naturally emphasizes heavily viewed content; sampling unique items uniformly requires exposure weights in the estimate.

When harm is rare, a small uniform sample may yield almost no positives. The Pinterest-affiliated preprint describes using model risk scores to prioritize sampling, while retaining positive selection probability and correcting for the unequal draw probabilities in the estimator. That is a model-assisted measurement design: risk scores decide what gets labeled more often, not whether the content is automatically harmful. It is also not the same as taking a convenience sample of the operational review queue. The paper discusses effective sample size because aggressive concentration can create unstable weights, even if it finds more positives.

An interview-ready design starts with a reserved random baseline and, if budget allows, a second risk-enriched stratum. Log the selection probability, sampling day, content and impression identifiers, policy version, and exposure count for each sampled item. Keep duplicate item draws and repeated impressions straight. If content is sampled once but has many views, its label can be attached to its eligible impression count; if impressions are sampled, repeated appearances of one item are part of the target quantity. Explain which design you chose before writing the estimator.

Design question	Decision to document
Population	Which impressions, surface, region and time window are eligible?
Unit	Is the random draw an impression or a unique content item?
Inclusion	Can every eligible unit enter, and is its probability stored?
Label	Which policy rubric and adjudication process define a violation?
Estimate	Which exposure and inverse-probability weights recover the target population?
Uncertainty	How will sampling variation and label error be reported separately?

Work through a weighted-stratum example

Suppose the product has 8,000 Home impressions and 2,000 Search impressions in one day. Reviewers examine 100 randomly sampled impressions from each surface. They find 2 violating impressions in Home and 6 in Search. Within each stratum, the estimated rates are 2% and 6%. If the target is all impressions, weight the rates by the traffic shares: 0.8 × 2% + 0.2 × 6% = 2.8%.

The naive mean of the two sampled groups is 4%, because the design sampled equal counts from unequal traffic populations. A candidate who says “8 positives out of 200, so 4% prevalence” has described the sample composition, not the platform. For this simple stratified case, the estimator is a traffic-weighted average of stratum rates. Under a more complex probability-proportional design, use each unit's recorded inclusion or draw probability as required by that design. Do not insert arbitrary weights after seeing the answers.

The example's numbers are invented for teaching. A real estimate needs a design-appropriate confidence interval, and rare categories may have wide intervals even after enrichment. If the interview asks for uncertainty, identify the randomization unit and whether repeated observations of the same content or user are correlated. A binomial formula that treats every impression as an independent labeled draw can understate uncertainty when one item generates thousands of impressions.

Labels, delays, and policy disagreement

The sampling design can be sound while the labels are wrong. Reviewers may disagree on context, language, satire, or evolving policy. A model-assisted labeler can drift after a model update. The Pinterest-affiliated preprint emphasizes gold-set validation and discusses sensitivity to label error; it is not a license to treat model outputs as truth. In the interview, separate sampling uncertainty from measurement error in labels.

Choose an explicit label protocol: blind a sample to queue score where practical, provide the policy and version, adjudicate ambiguous cases, and measure agreement on a quality set that includes relevant languages and surfaces. Compare automated and human labels by category, including false positives and false negatives. A high aggregate accuracy can hide weak performance on the rare harm whose prevalence you are estimating. If false-positive or false-negative rates vary with exposure or risk score, a simple one-number correction can also be misleading.

Labels may arrive after the impression day. Freeze a reporting cohort and state its maturity rule: for example, “preliminary through day three; finalized after the review window closes.” Keep the event time, label time, policy version, and any later appeal outcome. Do not silently overwrite last week's estimate with a newer label set and then interpret the resulting line movement as a product change. If latency is material, show preliminary and mature series or delay release consistently.

Explain changes without overstating causality

Suppose measured prevalence rises after a stricter classifier launch. First audit measurement: did the impression logging change, did a policy definition move, did sampling probabilities or labeler prompts change, did a high-reach incident occur, or did a segment's traffic share shift? Then inspect intervals and meaningful drill-downs. A single point estimate is not enough to attribute the change to the classifier.

The stronger analysis compares like with like: same target population, stable policy rubric, mature labels, and known sample design. If the change was experimental, compare randomized treatment and control populations using an exposure metric aligned with assignment. If it was a global launch, a time series with controls and explicit confounders may support a weaker conclusion. Make the limitations visible. A fall in prevalence with rising removals might be consistent with better enforcement, but it does not prove that causal story on its own.

Subgroup rates deserve care. A platform-wide estimate can improve while a smaller language or surface worsens. Report slice denominators, effective sample sizes, and uncertainty; avoid publishing a noisy ranking of tiny groups. The preprint's single-sample drill-down approach illustrates how one governed measurement sample can support consistent slices, provided the sample covers them adequately. If a slice has too few labeled positives, say the estimate is unstable rather than hiding it in the overall number.

Sources and further reading

• Meta: Measuring Prevalence of Violating Content on Facebook — company explanation of exposure-based prevalence and limits of enforcement counts.
• Dobi et al., Measuring the Prevalence of Policy-Violating Content with ML-Assisted Sampling and LLM Labeling, arXiv v2, August 2026 — platform-specific preprint on probability sampling, reweighting, label quality and uncertainty.
• Ofcom: Evaluating Online Safety Measures — regulator discussion of why user reporting alone can misstate impact.