Lead structured response to accuracy incident

Incident Response Plan: Spike in Incorrect Payments for CA Users Aged 18–24

Scenario

Since 00:00 today, the share of incorrect payments for users aged 18–24 in CA has risen from a historical 2.0% to 3.1% across approximately 120,000 transactions. You are the on-call lead on a payment-accuracy team with access to:

• Feature logs and model scores
• Recent code/deployment diffs
• A sampling tool to pull raw cases for audit

Finance is escalating and requests a mitigation within 24 hours.

Assume the system uses a real-time ML risk score to route transactions (approve vs send to review/decline). "Incorrect payment" is measured by a near-real-time audit/labeling pipeline.

Task

Describe, in a structured, time-boxed plan, exactly how you would:

1. Triage and quantify impact (owners, metrics, and acceptance criteria for "all clear").
2. Form testable hypotheses and design the minimum experiments/queries to isolate root cause (data slices, holdouts, backfills).
3. Decide on a safe mitigation (e.g., threshold change, feature hotfix, rule fallback), including rollback triggers and predicted side effects (precision/recall trade-offs).
4. Communicate status to executives and partner teams at specific times with concrete artifacts (dashboards/runbooks).
5. Prevent recurrence (observability, guardrails, pre-deploy checks).

Be explicit about risks you will accept vs avoid and how you would measure success by end of day.

Solution

Assumptions and Notation

• CA = California (confirm immediately; if Canada, repeat all slices using country_region).
• Score s ∈ [0,1] is the model-estimated probability a payment is incorrect.
• Action: if s ≥ t (threshold), route to review/decline; if s < t, auto-approve. Lowering t means catching more potential incorrect cases (lower incorrect rate, higher friction).
• Observed cohort volume since 00:00: n ≈ 120,000 for CA age 18–24 (validate; if 120k is total site volume, re-compute for the cohort).
• Incorrect rate (IR) = incorrect_count / total_count.

0. Timeline Overview (Day-of Plan)

• 09:00 Detect + page. Create an incident channel and ticket. Assign roles.
• 09:05–09:30 Rapid triage: confirm anomaly, scope who/what/when. Freeze risky deploys. Build live dashboard by cohort.
• 09:30–10:30 Root-cause probes: config/threshold, model version, feature missingness/drift, label pipeline, traffic mix. Pull 100–200 raw cases for audit.
• 10:30–11:00 Decision checkpoint 1: if clear cause found (config/model), roll back. If unknown, ship gated mitigation (tighten threshold for CA 18–24 on 25% canary) with guardrails.
• 11:00–14:00 Continue experiments (shadow scoring, backfills, merchant/device slices). Adjust mitigation as evidence arrives.
• 14:00 Decision checkpoint 2: expand/roll back mitigation based on metrics.
• 16:00 Draft postmortem outline; finalize prevention actions and owners.
• 17:30 End-of-day (EOD) report: results, metrics, next steps.

1) Triage and Quantify Impact

Owners (assign in incident ticket)

• Incident commander (IC): You (on-call DS). Decision maker/timekeeper.
• ML engineer: model configs, thresholds, rollbacks.
• Data engineer: feature pipeline and data quality.
• Analyst/BI: dashboards, SQL queries, CIs.
• Risk Ops: manual audit sampling, case categorization.
• Finance liaison: impact translation ($, volume), stakeholder updates.

Metrics to compute now

• IR_18_24_CA(t): 15-min rolling incorrect rate for CA age 18–24.
• Baselines: last 4 weeks, same day-of-week/time-of-day; mean and bands.
• Volume: transactions per 15 min.
• Confusion components (proxy in near-real time):
  • Block/review rate (BR): share routed to review/decline.
  • Auto-approve rate (AR): share approved immediately.
  • If labels are fast: False approvals (incorrect among auto-approvals), False reviews (correct among reviewed).
• Feature health: top-20 features’ null rate, staleness, distribution shift.
• Model health: score distribution per cohort; KS statistic vs baseline; PSI for features.
• Label pipeline health: label latency distribution and completeness.

Quick significance check (example)

• Observed p̂ = 3.1% with n = 120,000. 95% CI ≈ p̂ ± 1.96 * sqrt[p̂(1−p̂)/n] ≈ 0.031 ± 0.001 → (3.0%, 3.2%). Historical baseline = 2.0% → highly significant.
• Counts: expected incorrect at 2.0% = 2,400; observed = 3,720; delta ≈ +1,320.

Acceptance criteria for "all clear"

Any one of:

• IR_18_24_CA returns to ≤ 2.3% (≤ +15% rel vs 2.0% baseline) for 3 consecutive 30-min windows with stable volume; OR
• IR_18_24_CA within baseline 95% control limits and no active anomaly flags on score/feature drift; AND
• No new anomalies introduced in adjacent cohorts (other ages in CA, 18–24 in other states); AND
• Manual review queue remains < 80% capacity for 95th percentile SLA.

2) Hypotheses and Minimum Experiments/Queries

Prioritize checks that are fast, discriminative, and reversible.

H1. Threshold/config drift or bad rollout

• Check: Compare current threshold t, rule weights, and model_version in prod vs last stable. Diff config files, env vars, feature flags, and change logs since 00:00.
• Query: SELECT distinct model_version, threshold, count(*) FROM events WHERE ts >= today AND cohort = 'CA_18_24' GROUP BY 1,2;
• Action if true: Immediate rollback to last known-good config/model.

H2. Feature pipeline issue (missingness/staleness/time-zone)

• Check: For top features by importance, compare null rate, timestamp skew, PSI.
• Query (sketch):
  • Feature nulls: SELECT feature_name, avg(is_null(value)) null_rate FROM feature_logs WHERE ts >= today AND cohort='CA_18_24' GROUP BY 1 ORDER BY null_rate DESC LIMIT 20;
  • Staleness: SELECT avg(ts_event - ts_feature) FROM feature_logs WHERE cohort='CA_18_24';
• Experiment: Re-score a 1k-sample from today with yesterday’s pipeline artifact (backfill) and compare scores; large divergence indicates pipeline bug.

H3. Labeling pipeline change (definition/latency)

• Check: Label arrival latency and label completeness today vs prior.
• Query: SELECT percentile(latency, [50,90,99]), completeness FROM labels WHERE ts >= today AND cohort='CA_18_24';
• If labels slowed, observed IR may be inflated/biased. Validate with manual audit sample.

H4. Traffic mix shift (merchant, device, payment method, promo)

• Check: Contribution analysis by merchant_id, device_os, payment_method, BIN, referrer.
• Query: SELECT merchant_id, count(*) n, avg(is_incorrect) ir FROM tx WHERE ts>=today AND cohort='CA_18_24' GROUP BY 1 ORDER BY ir DESC LIMIT 20;
• Experiment: If 1–2 merchants drive delta, isolate mitigation to those.

H5. Geography/age parsing bug (CA vs CA-Province; age boundary at midnight)

• Check: Validate geo parser inputs; compare distribution of state/province codes; examine ages 17–18 and 24–25 edges around midnight.
• Query: SELECT state, country, count(*) FROM tx WHERE ts>=today AND age BETWEEN 18 AND 24 GROUP BY 1,2;

H6. Score distribution drift/miscalibration

• Check: Compare score histogram to baseline; KS test; calibration curves if labels available.
• Experiment: Shadow scoring with previous model on today’s features; if shadow aligns with baseline but prod drifts, model or feature change likely.

H7. Third-party dependency degradation

• Check: Timeouts/error rates for external services (device fingerprint, address verification). If fallbacks applied, feature quality degrades.
• Query: SELECT service, error_rate FROM svc_metrics WHERE ts>=today AND cohort='CA_18_24';

Minimum manual audit

• Pull 100–200 incorrect cases (and 100 near-threshold approvals) for CA 18–24.
• Tag root-cause categories: feature missing, merchant anomaly, time-zone, data mapping, truly new behavior. This directs mitigation choice.

3) Safe Mitigation Decision, Rollback Triggers, Side Effects

Decide by 10:30–11:00, even if root cause is not fully confirmed. Use canary and guardrails.

Candidate mitigations (ordered by safety)

1. Roll back config/model to last known-good

• Use if any diff is detected in threshold/model/rules since 00:00.
• Predicted effect: restores prior precision/recall. Validate on 30–60 min window.
• Rollback trigger: N/A; this is the rollback.

2. Targeted threshold tightening for CA 18–24 only (lower t)

• Goal: reduce incorrect approvals (improve recall) at cost of higher review/false positive rate.
• Deployment: 25% canary of CA 18–24 for 60 minutes; then ramp to 100% if improving and within capacity.
• Guardrails:
  • Manual review queue length < 80% capacity; 95th percentile review SLA < target.
  • IR improvement ≥ 20% relative within 60–90 minutes vs control.
• Side effects (quantify): Use yesterday’s ROC/PR to simulate. Example:
  • At t0: TPR=0.72, FPR=0.04.
  • At t1=t0−0.05: TPR=0.80 (+8 pp), FPR=0.055 (+1.5 pp).
  • With today’s cohort volume ~120k and incorrect prevalence ~3.1%:
    • Added reviews ≈ ΔFPR * (1−prevalence) * n ≈ 0.015 * 0.969 * 120k ≈ 1,747 extra reviews.
    • Incorrect approvals reduced ≈ ΔTPR * prevalence * n ≈ 0.08 * 0.031 * 120k ≈ 298 fewer incorrect approvals.
  • Ensure Ops can absorb +1.7k reviews over the hour(s).
• Rollback triggers:
  • Review SLA breach > 2x baseline for 2 consecutive 15-min windows.
  • No IR improvement ≥ 10% rel after 90 minutes.

3. Feature hotfix: drop degraded features and re-weight via rules fallback

• Use if one/few features show >30% missingness or high drift PSI>0.25.
• Implementation: Temporarily exclude affected features; enable rule-based overrides (e.g., merchant-block list, AVS mismatch strictness) for CA 18–24.
• Side effects: Loss of model discrimination; expect higher false positives; monitor BR and SLA.

4. Merchant-targeted policy

• If 1–2 merchants drive majority of delta, add temporary stricter rules for those merchants (e.g., lower t only for those IDs or route to review).

Measuring mitigation effect quickly

• Use interleaved A/B (canary vs control within CA 18–24) with 15-min metrics and CIs.
• Primary: IR among auto-approveds; Secondary: total IR, review queue metrics, customer impact (approval rate).
• If labels are delayed, use proxy: reduction in high-risk approvals near threshold and audit sampling outcomes.

4) Communication Plan and Artifacts

Cadence

• 09:15 Initial incident notification (execs + finance + partner teams)
  • What: anomaly detected; scope (cohort, magnitude), immediate actions (deploy freeze, triage), next update at 09:45.
• 09:45 Triage readout
  • Share dashboard link, IR time series with CIs, top slices, preliminary hypotheses, and owner assignments.
• 11:00 Decision checkpoint 1
  • Announce chosen mitigation (rollback/threshold canary), guardrails, expected side effects, and when to expect impact (next 60–90 min).
• 13:00 Midday update
  • Results of canary/experiments, whether to ramp/adjust; any merchant-specific actions.
• 15:00 Decision checkpoint 2
  • Finalize mitigation state (ramp to 100% or revert), outline prevention work.
• 17:30 EOD report
  • Metrics vs baseline, incident timeline, root cause (or leading hypothesis), actions taken, next-day items.

Artifacts to produce/share

• Live dashboard (Looker/Superset):
  • IR by 15-min for CA 18–24 vs baseline; adjacent cohorts for regression-to-mean check.
  • Score distribution (KS), feature nulls/staleness, label latency.
  • Review queue and SLA utilization.
• Runbook page: mitigation levers, thresholds, rollback commands, owners and escalation paths.
• JIRA/Incident ticket: timeline, decisions, diffs, SQL links, shadow/backfill results.
• One-pager for execs: problem, impact ($/volume), mitigation, risk trade-offs, ETA for steady state.

5) Prevent Recurrence

Observability and Guardrails

• Segment monitors: real-time IR for key cohorts (age, geo, merchant), with control charts and anomaly alerts.
• Model monitors: score drift (KS), calibration drift (Brier/NLL), feature PSI and missingness alerts with auto-fallback to rules when thresholds exceeded.
• Labeling monitors: label latency/completeness SLOs; alert on shifts.
• Capacity monitors: review queue SLA and auto-throttle on mitigation to avoid overload.

Pre-deploy checks

• Canary by cohort: 10–20% rollout with guardrail checks (IR, approval rate, SLA) and auto-rollback on breach.
• Config integrity: checksums/signature and diff approvals; disallow same-day threshold changes without canary.
• Data contracts and schema tests for upstream features; time-zone and geo parsing unit tests.
• Backtest gate: require day-of-week and cohort-sliced validation on last 2–4 weeks before promoting a model/config.

Process

• Blameless postmortem within 48 hours with concrete action items and due dates.
• Expand coverage of runbooks; add decision trees for common root causes (feature missingness, merchant surge, label delays).

Risks: Accepted vs Avoided

• Accepted: temporary higher friction (more reviews/false positives) within Ops capacity; localized mitigations (CA 18–24 and/or specific merchants); short-term revenue impact to protect accuracy.
• Avoided: broad global threshold changes without canary; untested feature code changes; changes that exceed Ops capacity or breach SLAs system-wide; permanent policy shifts without analysis.

Success by End of Day

• Quantitative:
  • IR_18_24_CA reduced ≥ 20% relative from peak and within baseline tolerance bands for 90 minutes; OR explained by a verified label/measurement artifact and mitigated.
  • No new anomalies in adjacent cohorts; review SLA within target; approval rate impact ≤ agreed guardrail (e.g., ≤ 2 pp drop).
• Qualitative:
  • Root cause identified or top hypothesis validated with evidence; mitigation in place; rollback plan documented.
  • Dashboards, runbook updates, and postmortem draft complete; owners assigned for prevention items.

Example SQL/Analysis Snippets (Sketches)

• IR by cohort/time:
  • SELECT time_bucket('15 min', ts) t, count(*) n, avg(is_incorrect::int) ir FROM tx WHERE ts>=today AND geo='CA' AND age BETWEEN 18 AND 24 GROUP BY 1 ORDER BY 1;
• Score drift:
  • SELECT width_bucket(score,0,1,20) b, count(*) FROM scores WHERE ts>=today AND cohort='CA_18_24' GROUP BY 1 ORDER BY 1;
• Feature nulls:
  • SELECT feature, avg((value IS NULL)::int) null_rate FROM feature_logs WHERE ts>=today AND cohort='CA_18_24' GROUP BY 1 ORDER BY null_rate DESC LIMIT 20;
• Merchant contribution:
  • SELECT merchant_id, count() n, avg(is_incorrect::int) ir FROM tx WHERE ts>=today AND cohort='CA_18_24' GROUP BY 1 HAVING count()>200 ORDER BY ir DESC LIMIT 20;

Final Notes

• Confirm whether "CA" means California or Canada immediately; misclassification alone can cause the apparent spike.
• If the 120k volume refers to all users, recompute n for CA 18–24 before declaring statistical significance and sizing mitigation.
• Keep all changes small, targeted, and reversible; prefer canaries and automatic rollbacks with clear guardrails.