Present a DS project with business impact

7-Minute Data Science Project Presentation (Onsite)

Context

You are interviewing for a Data Scientist role and will present a past project to a mixed audience of a Product Manager, Software Engineer, and Data Scientist. The goal is to demonstrate business impact, analytical rigor, and communication clarity.

Task

Prepare and deliver a 7-minute presentation (plus a 60-second roadmap) covering:

1. Business problem, decision at stake, and north-star metric.
2. Data sources, key assumptions, and their risks.
3. Modeling/analysis approach and why alternatives were rejected.
4. Results with confidence intervals and sensitivity checks.
5. Shipped impact vs. projected impact and how you validated it post-launch.
6. Two failures or trade-offs you consciously accepted.

Conclude with a 60-second roadmap for the next iteration.

Constraints

• Audience is mixed; avoid jargon or explain it.
• 5–6 slides maximum; keep within 7 minutes + 60 seconds.
• Use real experiences; omit proprietary details.
• Be explicit about metrics, decision-making, and validation.

Solution

How to Structure and Deliver a Strong 7-Minute DS Project Talk

Below is a step-by-step framework, a slide-by-slide timing plan, and a complete example you can adapt. The example uses a consumer-services marketplace scenario that maps well to many product/data contexts.

Time & Slide Plan (7 minutes + 60-second roadmap)

• Slide 1 (1:00): Problem, decision, north-star metric
• Slide 2 (1:15): Data sources, assumptions, risks
• Slide 3 (2:00): Modeling/analysis approach and rejected alternatives
• Slide 4 (1:30): Results with confidence intervals and sensitivity checks
• Slide 5 (0:45): Shipped impact vs. projected; post-launch validation
• Slide 6 (0:30): Two failures/trade-offs
• Roadmap (60 seconds): Next iteration plan

Example Presentation You Can Deliver

Title: Improving Booking Conversion with a Lead-Quality Model

1. Business Problem, Decision, North-Star Metric

• Context: In a services marketplace, customers submit job requests (e.g., plumbing), and professionals (“pros”) respond. Low-quality leads reduce match rates, increase refunds, and hurt pro retention.
• Decision at stake: Replace a rule-based lead distribution with a machine-learning lead-quality score to prioritize which requests to surface, notify, and incentivize.
• North-star metric: Net Jobs Booked (booked jobs minus refunded/canceled) within 14 days of request. Secondary metrics: refund rate, pro response rate, and gross booking value per request (GBV/R).

2. Data Sources, Key Assumptions, Risks

• Data sources:
  • Historical requests: category, location, time, budget, textual description length.
  • Buyer signals: prior request history, device, on-site behaviors (e.g., message opens).
  • Pro supply signals: nearby supply density, pro ratings, response latency.
  • Outcomes: whether the request led to a booked job within 14 days; refunds.
• Labels: y = 1 if booked within 14 days; else y = 0.
• Key assumptions:
  • 14-day window captures >95% of bookings and is stable across categories.
  • Historical outcomes are representative of future behavior (stationarity).
  • Logged features are complete and timestamped to avoid label leakage.
• Risks and mitigations:
  • Selection bias: Only leads exposed to pros can become bookings. Mitigate via inverse propensity weighting in offline evaluation and by running an online A/B test.
  • Leakage: Post-request signals (e.g., quote count) could leak future information. Strict feature windowing and feature audits prevent leakage.
  • Drift/seasonality: Add time features, monitor calibration and AUC weekly, retrain monthly.
  • Cold starts: Back-off to category-geo priors when features are sparse.

3. Modeling/Analysis Approach and Alternatives Rejected

• Problem framing: Binary classification to estimate P(booking | request, context). We use the score to rank and set policy thresholds (e.g., notify top X%).
• Approach:
  • Baseline: Heuristic rules from domain knowledge (e.g., minimum budget, category filters). Baseline AUC ≈ 0.62.
  • Model: Gradient Boosted Trees (XGBoost) for non-linearities and feature interactions; 5-fold time-split cross-validation.
  • Calibration: Isotonic regression so scores map to well-calibrated probabilities used for policy tuning and LTV simulations.
  • Offline metrics: AUC, PR-AUC, Brier score (calibration), and top-decile lift.
  • Policy simulation: Convert calibrated probabilities into expected bookings under different thresholds; choose threshold to maximize expected Net Jobs Booked subject to guardrails (refund rate non-increasing).
• Why not these alternatives (and why):
  • Deep learning (tabular MLP): Rejected for interpretability, marginal lift in early tests, higher latency, and infra complexity.
  • Uplift modeling: Requires randomized notifications/exposure at scale and more complex experimentation; targeted for v2.
  • Two-sided optimization (supply constraints) end-to-end: Scoped out for v1 to reduce coupling; we used a modular ranking + simple throttling policy first.

4. Results with Confidence Intervals and Sensitivity Checks

• Offline:
  • AUC: 0.79 (model) vs. 0.62 (baseline).
  • Top 10% leads by score captured ~3.1× booking rate of average traffic.
  • Policy simulation projected +3.5% to +5.0% Net Jobs Booked at steady state.
• Online A/B Test (50/50, 2 weeks; clusters by city to limit interference):
  • Control booking rate: p_c = 11.8% (n_c = 120,000 requests)
  • Treatment booking rate: p_t = 12.3% (n_t = 118,000 requests)
  • Absolute uplift: Δ = p_t − p_c = 0.5 pp (relative +4.2%)
  • 95% CI for Δ using normal approximation:
    • SE = sqrt[p_c(1−p_c)/n_c + p_t(1−p_t)/n_t]
    • Numerically: SE ≈ sqrt(0.118×0.882/120,000 + 0.123×0.877/118,000) ≈ 0.001335
    • CI = 0.005 ± 1.96 × 0.001335 ≈ [0.0024, 0.0076] (i.e., +0.24 to +0.76 pp)
  • Refund rate: 3.1% → 2.9% (Δ = −0.2 pp; 95% CI ≈ [−0.35, −0.05] pp)
  • Secondary guardrails: Pro response rate flat; latency +5 ms (within SLO).
• Sensitivity Checks:
  • Label window: 7/14/21-day windows produced consistent ranking (Spearman 0.96+); v1 stuck with 14-day for stability.
  • Segment robustness: Gains observed across top categories and geos; no single segment dominated uplift.
  • Calibration: Reliability plots within ±2% in mid-probability bins after isotonic calibration.
  • CUPED-adjusted analysis reduced variance; conclusions unchanged.

5. Shipped Impact vs. Projected Impact; Post-Launch Validation

• Projected (offline sim): +3.5% to +5.0% Net Jobs Booked.
• Shipped (ramped to 100%): +3.8% (95% CI: +1.6% to +6.0%). Slightly below mid-point of projection due to tighter notification throttling after week 1 (ops feedback on message volume).
• Post-launch validation:
  • Kept a 5% long-lived holdout for 4 weeks; effects persisted.
  • Difference-in-differences across cities to sanity-check seasonal drift.
  • Monitoring: Weekly drift checks (PSI on key features), AUC/calibration tracking, and alerting on refund rate.

6. Two Failures/Trade-offs We Consciously Accepted

• Coverage trade-off: Limited v1 to top 12 categories (~65% of volume) to ensure model stability; long-tail users saw no improvement in v1.
• Objective short-termism: Optimized bookings within 14 days, not LTV. This may underweight high-value but slower-to-close categories.

60-Second Roadmap (Next Iteration)

• Expand scope and objectives:
  • Incorporate LTV-weighted targets and category-specific calibration.
  • Roll out to long-tail categories with transfer learning and hierarchical priors.
• Smarter decisioning:
  • Move from probability model → causal uplift + contextual bandits for notifications.
  • Dynamic thresholds by segment (geo, category, supply density).
• Reliability & fairness:
  • Real-time features (e.g., live supply load), quarterly bias audits, and automated regression tests for policy changes.

Teaching Notes: How to Adapt This to Your Project

• Replace the marketplace context with your domain, but keep the structure: decision → metric → data/assumptions → method → results with CIs → shipped impact → trade-offs → roadmap.
• If you lack an online experiment:
  • Use quasi-experimental designs (matched controls, diff-in-diff), show sensitivity to unobserved confounding (e.g., Rosenbaum bounds), and report uncertainty.
• Confidence intervals refresher (difference in proportions):
  • CI(Δ) = (p_t − p_c) ± z_{1−α/2} × sqrt[p_t(1−p_t)/n_t + p_c(1−p_c)/n_c]
  • For small samples or clustering, prefer bootstrap or cluster-robust variance.
• Common pitfalls:
  • Label leakage, selection bias from policy exposure, non-stationarity, and misaligned metrics (optimize for proxy not business outcome).
• Guardrails to mention in interviews:
  • Predefine metrics and stopping rules; calibrate and segment-check; keep a holdout; monitor drift; and define safe rollback criteria.