Lead a zero-to-one initiative effectively

Take a Vague Mandate ("Improve Shopper Retention") from Idea to Launch

Context

You work in a two‑sided, on‑demand marketplace where "shoppers" are independent contractors who pick and deliver orders. Leadership asks you to "improve shopper retention" without a defined scope. Describe how you would drive this from idea to launch as a data‑oriented leader.

Tasks

1. Define the problem statement (with any minimal assumptions you need).
2. Specify success metrics (north star, leading indicators) and guardrails.
3. Propose a discovery plan (quantitative and qualitative) and key hypotheses.
4. Provide a PRD outline you would expect to use with Product/Eng.
5. Map stakeholders and propose a simple RACI.
6. Lay out milestones and explicit kill/gate criteria.
7. Explain how you would de‑risk with a prototype/MVP and experimentation design.
8. Explain how you would obtain resources and budget.
9. Describe how you would manage change with CX/Support, Legal/Compliance, and Sales/Partners.
10. Describe your post‑launch review and learning plan.
11. Provide a 30/60/90‑day plan.
12. Give one example of a tough trade‑off you would make and why.

Solution

1) Problem Statement

• Goal: Increase shopper retention in a way that sustainably improves marketplace health and economics without harming customer experience, partner relationships, or compliance.
• Assumptions (explicit):
  • "Shopper" = independent contractor fulfilling orders.
  • Retention is measured at multiple horizons (D7, D30, D60, D90) with cohorting by shopper start date.
  • Marketplace health depends on fill rate, on‑time delivery, shopper earnings/hour, and customer NPS.
• Proposed scope v1: Focus on early‑lifecycle retention (first 30–60 days), where hazard of churn is typically highest.

2) Success Metrics and Guardrails

• North star
  • 60‑day retention of new shoppers (S60): proportion of a start‑month cohort who complete at least one batch in days 31–60.
  • Formal: S(t) = survival probability to day t. Target: ΔS60 ≥ +3 percentage points within two quarters.
• Leading indicators
  • D7 activation rate: share completing ≥1 batch in first 7 days.
  • Time‑to‑first‑batch (TTFB): median hours from onboarding complete → first batch.
  • Early hours worked: median online hours in week 1.
  • Early experience quality: cancellations, support contacts in first 10 batches.
• Economic metrics
  • Incremental LTV per shopper (ILTV) = Δ(hours worked) × margin/hour.
  • Cost per retained shopper (CPRS) = incentive + ops cost / incremental retained shoppers.
• Marketplace health guardrails (no worse than thresholds vs control):
  • Order fill rate: ≥ −0.5 pp.
  • On‑time delivery: ≥ −0.5 pp.
  • Customer NPS/CSAT: ≥ −1.0 pt.
  • Shopper earnings/hour: ≥ baseline (no reduction in p50 or p25).
  • Support contacts/order: ≤ +5%.
  • Compliance: Pay transparency, classification, and disclosure requirements met.
  • Budget guardrail: CPRS ≤ $150 (example) with ROI ≥ 1.5× at 6 months.

Small numeric example

• Baseline: 10,000 new shoppers/month; S60 = 35% → 3,500 retained at D60.
• Target: 39% → 3,900 retained; +400 incremental retained.
• If each retained shopper contributes 40 orders in next 60 days at $1 margin/order → +$16k gross margin over 60 days.
• If incentives/ops cost = $60k → short‑term ROI < 1, but over 6–12 months ILTV may justify; set CPRS and ROI thresholds to ensure sustainability.

3) Discovery Plan

• Quantitative (weeks 1–4)
  • Build cohort retention curves (D7/D30/D60/D90), survival and hazard analysis.
  • Segment by cohort month, geo, tenure, device, earnings decile, time‑of‑day, batch type, cancellation, support contacts.
  • Feature correlation/importance (e.g., SHAP over survival model) to identify drivers: TTFB, idle time, pay variability, batch distance, cancellation exposure, support friction.
  • Funnel: onboarding complete → first login → first batch → 5th batch → 20th batch; find biggest drop‑offs.
  • Power analysis for experiments; estimate variance and MDEs.
• Qualitative (weeks 1–4)
  • 1:1 interviews with 20–30 shoppers (new, ramped, churned/reactivated); ride‑alongs/shadowing.
  • Survey to quantify pain points (clarity of earnings, navigation, batching fairness, tip expectations, payments timing).
  • Code support tickets for early‑lifecycle themes.
• Competitor and policy review
  • Benchmark payouts cadence, sign‑up bonuses, earnings transparency, guarantees, and compliance constraints.
• Synthesize hypotheses
  • H1: Long TTFB and idle time drive early churn.
  • H2: Earnings variability and lack of transparency reduce perceived fairness.
  • H3: Early bad experiences (cancellations, complex substitutions, parking issues) create outsized churn risk.
  • H4: Payment cadence (slow first payout) weakens reinforcement loop.
• Prioritize with RICE/impact vs. effort; run a pre‑mortem (how could this fail?).

4) PRD Outline (for the initial wedge)

• Title: Fast Start for New Shoppers (v1)
• Background & Problem
• Goals and Non‑Goals
• Hypotheses
• Target users and segments
• User stories (e.g., "As a new shopper, I want my first batch quickly with predictable earnings")
• Solution overview
  • Example components: queue prioritization for first 3 batches, earnings guarantee on first day, real‑time guidance, accelerated first payout.
• Requirements
  • Feature flags, geo eligibility, comms, payments config.
• Experiment/rollout plan
  • Geo‑cluster randomization, holdouts, duration, sample size, MDE.
• Metrics & telemetry
  • Primary, leading, guardrails, dashboards.
• Risks & mitigations
  • Demand displacement, fairness perceptions, legal.
• Compliance & Legal checklist
• Support/Operations plan
• Analytics plan (analysis, HTE, CUPED/diff‑in‑diff as needed)
• Launch criteria and kill criteria

5) Stakeholder Map (RACI exemplar)

• Responsible: PM (Shopper Experience), Data Scientist, Eng Lead, Designer, UXR.
• Accountable: GM/Director for Supply/Marketplace.
• Consulted: Shopper Ops, CX/Support, Trust & Safety, Legal/Compliance, Finance, Marketing/CRM, Dispatch/Matching, Payments, Data Engineering, Partner/Sales.
• Informed: Exec sponsor, Regional Ops, Partner Success.

6) Milestones and Kill/Gate Criteria

• Milestones
  • Week 0–2: Metric definitions finalized; discovery complete; proposal stack‑rank; select v1 wedge.
  • Week 3–4: PRD v1, experiment design, instrumentation plan; resource/budget approvals.
  • Week 5–8: Build + internal dogfood; geo pilot launch behind flags.
  • Week 9–14: Pilot run (6 weeks) with weekly guardrail monitoring.
  • Week 15–16: Readout; go/no‑go; iterate or scale.
• Kill/Gate criteria (examples)
  • Pre‑pilot: No green light without legal signoff and guardrail dashboards.
  • During pilot: Immediate pause if any guardrail breach persists >48h (e.g., on‑time −1 pp, earnings/hour −$0.25 at p50) or incident rate spikes.
  • End of pilot:
    • Proceed if: ΔS60 ≥ +2 pp (lower bound of 95% CI > +0.5 pp), CPRS ≤ $150, fill rate ≥ −0.3 pp, and NPS ≥ −0.5.
    • Iterate if: effect positive but below ROI threshold; redesign and retest.
    • Kill if: effect null/negative or cost/guardrail violations; document and sunset.

7) De‑risking with Prototype/MVP and Experimentation

• MVP concept: "Fast Start"
  • Queue prioritization for first 3 batches to reduce TTFB.
  • First‑week earnings guarantee (e.g., $X for first Y hours), reconciled at payout.
  • Early guidance: checklists, in‑app tips, hotline for first 5 batches.
  • Accelerated payout for first week.
• Low‑lift prototypes
  • Wizard‑of‑Oz ops: manual priority assignment in a few geos for 2 weeks.
  • Off‑app comms: SMS nudges, help line, and survey prompts to validate messaging.
  • Earnings transparency card mock (no ML yet): show conservative expected earnings range.
• Experiment design
  • Geo‑cluster A/B with matched markets; or time‑based randomized windows if interference risk.
  • CUPED or pre‑period covariate adjustment to reduce variance.
  • Sample‑size sketch: Detect +3 pp on S30 from 50% baseline (two‑sided, α=0.05, power=0.8) → ~3,200–4,000 shoppers/arm; adjust for clustering and seasonality.
  • Heterogeneity of treatment effects: tenure, geo, time‑of‑day, demand elasticity.
  • Interference checks: monitor control‑geo displacement in fill rate.

8) Obtaining Resources and Budget

• One‑pager/PRFAQ: problem, hypotheses, expected ROI, risks, plan, asks.
• Business case
  • Forecast ILTV uplift and CPRS under base/best/worst scenarios.
  • Show sensitivity to demand, seasonality, and incentive size.
• Asks
  • People: 1 PM, 1 DS, 2–3 Eng, 0.5 DE, 0.5 Designer, UXR support; Ops coverage.
  • Budget: incentive pool for guarantees/payouts; UXR incentives.
  • Platform: experiment infra, feature flags, dashboards.
• Governance: biweekly steering with GM/Legal/Finance; pre‑reads to accelerate approvals.

9) Change Management (CX/Legal/Sales/Partners)

• CX/Support
  • Update macros/FAQs; train agents on Fast Start rules and edge cases.
  • Real‑time escalation path during pilot; monitor contact reasons.
• Legal/Compliance
  • Review incentive structures, disclosures, pay transparency, and classification sensitivities.
  • Ensure clear T&Cs and opt‑in where needed; regional policy checks.
• Sales/Partners
  • Communicate pilot geos and expected impact on fill/on‑time; align on service levels.
  • Set expectations: no partner pricing changes; share guardrails and monitors.
• Shopper comms
  • In‑app banners, lifecycle emails/SMS; explain how guarantees work; avoid over‑promising.

10) Post‑Launch Review and Learning Plan

• Timing: interim at 2–3 weeks; full read at 6 weeks; final at 12 weeks.
• Contents
  • Primary effects (S30/S60), leading indicators, guardrails, CPRS, ILTV.
  • Heterogeneity analysis; external validity; seasonality effects.
  • Operational learnings (support load, fraud/abuse, gaming of guarantees).
  • Decision: scale, iterate, or sunset; revised PRD for v2 if applicable.
  • Documentation: write‑up, dashboard links, decisions log; add to playbook.

11) 30/60/90‑Day Plan

• Days 0–30
  • Align definitions; build cohort/survival dashboards; run discovery (quant + qual).
  • Select v1 wedge (Fast Start); write PRD and experiment design; complete legal review.
  • Instrumentation plan; power analysis; resource/budget approvals.
• Days 31–60
  • Build MVP; set up feature flags and monitoring; train CX; finalize comms.
  • Launch geo‑pilot; weekly reviews; enforce guardrails.
  • Start concurrent low‑lift tests (nudges, transparency card) for learning velocity.
• Days 61–90
  • Complete pilot; deep‑dive results and HTE; ROI analysis.
  • Go/no‑go; scale plan or pivot to next hypothesis (e.g., payment cadence or batching fairness).
  • Publish learnings; backlog v2; align roadmap for Q+1.

12) Example Tough Trade‑off

• Choice: Broad hourly incentive for all shoppers vs. targeted "Fast Start" for new shoppers.
  • Broad incentive likely boosts short‑term supply but is expensive and may not improve long‑term retention; risks fairness expectations and demand displacement.
  • Targeted Fast Start focuses on the highest hazard period with lower CPRS and clearer causal link to retention, but may create perceived inequity with veterans.
• Decision: Choose targeted Fast Start to maximize ILTV/ROI, with mitigation:
  • Communicate purpose clearly; add lightweight recognition for veterans (e.g., loyalty badges, occasional targeted boosts) without undermining economics.

Common Pitfalls and Guardrails

• Goodhart’s Law: Don’t optimize D7 at the expense of D60 or earnings/hour.
• Selection bias: Ensure randomized or strong quasi‑experimental designs; avoid survivorship bias.
• Interference: Geo‑level randomization to avoid spillovers in a shared marketplace.
• Seasonality and competitor actions: Use matched‑market controls; extend pilots across periods.
• Compliance: Incentives, disclosures, and communications must be legally vetted.

Optional Alternatives if RCT Is Hard

• Matched‑market diff‑in‑diff with synthetic controls.
• Interrupted time series with guardrails and falsification tests.
• Instrumental variables (e.g., exogenous weather shocks) for diagnostics only.

This plan turns a vague mandate into a measurable, de‑risked, and cross‑functionally executable path from idea to launch, with explicit metrics, safeguards, and decision gates.