Lead XFN decision under tight timeline

Scenario: 72-Hour VP-Level Recommendation on Expanding a New Quoting Workflow

You have 72 hours to deliver a VP-level deck recommending whether to expand a new quoting workflow. Stakeholders are misaligned (PM prioritizes speed-to-ship, Operations prioritizes professional quality, Sales prioritizes near-term volume). The data landscape is messy: multiple Snowflake tables, some stale dashboards, and experiment logs with missing fields.

Describe, in concrete steps:

1. Clarify goals and alignment: How you will lock a single decision question, define must-have metrics and guardrails, and secure stakeholder sign-off within the first 6 hours.

2. Plan and delegation: How you break down work across 2 ICs (one analytics, one engineering), set SLAs, create a risk register, and design a daily checkpoint cadence. Include a RACI snapshot.

3. Data triage under ambiguity: Your approach to schema discovery, sampling checks, reconciling conflicting sources, and documenting known data quality gaps with impact assessment and contingency paths.

4. Narrative and persuasion: The storyline of the deck (1-page exec summary, metric deep-dives, trade-off analysis, recommendation with confidence intervals), how you pre-wire with critics, and how you handle live pushback with alternative scenarios.

5. Decision and follow-through: The explicit go/no-go criteria, owner assignments, and a 2-week post-decision plan with success metrics and an incident rollback plan.

6. Leadership behaviors: Specific examples of how you model calm under time pressure, protect focus, and escalate appropriately without creating churn.

Solution

Overview

This is a 72-hour, high-stakes decision under imperfect data. The strategy is: align fast on a single decision question and thresholds, triage data for decision-quality insights (not perfection), deliver a persuasion-first narrative, and pre-commit to post-launch guardrails and rollback. Below is a step-by-step plan you can execute with one analytics and one engineering IC.

1) First 6 Hours: Clarify Goals and Alignment

Objective: Lock the decision question, must-have metrics, guardrails, and get written sign-off.

• 0:00–0:30 Kickoff with PM, Ops, Sales, Eng Lead

  • Frame a single decision question: "Should we expand the new quoting workflow to [target scope] now, or gate expansion by cohort, given speed, quality, and volume trade-offs?"
  • Enumerate options upfront for decision clarity:
    • A: Expand to 100% of traffic.
    • B: Partial expansion (e.g., 50% or specific categories/geos).
    • C: Hold and instrument/mitigate gaps.

• 0:30–1:30 Define must-have metrics (primary and guardrails)

  • Speed-to-ship (PM’s north-star)
    • Median time-to-first-quote (TTFQ)
    • % of requests with ≥1 quote within 30 minutes
    • % of requests with ≥3 quotes within 24 hours
  • Pro quality (Ops’ north-star)
    • Hire quality proxy: 7-day hire rate and post-hire refund/dispute rate
    • Customer CSAT/NPS for completed jobs (if lagging, use support contact rate per 100 jobs)
    • Pro-side quality proxy: rehire rate or average rating of hired pros
  • Near-term volume (Sales’ north-star)
    • Quotes sent per request
    • Hire conversion within 7 days
    • GMV/revenue per request
  • Guardrails (must not breach)
    • Support tickets per 100 requests (+X% max delta)
    • Fraud/spam flags per 1,000 quotes (+Y% max delta)
    • Pro churn risk proxy: 7-day returning pro rate (−Z% max delta)

• 1:30–2:15 Set explicit decision thresholds and confidence level

  • Example thresholds (to be agreed):
    • TTFQ improvement ≥ 10% with 90% CI lower bound ≥ 5%
    • 7-day hire rate delta ≥ −0.5 percentage points (90% CI lower bound ≥ −1.0pp)
    • Guardrails within pre-set bounds (e.g., support tickets ≤ +10%, spam ≤ +5%)
  • Confidence: Use 90% CIs given time constraints; complement with sensitivity/bounds.

• 2:15–3:45 Create and circulate a one-page Decision Brief for sign-off

  • Includes: decision question, options A/B/C, primary metrics, guardrails, thresholds, confidence level, and data limitations.
  • Send for written ack in Slack/email; timebox responses to 2 hours.

• 3:45–6:00 Data quick-scan + finalize scope

  • Confirm the decision cohort (e.g., all categories vs. top 5 by volume; exclude long-tail if data is too sparse).
  • Lock the observation window (e.g., last 14–28 days) and unit of analysis (request-level for TTFQ and conversion). Document assumptions.

Deliverables (by hour 6)

• Signed 1-pager with metrics, thresholds, and options.
• Finalized cohorts, timeframe, unit of analysis, and confidence level.

2) Plan, Delegation, SLAs, Risk Register, and Cadence

• Team structure and SLAs

  • Analytics IC (A): Owns metric definitions, queries, QA, CIs, deep-dives; SLA: first-cut metrics by hour 18; refined by hour 36; final by hour 60.
  • Engineering IC (E): Owns instrumentation patches/backfills, schema clarification, query performance and extract; SLA: schema map by hour 10; missing-field workaround by hour 24; final extract stability by hour 36.
  • Lead (You): Decision framing, stakeholder mgmt, narrative, pre-wire, deck; SLA: deck skeleton by hour 24; pre-wire by hours 36–60.

• Work breakdown (high level)

  • A: Build canonical request→quote→hire dataset; compute speed/quality/volume metrics; run stratified cuts; compute CIs; write metric notes.
  • E: Snowflake schema discovery; fix or backfill missing experiment fields; build views/materialized tables; provide performance-optimized datasets.
  • You: Decision brief; risk register; RACI; stakeholder pre-wire; storyline; exec summary; final recommendation.

• Risk register (top items with mitigations)

  • Missing experiment assignment for a subset of events → Mitigate via joining to assignment tables, cookie/user/device stitching, or impute via first treatment exposure; mark confidence as medium.
  • Stale dashboards → Ignore; rebuild from raw with validated logic.
  • Seasonal/category mix shifts → Use stratification and difference-in-differences vs. unaffected cohorts.
  • Sparse lagging outcomes (e.g., post-hire ratings) → Use leading proxies (support rate) with bounds; plan 2-week validation.
  • Time/bandwidth → Maintain a cut list of non-critical analyses; escalate only if it changes the decision.

• Daily checkpoint cadence (72-hour timeline)

  • Day 1 (hours 6–24): 15-min morning stand-up; 30-min EOD review; async mid-day updates in channel.
  • Day 2 (hours 24–48): Same cadence; add 30–60 min pre-wire sessions.
  • Day 3 (hours 48–72): Dry run; exec pre-read; final polish.

• RACI snapshot (bulleted by workstream)

  • Metric definitions and SQL: R = A; A = You; C = PM, Ops; I = Sales, E
  • Instrumentation/backfills: R = E; A = E; C = You, PM; I = Ops, Sales
  • Decision brief and thresholds: R = You; A = You; C = PM, Ops, Sales; I = Eng
  • Analysis and CIs: R = A; A = You; C = PM; I = Ops, Sales, E
  • Deck + pre-wire: R = You; A = You; C = PM, Ops, Sales; I = Eng
  • Go/no-go meeting: R = You; A = VP; C = PM, Ops, Sales; I = Eng, A

3) Data Triage Under Ambiguity

• Schema discovery (Snowflake)

  • Inventory relevant objects:
    • information_schema.tables / columns and SHOW commands
    • Pattern search: table_name ILIKE '%quote%', '%request%', '%hire%', '%experiment%'
  • Example:
    • SELECT table_schema, table_name, row_count, last_altered FROM information_schema.tables WHERE table_schema IN ('PROD','ANALYTICS') AND table_name ILIKE '%quote%';

• Freshness and completeness checks

  • Freshness: SELECT DATEDIFF('hour', MAX(event_time), CURRENT_TIMESTAMP) AS hours_since_last FROM schema.events_quotes;
  • Completeness: NULL rates per critical fields (request_id, pro_id, variant, timestamp).
  • Uniqueness: COUNT(*) vs COUNT(DISTINCT request_id, pro_id, event_time) for duplicate detection.
  • Referential integrity: Anti-joins for orphan events.

• Build a canonical fact set

  • requests (request_id, created_at, category, geo)
  • quotes (request_id, pro_id, quoted_at, quote_metadata)
  • assignments/experiments (entity_id, variant, start_at)
  • hires (request_id, hire_at, amount, outcome)
  • Join keys: request_id primary; handle device/user stitching cautiously.

• Reconcile conflicting sources (hierarchy of truth)

  • Event logs as primary for timestamps; transactional tables for outcomes (hire, amount, refunds).
  • If experiment logs are missing variant for X% rows:
    • Join to user/assignment table by request or session; if still missing, infer by exposure window or exclude from variant-level cuts but include in overall trend analysis; quantify impact.

• Document data quality gaps with impact assessment

  • Template per issue: description, affected metrics, severity (blocker/high/medium/low), direction of bias, mitigation, residual risk.
  • Example: 18% quotes missing variant → affects variant comparisons; severity medium; mitigation join to assignment table reduces to 6%; residual risk noted; run scenario bounds assuming missing skewed entirely to control/treatment.

• Contingency paths

  • If TTFQ timestamps inconsistent, compute from request created_at to earliest quote or first message time; cross-check with server logs.
  • If hire flags lag, use 3-day hire as leading proxy with historical uplift factor to 7-day (validated from prior months), and show sensitivity range.
  • If session-based assignment is noisy, switch to request-level intent cohorts and analyze difference-in-differences vs. categories not exposed to the new workflow.

4) Narrative and Persuasion

• Deck storyline

  • 1-page exec summary
    • Decision ask; recommended option (A/B/C); headline metrics vs. thresholds; confidence level; risks and mitigations; rollout + rollback plan.
  • Context and method
    • What changed (new quoting workflow); cohorts, timeframe, unit of analysis; known data gaps and mitigations.
  • Metric deep-dives (speed, quality, volume)
    • Overall and key segments (top categories, new vs returning pros, peak vs off-peak hours, top geos).
    • Show distributions (e.g., TTFQ median and P90); include 90% CIs.
  • Trade-off analysis
    • Efficiency frontier: speed gains vs. hire rate/quality impacts.
    • Sensitivity to missingness/seasonality; scenario bounds (best/base/worst).
  • Recommendation with confidence intervals
    • Example framing: "Expand to 50% in top-5 categories; hold in long-tail; revisit in 2 weeks when lagging outcomes mature; expected TTFQ −12% [−15%, −8%], hire rate −0.2pp [−0.6pp, +0.1pp]; guardrails within limits."
  • Implementation plan
    • Owners, milestones, monitoring, rollback triggers.

• Confidence intervals (quick methods)

  • Proportions (e.g., hire rate p): 90% CI via Wald or better, Wilson:
    • CI(p) ≈ p ± z_0.95 * sqrt(p(1−p)/n), with z_0.95 ≈ 1.645.
  • Difference in proportions Δ = p_t − p_c: CI(Δ) ≈ Δ ± z * sqrt(p_t(1−p_t)/n_t + p_c(1−p_c)/n_c).
  • Medians (TTFQ): bootstrap by request-level resampling; report median delta CI.
  • Small numeric example: If hire rate increases from 12.0% (n=10,000) to 12.5% (n=10,000), Δ=+0.5pp. SE≈sqrt(.125*.875/10k + .12*.88/10k) ≈ 0.0046 → 90% CI ≈ 0.5pp ± 0.76pp → [−0.26pp, +1.26pp].

• Pre-wire strategy

  • 1:1 pre-reads with PM (speed), Ops (quality), Sales (volume) by hours 36–60.
  • Ask for explicit buy-in on thresholds and rollout phasing; capture dissent and propose guardrails.

• Handling live pushback

  • Prepare alt scenarios slides:
    • Option B (phased expansion) with narrower guardrails.
    • Option C (hold) with a dated plan to close gaps.
  • Use "If true, then do" logic: If hire rate CI lower bound < −1.0pp, then phase by categories with strongest speed gains and stable quality; monitor daily.

5) Decision and Follow-Through

• Go/no-go criteria (example; customize to signed brief)

  • Go (partial): TTFQ median improves ≥ 10% with 90% CI lower bound ≥ 5%; 7-day hire rate CI lower bound ≥ −1.0pp; support tickets ≤ +10%; spam ≤ +5%; no category shows both hire decline >1pp and support increase >10%.
  • Go (full): Same as above across top-10 categories; no critical guardrail breached; data gaps <10% residual missingness on variant.
  • No-go: Any critical guardrail breached or speed gains <5% (lower bound), or substantial unresolved data issues that could flip sign.

• Owner assignments

  • DRI for decision and measurement: You
  • Rollout: PM (schedule), Eng (feature flag/rollout tooling), Ops (pro comms/quality ops), Sales (messaging), Analytics IC (monitoring)

• 2-week post-decision plan

  • Monitoring (daily for week 1; 3x per week for week 2)
    • Dashboards: TTFQ, hire rate, support tickets, spam flags, GMV per request, pro retention proxy.
    • Alerting thresholds: e.g., hire rate −1.0pp day-over-day for 2 consecutive days triggers review.
  • Validation tasks
    • Replace proxies (e.g., support rate) with lagging true outcomes (ratings, refunds) as they mature.
    • Category/geo deep-dive for outliers; tighten eligibility if needed.
  • Incident rollback plan
    • Keep feature-flagged; one-click rollback to previous workflow.
    • Runbook: who flips (Eng), who communicates (PM/Ops), who postsmortems (You + A), SLA: rollback within 30 minutes of breach.

6) Leadership Behaviors Under Time Pressure

• Model calm and focus

  • Publish the plan and thresholds early; remind the team we need decision-quality, not perfection; use a cut list for non-essentials.
  • Timebox debates; decide with the best available data and documented assumptions.

• Protect IC focus

  • Block calendar for heads-down windows; route stakeholder questions to you; batch feedback at checkpoints; freeze metric definitions after hour 12 unless a material error.

• Escalate without churn

  • Escalate only with clear options and implications ("We can patch missing variant today with 6% residual missingness; accept medium confidence or delay 24h for a cleaner backfill").
  • Document decisions in the channel; avoid revisiting closed topics unless new data changes the expected outcome.

Practical Analytics/Engineering Details

• Metric computation tips

  • TTFQ: For each request, TTFQ = MIN(quote_at) − request_created_at; compute median and percentiles by variant and segment.
  • Hire rate: hires_7d / requests; ensure consistent lookback window; censoring: exclude last 7 days from window or use Kaplan–Meier for partial.
  • Support rate: support_tickets / requests; tag by request_id to avoid double counting.

• Sampling/QA checks

  • Randomly sample 100 requests; manually verify TTFQ and quote counts across sources; check 5–10 edge cases (no quotes, many quotes, long TTFQ).
  • Compare to historical baselines; if deltas are implausible (e.g., TTFQ –70%), investigate timestamp timezone or duplication.

• Sensitivity/robustness

  • Stratify by category volume quartiles; if long-tail drives negative effects, propose phased rollout.
  • Difference-in-differences: compare exposed vs. unexposed categories over the same time window to control for seasonality.

• Communication guardrails

  • Always pair a metric with its CI and a plain-English interpretation.
  • Call out known biases and show scenario bounds: best/base/worst.

This approach gets you fast alignment, credible decision-quality analytics despite data ambiguity, a persuasive story tuned to each stakeholder, and a safe operational plan with monitoring and rollback.