Decide under adverse signals and conflicts

Scenario: Pre-Launch Decision Under Mixed Signals

You are preparing to launch a new messaging/notifications feature. Leading indicators are mixed: some guardrails look risky (e.g., unsubscribe complaints trending up), while mission-aligned outcomes are promising but lagging (e.g., offline connections, long-term retention).

Answer the following:

1. Pre-commitment plan

• Describe a rigorous pre-commitment process to define success and stop-loss criteria before launch.
• Make explicit the trade-offs between department-level metrics (e.g., CTR, send volume) and company-level metrics (e.g., retention, time-on-site).

2. Decision memo

• Provide a clear memo structure to argue for launch vs. no-launch when leading indicators are adverse.
• Include risk mitigation, phased rollout, and contingency rollback triggers.

3. Cross-functional communication

• Explain how you would communicate this plan to engineering and cross-functional partners to secure alignment despite conflicting metrics.

Solution

Overview and Assumptions

• We are evaluating a new notifications/messaging feature intended to increase meaningful engagement.
• Department-level metrics (DL): CTR, send volume, open rate.
• Company-level metrics (CL): retention, time-on-site, complaints/unsubscribes (negative), long-term connections.
• Mixed signals: some guardrails worsening (unsubscribe complaints ↑), while mission outcomes are positive but lagging.
• Goal: Pre-commit objective criteria, decide a cautious rollout, and align partners.

---

1) Pre-commitment Process: Success and Stop-Loss with Explicit Trade-offs

1. Define the Objective and Metric Hierarchy

• Primary Objective: Maximize long-term user value subject to safety/brand constraints.
• Metric hierarchy (lexicographic priority):
  1. Hard guardrails (must-pass): legal/compliance/privacy, complaint/unsubscribe thresholds, crash rates, deliverability reputation.
  2. Company-level outcomes: retention (D7/D28), time-on-site/session depth, meaningful social interactions.
  3. Department-level metrics: CTR, send volume, open rate.
• Rationale: Company-level outcomes dominate; DL metrics are inputs, not ends.

2. Choose Decision Framework

• Option A: Lexicographic (recommended). Pass all guardrails; then require CL metrics to be neutral or positive; only then consider DL improvements.
• Option B: Weighted OEC (Overall Evaluation Criterion) when trade-offs must be quantified: OEC = w1*(ΔRetention) + w2*(ΔTime-on-site) − w3*(ΔComplaints) + w4*(ΔMeaningful Interactions) + w5*(ΔCTR) with w3 ≫ w5 to encode brand/safety priority. Pre-commit weights from historical LTV studies.

3. Quantify Trade-offs Using LTV/Economic Terms

• Estimate value of outcomes and cost of harms: Expected Net Impact per user = LTV_gain_from_retention − Cost_of_churn/complaints − Reputational/Deliverability risk penalty + Short-term engagement value
• Example (illustrative numbers):
  • +0.20 pp D28 retention ⇒ +$0.30 LTV
  • +0.8% time-on-site ⇒ +$0.05 LTV
  • +0.05 pp unsubscribe complaints ⇒ −$0.40 LTV
  • +5% CTR ⇒ +$0.03 LTV (only if not cannibalizing) Net = 0.30 + 0.05 − 0.40 + 0.03 = −$0.02 ⇒ Do not launch unless we mitigate complaints.

4. Pre-define Success, Neutral, and Stop-Loss Criteria

• Hard Guardrails (stop-loss if any breached):
  • Complaints/Unsubscribes: Δ ≥ +10 bps over baseline for ≥12 consecutive hours or ≥2x SEM above baseline.
  • Deliverability/Reputation: bounce/spam trap rate ≥ X threshold; blocklist incidents any ⇒ rollback.
  • Latency/Errors: p95 latency > Y ms or error rate > Z% for 2 ramps ⇒ hold.
• Company-Level Outcomes (must be non-negative within CI):
  • D7 retention: Δ ≥ 0 or 95% CI excludes −MDE_neg (e.g., −0.05 pp). If lagging, use validated proxy models (see below).
  • Time-on-site/session depth: Δ ≥ 0 or neutral within alpha-spending plan.
• Department-Level Targets (nice-to-have, cannot override guardrails):
  • CTR: Δ ≥ +3% (95% CI > 0) OR CTR × quality (post-click engagement ≥ baseline).
  • Send Volume: capped; no increase if quality drops (quality = downstream engagement − complaints).

5. Handle Lagging Outcomes with Proxies and Models

• Pre-register proxy metrics with validation:
  • Short-term proxy for retention: 7-day return intent model; serial correlation of session depth; meaningful interactions per active day.
  • Backtest: R² and calibration on prior launches; pre-commit acceptance bounds (e.g., proxy predicts ≥0 effect with 90% PI).

6. Power, MDE, and Monitoring Plan

• Compute sample sizes for guardrails and CL outcomes; ensure each ramp has enough exposure to detect harmful deviations.
• Sequential monitoring with alpha spending (e.g., Pocock/O’Brien-Fleming) to avoid p-hacking.
• Heterogeneity checks pre-committed (new vs. long-tenure users, high-frequency vs. low-frequency, region).

7. Rollout and Rate-Limit Caps (pre-committed)

• Per-user rate limits (e.g., ≤1 new notification/day; quiet hours; frequency cap by channel).
• Content quality filters (only top decile relevance score during early ramps).
• Automatic backoff if complaint rate in last N sends exceeds threshold.

8. Governance and Decision Rights

• DRI (Directly Responsible Individual) named; approvers list.
• Single source of truth doc with thresholds, formulas, dashboards, and runbooks.

---

2) Decision Memo Structure for Launch Under Adverse Leading Indicators

1. Title and Summary (1–2 paragraphs)

• Decision asked: Proceed with phased rollout to X% or hold.
• Current evidence: CTR +5–7%; complaints +6–8 bps; proxies suggest neutral-to-positive retention; long-term outcomes lagging.
• Recommendation: e.g., Proceed to 5% with tightened guardrails and new mitigations; do not exceed 10% unless complaint rate ≤ +3 bps.

2. Context and Goals

• Problem, target users, expected value, risks. Metric hierarchy and OEC definition.

3. Experimental Evidence to Date

• A/A checks passed; data quality verified.
• Results by metric tier: guardrails, CL, DL with CIs and MDEs.
• Heterogeneity: any segments at risk.
• Externalities: deliverability, saturation, cannibalization.

4. Modeled/Proxy Evidence for Lagging Metrics

• Retention proxy model performance, backtests, prediction intervals for current ramp.
• Sensitivity analysis (best/base/worst case) and LTV translation.

5. Risk Assessment and Mitigations

• Risks: complaints, deliverability, legal, privacy, infra load, user trust.
• Mitigations: frequency caps, content relevance floor, quiet hours, onboarding education, improved unsubscribe UX, rate-limited backoff, geo/user cohort scoping.

6. Phased Rollout Plan

• Proposed ramp: 1% (24–48h) → 5% (48–72h) → 10% (72h) → 25% (1 week) → 50% (1 week) → 100%.
• Entry criteria for each ramp: all guardrails within bounds; CL proxies ≥ 0; no high-risk segment regressions.
• Exit/hold criteria: any hard guardrail breach; CL outcomes negative beyond pre-committed MDE_neg; infra/ops alerts.

7. Contingency Rollback Triggers (pre-registered)

• Auto-rollback if:
  • Complaints +≥10 bps for ≥12h window (or 2x SEM);
  • Deliverability blocklist or spam trap spike > threshold;
  • D7 return intent proxy < −0.3σ from baseline for 24h;
  • Repeated p95 latency breaches across 2 checks.
• Manual rollback authority and on-call rotation defined; action time ≤ 60 minutes.

8. Decision Log and Approvals

• DRI, Data Science, Eng Lead, PM, Policy/Legal (if applicable), Support.
• Timestamped sign-offs; dissenting opinions captured.

9. Next Steps and Owner Matrix

• Owners for mitigations, dashboard updates, rollout, and comms.

Appendices: Detailed metrics, model validation, dashboards, and runbooks.

---

3) Communication Plan to Secure Cross-Functional Alignment

1. Upfront Alignment on Principles

• Share the metric hierarchy and OEC; emphasize that company-level outcomes and safety guardrails override local optimizations.
• Conduct a 30-minute pre-mortem: enumerate plausible failure modes and map to mitigations and triggers.

2. Clear Docs and Artifacts

• Living spec/PRD section with: goals, experimentation plan, thresholds, rollout schedule, dashboards, runbook.
• One dashboard per tier: Guardrails (always top), Company-level, Department-level; red/amber/green statuses tied to thresholds.

3. Cadence and Decision Rituals

• Daily 15-minute stand-up during ramps; a single Slack/Teams channel for alerts.
• Checkpoints at each ramp gate to review entry/exit criteria; minutes recorded in the decision log.

4. Roles and Decision Rights (RACI)

• DRI = PM or DS; Approvers = DS Lead, Eng Lead; Consulted = Policy/Legal/Support; Informed = broader org.
• Clarify that any guardrail breach auto-triggers hold/rollback without committee delay.

5. Engineering Partnering

• Translate thresholds into SLOs and automated monitors (alerts, feature flags, circuit breakers).
• Provide event schemas and data quality checks; define on-call rotations and rollback playbook.

6. Handling Conflicting Metrics in Meetings

• Start with guardrail status; then CL outcomes; then DL metrics.
• Use pre-committed thresholds to avoid opinion battles; show sensitivity and LTV translation.
• If adverse leading indicators persist, propose concrete mitigations (e.g., raise relevance threshold, lower send cap) and a re-test plan.

7. Stakeholder Confidence

• Publish a one-page weekly update: ramp status, any incidents, actions taken, forecast vs. actual.
• Invite dissent: document and address concerns explicitly; adjust mitigations if new risks surface.

---

Pitfalls and Guardrails

• Do not let CTR or send volume override complaints/unsubscribe signals.
• Beware heterogeneity: protect vulnerable segments (e.g., new users) with stricter caps.
• Avoid peeking without alpha control; pre-commit analysis plan.
• Validate proxies; do not over-rely on uncalibrated models for retention.
• Ensure privacy and policy reviews are complete; treat these as hard blockers.

Minimal Example of Pre-Commit Table (illustrative)

• Hard stop-loss: Complaints Δ ≥ +10 bps/12h; Deliverability blocklist any; p95 latency > 2x SLO.
• Must-pass CL: D7 return intent proxy ≥ 0; Time-on-site Δ ≥ 0 (CI includes 0 at worst).
• DL goals: CTR Δ ≥ +3% with downstream engagement ≥ 0.
• Ramp gates: 1%→5% only if all above satisfied; hold otherwise, apply mitigations, re-measure 24–48h.

This approach makes the trade-offs explicit, encodes them in pre-committed thresholds, and operationalizes a safe, transparent decision process that partners can execute and trust.