Describe your operations experience and impact

Operations Experience (Behavioral & Leadership — Technical Screen)

Provide a concise but concrete overview of your production operations experience as a software engineer. Address the following:

1. On-call participation
   • Rotation model (size, hours, primary/secondary), responsibilities, and page hygiene.
2. Incident response and management practices
   • Severity definitions, roles (e.g., incident commander), communication, escalation, and tooling.
3. SLO/SLI definition and tracking
   • Key SLIs, SLO targets, error budget policy, alerting strategy, and reporting cadence.
4. Runbook creation and maintenance
   • Structure, content, ownership, testing, and where you store them.
5. Change management
   • Release process, approvals, canary/progressive delivery, rollback, freezes, and risk controls.
6. Tooling used
   • Observability, alerting/on-call, incident management, CI/CD, change control, and documentation.
7. A high-severity incident example
   • Your role and decisions, timeline, key actions, resolution, postmortem outcomes, and measurable impact.

Use specific metrics (e.g., MTTA/MTTR, % error rate, latency percentiles, availability) and concrete examples. A STAR structure (Situation, Task, Action, Result) is encouraged.

Solution

How to Answer: Structure, Examples, and Best Practices

Below is a step-by-step guide and a sample answer you can adapt. Aim for a 60–90 second overview, then be ready to dive deep on any area.

1) 60-second overview template

• Scope: “I’ve spent X years owning services in production, participating in a Y-person on-call rotation.”
• Incident practice: “We use SEV levels, an incident commander model, and blameless postmortems.”
• Reliability: “I defined SLIs/SLOs for A and B; we track error budgets and burn-rate alerts.”
• Operations assets: “I maintain runbooks/playbooks and automated remediation for common faults.”
• Change management: “We ship via canaries/feature flags with auto-rollback guards.”
• Tooling: “PagerDuty, Datadog/Prometheus/Grafana, Splunk/ELK, GitHub Actions/Argo, incident.io/Statuspage.”
• Impact: “Reduced MTTR by Z%, cut alert noise by W%, maintained 99.9x% availability.”

2) Detailed talking points and examples

On-call participation

• Rotation: 6–10 engineers, weekly primary/secondary coverage, handoffs with context notes and current hot issues.
• Hygiene: Alert review weekly; consolidate duplicate alerts; convert recurring alerts to engineering work.
• Metrics: MTTA (mean time to acknowledge) target < 5 minutes; page volume ≤ 1 actionable/night on average.

Incident response & management

• Severity: SEV-1 (critical user/business impact), SEV-2 (degraded), SEV-3 (localized/limited).
• Roles: Incident Commander (IC), Communications Lead, Ops/SME Leads. IC coordinates; others execute and update stakeholders.
• Flow: Auto-page → IC appoints roles → war room (Slack/Zoom) → status updates cadence (e.g., 15 min) → mitigation first, then diagnosis → postmortem within 3–5 business days.
• Tools: PagerDuty/OPSGenie, Slack with incident bot, Zoom/Meet, status page, incident.io/FireHydrant for timelines and follow-ups.

SLO/SLI definition and tracking

• SLIs (examples):
  • Availability: request success rate (non-5xx responses) from user edge.
  • Latency: p95 or p99 below threshold for key endpoints.
  • Correctness: ratio of validated successful outcomes (e.g., payment succeeds end-to-end).
• SLOs (examples):
  • 99.9% monthly availability (error budget ≈ 43.2 minutes/month).
  • p95 latency < 300 ms for read endpoints, < 800 ms for write endpoints, 99% of the time.
• Error budget policy: If burn rate is high, slow or freeze changes; prioritize reliability work.
• Alerting: Multi-window, multi-burn-rate alerts (e.g., fast-burn 1-hour window and slow-burn 6–24-hour window) to reduce noise and catch both spikes and smoldering issues.
• Reporting: Weekly SLO review; monthly error budget report and top reliability risks.

Formulas (plain):

• Availability = 1 - (minutes of user-impacting downtime / total minutes)
• Error budget used (%) = (allowed errors consumed / total allowed errors) * 100

Runbooks

• Structure:
  • Context: service, owners, dependencies, dashboards, logs, run commands.
  • Symptoms & diagnosis: what to check first (graphs, logs, feature flags, recent deploys).
  • Remediation steps: safe, ordered, reversible actions with expected outcomes.
  • Rollback/disable steps: scripts/commands and validation checks.
  • Escalations: who and when; vendor contacts.
• Quality: Version-controlled (Markdown), linked from alerts, tested via game days, includes time-boxes (e.g., spend ≤ 10 minutes on path A before escalating).

Change management

• Process: PR with 2 reviewers; CI tests; deploy to staging → canary (1–5%) → progressive rollout (25/50/100) with automated health checks.
• Risk controls: Feature flags for risky paths, automatic rollback on SLO breach, deployment freeze during major events, RFCs for high-risk changes.
• Metrics: Change fail rate, mean time to restore (MTTR), deployment frequency, rollback rate.

Tooling (examples)

• Observability: Datadog or Prometheus + Grafana; OpenTelemetry for traces.
• Logging: Splunk or ELK; distributed tracing via Datadog APM/Jaeger.
• On-call/alerting: PagerDuty/OPSGenie.
• CI/CD: GitHub Actions, Jenkins, Argo Rollouts/Argo CD/Spinnaker.
• Config/infra: Terraform, Helm, Kubernetes.
• Incident management: incident.io/FireHydrant; Statuspage/Uptime checks.
• Docs/runbooks: Markdown in repo, Confluence/Wiki, links embedded in alerts.

3) Sample high-severity incident (STAR)

• Situation: Peak traffic on a Friday, our Checkout API started returning elevated 5xx errors and p95 latency spiked from 250 ms to 4 s. Error budget for the month was at 60%; revenue at risk was high.
• Task: As on-call primary, I took the Incident Commander role within 3 minutes. Goals: stop the bleeding, restore service, and minimize financial/user impact.
• Actions:
  • Declared SEV-1, opened a Slack incident channel, assigned Comms Lead and SME for payments.
  • Froze deployments and enabled maintenance mode for non-critical features (degrade recommendations, not checkout).
  • Used dashboards to correlate a spike in cache misses with a rollout 20 minutes prior. Suspected cache thrash causing DB saturation.
  • Executed runbook steps: toggled a feature flag to revert to the prior pricing cache strategy; scaled read replicas; set connection pool caps and enabled rate limiting to protect the primary DB.
  • Verified improvement via SLIs (5xx rate and p95 latency). Rolled back the canary to the previous version.
  • Scheduled a postmortem and captured a clean incident timeline using incident tooling.
• Results:
  • Time to acknowledge (MTTA): 2 minutes; time to mitigate: 12 minutes; full recovery: 31 minutes.
  • Peak 5xx: 3.1% of requests for ~18 minutes; estimated revenue impact reduced by ~40% due to rapid mitigation.
  • Root cause: config change reduced cache TTL from 15 minutes to 15 seconds during a rollout, causing cache evictions and DB overload.
  • Postmortem outcomes:
    • Added pre-deploy config validation and canary guardrails for cache config.
    • Implemented circuit breakers and backpressure to protect DB.
    • Introduced burn-rate alerts tied to availability and latency SLIs.
    • Authored a detailed cache-thrashing runbook and built a synthetic check for cache churn.
  • Measurable impact over next quarter: MTTR reduced from 42 → 14 minutes (−67%), paging volume −35%, availability improved from 99.88% → 99.95%, zero SEV-1 incidents for 2 quarters.

4) Common pitfalls and guardrails

• SLIs measured at the wrong point (e.g., upstream vs user edge) can hide problems. Measure user-perceived outcomes.
• Overly tight SLOs cause alert fatigue; set targets that reflect desired reliability and realistic budgets.
• Alerting on every internal metric creates noise; prefer SLO burn alerts and a few high-signal symptoms.
• Runbooks without time boxes cause thrash; define escalation triggers and limits.
• Canary without automated health checks is just staging in production; enforce objective rollback criteria.
• Practice: Run game days, chaos tests, and DR failovers to validate runbooks and paging paths.

5) Quick template you can adapt

• On-call: “8-engineer rotation, weekly primary/secondary, PagerDuty. Reduced pages per week from ~14 to ~8 by consolidating alerts and fixing top 3 recurring issues.”
• Incident management: “SEV-1/2/3 model with IC/Comms/SME roles; 15-min update cadence; blameless postmortems within 5 days.”
• SLO/SLI: “99.9% availability and p95 latency SLOs; track error budgets and multi-window burn alerts; weekly SLO review.”
• Runbooks: “Markdown in repo, linked in alerts; tested quarterly via game days; include rollback and safety checks.”
• Change management: “Trunk-based, 2 approvals, canary + progressive delivery, feature flags, auto-rollback on SLO breach.”
• Tooling: “Datadog, Grafana/Prometheus, Splunk, PagerDuty, GitHub Actions, Argo Rollouts, incident.io.”
• Impact: “MTTR −60%, alert noise −30%, 99.95% availability over last 6 months.”

Use the STAR method for your incident example, quantify outcomes, and be ready to deep-dive into any decision, metric, or tool you mention.