Meta Production Engineer Troubleshooting Interview Guide: Linux, Networking, and Debugging

A Meta Production Engineer troubleshooting interview asks you to explain how a production system behaves when something goes wrong. Expect to scope a symptom, compare a failing path with a healthy one, choose a useful probe and explain what its result means. Knowing Linux commands helps. Choosing the right next command matters more.

Meta describes Production Engineers as engineers who work across software and systems to improve reliability, scalability, performance and security. Its Production Engineering overview supports that description; it does not establish a fixed interview schedule. Confirm the rounds, tools and depth in your recruiter’s instructions.

The exercise below turns one of this guide’s main preparation topics—regional latency—into an investigation you can attempt before reading the answer. It is an original practice scenario, not a reported Meta interview question.

What should you prepare for?

Prepare for troubleshooting, Linux internals, networking, practical coding and incident ownership. Infrastructure design may also be relevant to your role. A network-specialist position can require much deeper routing knowledge than a general Production Engineer position; use the job description to set that boundary.

For each topic, practise explaining a mechanism. High load does not establish CPU saturation. Low free memory does not establish a leak. A successful TCP connection does not establish a healthy application. A strong answer connects the symptom to evidence and keeps track of what remains uncertain.

Bring a project where you owned a production problem. Be ready to explain the first symptom, the misleading clue, the decision you made, the risk of that decision and how you checked the result. Avoid attributing the entire team’s work to yourself.

Use SCOPE to keep the investigation organized

SCOPE is this guide’s practice framework, not Meta’s published scoring rubric.

State the symptom and protect users. Ask when the problem began, which users or operations are affected, and whether impact is growing. Consider a reversible mitigation while diagnosis continues. Before shifting traffic, check spare capacity and whether the destination shares the suspected dependency.

Compare healthy and unhealthy paths. Hold something constant: the same asset, request type, client version or workload. Compare by region, host, release and dependency. Otherwise, you may mistake different traffic for different system behavior.

Observe one layer at a time. Move through resolution, connection, TLS, application and dependencies. Choose a probe that separates plausible causes. State what result would send you down each branch.

Prove or disprove the leading hypothesis. Correlate a request ID, timestamp and configuration version. Treat deployment timing as a clue until you can explain the mechanism. Keep at least one competing explanation in view.

Execute and verify. Choose an action with a bounded blast radius and a way back. Measure recovery using the original user symptom, then check errors, saturation and the destinations receiving extra work. A restart that changes a dashboard is not enough.

A regional-latency mock you can run in 25 minutes

One region begins loading images slowly at 14:02. Other regions remain healthy. The same asset takes about two seconds on an affected request. Error rates barely move.

All numbers, names and diagnostic output in this exercise are hypothetical. They describe a simplified Linux edge service, not Meta’s infrastructure. In a practice session, reveal each evidence packet only after the candidate asks a question that would obtain it.

Initial observation	Healthy region	Affected region
Image-request p99	210 ms	2,400 ms
HTTP 5xx rate	0.1%	0.2%
Mean edge-host CPU	26%	28%
Origin-bound requests per host	120/s	120/s
New edge configuration	Unchanged	Applied at 14:02

Take three minutes to state the impact and list no more than three hypotheses. Then choose one comparison. Do not diagnose from the configuration timestamp alone.

A useful opening is: “The failure is regional and latency-led. I want to compare the same asset through a healthy and affected edge, separating connection time from time waiting for the response. I would also check whether we can safely reduce traffic on the affected edge while we investigate.”

Evidence packet 1: where does the request wait?

The interviewer supplies representative client timings. These are cumulative times since the request began, like curl’s timing fields; subtract adjacent values to estimate individual stages.

                   healthy     affected
time_namelookup       0.004        0.004
time_connect          0.022        0.023
time_appconnect       0.046        0.047
time_starttransfer    0.135        1.947
time_total            0.197        2.010

DNS, TCP setup and TLS are similar. The affected request spends roughly 1.9 seconds between completed TLS and the first response byte. That makes a connection-establishment problem a weaker explanation for this request. It does not rule out every network problem: the edge might still wait on a slow connection to an upstream service.

Ask for the same request’s edge and origin spans. Useful questions are: “How long did it wait before an origin connection was assigned?” and “Once sent, how long did the origin take to respond?” Those measurements distinguish local queueing from upstream execution.

Saying “the network is fine” would overstate this evidence. Say which part of the path looks healthy and which part still needs measurement.

Evidence packet 2: normal CPU hides a finite resource

The second packet compares edge-to-origin behavior:

                         healthy     affected
origin_pool.max               64            8
origin_pool.busy               10            8
origin_pool.waiting             0           72
origin_pool.acquire_p99_ms      6         1860
origin_service_mean_ms        80           80
tcp_retransmission_delta       low          low

The origin is not taking longer to execute a request. The affected edge is waiting to obtain a connection. Its pool is at the configured limit, and the acquisition delay explains most of the client delay.

Check the arithmetic. Assume each connection handles one active request at a time. With eight connections and a simplified mean occupancy of 80 ms, the pool can complete about 8 / 0.08 = 100 origin requests per second. The offered load is 120/s. Under these assumptions, the pool cannot keep up even though host CPU is mostly idle. This is a rough capacity model; real occupancy, burstiness and multiplexing must be measured.

The configuration diff shows origin_pool.max changed from 64 to 8 at 14:02. Now the timing, changed limit and measured queue agree. That is stronger evidence than “a deployment happened.”

What Linux output would add useful evidence?

An established-socket count from ss can support the connection picture. Socket statistics do not tell you how many application requests wait for a pooled connection, so ask for the client-library metric too. The ss manual explains the tool’s socket-level view.

For example, ask to inspect established TCP connections whose peer port is 443:

ss -tan state established '( dport = :443 )'

Two illustrative rows from the affected host might look like this:

Recv-Q Send-Q Local Address:Port   Peer Address:Port
0      0      192.0.2.20:51001      192.0.2.10:443
0      0      192.0.2.20:51002      192.0.2.10:443

The addresses are documentation placeholders, and these are selected rows, not the complete eight-connection listing. Confirm the origin address and process before attributing sockets to this pool. Zero socket queues in a snapshot can coexist with requests waiting inside the application. That distinction explains why the pool’s waiting counter is the more direct probe here.

Use process/thread profiles if CPU remains a plausible competing cause. Compare memory pressure, file-descriptor limits and downstream errors if the pool cannot create the connections it is allowed to create. Do not raise every limit merely because a queue exists.

Evidence packet 3: choose a safe mitigation

The origin team confirms capacity headroom, and the previous pool limit is available for rollback. Restore the known-good configuration on one affected edge group first. Monitor origin load and connection counts while comparing that group with the affected control group.

Another option is to drain traffic from the affected group, provided healthy groups have sufficient headroom. Do not combine unrelated mitigations immediately: if several changes happen together, it becomes harder to identify what helped or to undo a harmful action.

The exercise’s canary produces this result:

Check	Before canary	After canary
Image-request p99	2,400 ms	230 ms
Pool-acquisition p99	1,860 ms	7 ms
Waiting requests	72	0
HTTP 5xx	0.2%	0.1%

Watch a sustained window and the affected cohorts, not one successful request. Check that origin saturation and errors remain acceptable before extending the rollback. Preserve the configuration diff and representative traces for the incident review.

Prevention follows the failure mechanism: validate concurrency settings, test at expected traffic and bursts, canary configuration changes and alert on pool wait. A fleet-average CPU alert would have missed this incident.

What would a different packet change?

If TCP setup were slow, inspect address selection, SYN/SYN-ACK timing, retransmissions, route changes and reachability. If connection setup were fast but origin execution were slow, investigate the origin’s queues, storage, locks or dependencies. If only one asset were affected, compare cache state and the asset’s storage path.

If the pool limit were unchanged, the queue would still be real. You would need to explain why connections remain occupied longer, why demand rose or why usable connections fell. The metric identifies the constrained resource; it does not automatically identify the change that constrained it.

These variations are useful practice. Ask a partner to change one packet and make you revise the next probe. The objective is to respond to evidence, not remember that “latency means connection pool.”

Prepare Linux and coding alongside the incident work

For CPU, distinguish runnable work, blocked tasks and per-thread hot spots. For memory, compare resident growth, allocation behavior, cgroup limits and pressure rather than declaring a leak from one number. Linux pressure stall information measures time workloads lose to resource contention; it complements utilization counters.

For disk problems, distinguish bytes, inodes, latency and deleted files held open. Be able to explain why df and du can disagree. For startup failures, inspect the exit status, logs, permissions, ports and resource limits before restarting.

For practical coding, write a bounded-memory log parser or streaming alert. Define malformed-line behavior, window boundaries and out-of-order input before implementation. Test an empty stream, duplicate records and a boundary timestamp. Explain how much state grows with the input and what assumption keeps it bounded.

Meta’s public oomd work illustrates pressure-aware operation. It is an engineering example, not evidence that your interview will ask you to implement that service.

Turn the next seven sessions into reviewable practice

Session	Work	Evidence to keep
1	Explain three Linux failure mechanisms	Symptoms, probes and expected results
2	Trace DNS, TCP, TLS and application timing	One annotated request timeline
3	Attempt the regional-latency mock	Your hypotheses before each reveal
4	Write a streaming parser or alert	Code, boundary tests and memory explanation
5	Repeat the mock with a changed packet	Revised diagnosis and rejected explanation
6	Rehearse an infrastructure design and incident story	Capacity assumption, failure path and personal decisions
7	Run a timed mixed session	Recording or notes and two specific improvements

The existing practice bank includes Troubleshoot a production server outage, Troubleshoot a Midnight Web Server Outage, Troubleshoot a website outage with disk full and Implement a Streaming VMStat Alert. Choose the prompt that exposes your weakest mechanism and attempt it before consulting the solution. These are preparation records, not a promised interview set.

Score the mock on reasoning, not command count

This is an editorial self-check, not Meta’s internal rubric. Give yourself one point for each behavior: scoped impact; compared a control; chose a discriminating probe; interpreted cumulative timing correctly; explained the pool-capacity assumption; tested a competing hypothesis; bounded the mitigation; verified client and downstream recovery.

Review every missing point with a specific correction. “Learn more networking” is too vague. “I ruled out the whole network after seeing a fast client handshake” tells you what to practise next. Re-run that branch and state exactly what the evidence proves.

Sources and Further Reading

• Meta: Production Engineering
• Meta: Open-sourcing oomd
• Linux kernel: Pressure Stall Information
• ss manual: socket statistics
• curl: transfer timing fields

Research checked on 8 October 2026. The incident, outputs, capacity figures and scorecard are original practice material. Employer instructions determine your actual interview format.