AnchorFree Software Engineer Interview Guide

The 60-second brief

Candidates describe AnchorFree as a consumer security and privacy company known for virtual private network (VPN) technology. The Software Engineer role they describe sits in the network layer of that product: low-latency data transfer, traffic routing and system tuning on servers deployed in many regions. Candidates describe the daily work as VPN performance tuning, debugging network problems in production with the help of packet captures, and kernel tuning for busy servers that hold very many connections at once, alongside security protocols and custom transport layers. Code reviews and written documentation of internal protocols are also part of the described job.

This guide is for engineers preparing for the AnchorFree Software Engineer loop. It covers the three reported stages (Recruiter Conversation, Technical Phone Interviews, Take-Home Test), the networking, Linux, coding and SQL questions candidates report, and a seven-day plan. The requirements candidates describe list C, C++, Go or Python, so the examples lean toward systems code in those languages.

Candidates report three stages over roughly 3-5 weeks: a recruiter conversation, several technical phone screens on core networking, Linux and sockets, and a take-home in which you write a small working network tool. The take-home appears in some loops depending on the team.

Focus areas

• Socket programming
• System calls (syscalls)
• TCP connection establishment

Overview

The role candidates describe is systems and network programming rather than web application work. The listed requirements are proficiency in C, C++, Go or Python, hands-on Linux socket programming, TCP/IP knowledge, and experience debugging network protocols and managing Linux servers. Cryptography basics, SQL optimization and high-concurrency distributed systems are listed as extras. For preparation, that means you should be able to explain mechanisms, not just name them: what the kernel does when a connection arrives, where a packet can be lost between the wire and your process, and why one multiplexing call scales better than another.

Candidates group the technical content into three areas. The first is low-level networking and socket programming, which they describe as the most heavily weighted: the socket lifecycle, the TCP state machine (SYN_SENT, ESTABLISHED, FIN_WAIT, TIME_WAIT, half-open connections), flow and congestion control, and less common topics such as raw sockets, custom protocol headers and reliable transport built on UDP. The second is Linux and kernel interaction: system calls, threads and synchronization, read and write buffers, and kernel parameter tuning for busy servers. The third is software design, where data structures, SQL and take-home structure come in. Reported scenarios include building a client-server program on raw sockets with manual retransmission, keeping latency low when a VPN user moves from Wi-Fi to cellular, tracing what happens inside the operating system on a read() from a socket, and finding a file descriptor leak in a long-running daemon.

The coding and database questions are small and concrete, and they are shaped by the same domain: a SQL query for duplicate active sessions in a connection log, a daily-active-users and average-session-duration query, a thread-safe packet buffer in C or C++, a structure for fast IP lookup with CIDR matching, a cache that evicts the least recently used IP addresses, and reversing a linked list with its complexity. Practise these in a language you can write without an IDE, and practise saying why you chose each data structure, because candidates also report being asked to defend those choices when presenting the take-home.

Use the guide in this order: finish the TCP and socket mechanics first, because the other areas build on them; then work through the Linux path from NIC to application; then write the coding answers and the SQL; and finish with a small take-home style project so the structure, tests and README are habits before you receive the real task.

The AnchorFree interview process

These are the three stages candidates report, in the order they report them. The take-home is described as present in some loops rather than all.

Recruiter Conversation

Candidates report that the loop opens with a conversation covering your background, your interest in the privacy space and your general technical experience. No technical question set is reported for this stage. Treat it as the place to connect your own projects to the work described for the role: low-level systems, Linux servers and network debugging.

What to demonstrate

• Your background and the projects behind it
• Your interest in the privacy space
• Your general technical experience

How to prepare

• Write a short account of your two or three most relevant projects and name the layer each one touched: sockets, kernel parameters, packet captures, concurrency, or storage.
• Prepare your own reason for wanting privacy and VPN software; a specific project or problem you have worked on is more convincing than a general statement.
• Ask which stages your team uses, whether a take-home is included, and which language you should expect to use, so you can plan the following weeks.
• Have your location, notice period and compensation expectations ready in case they come up.

Technical Phone Interviews

After the recruiter call, candidates report several technical phone screens centred on three themes: core networking, Linux administration and programming against sockets. The questions candidates report for the loop as a whole span TCP connection setup and teardown, the control window versus the congestion window, DNS resolution, the socket lifecycle, select and epoll, how the kernel receives packets, user space versus kernel space, virtual memory, diagnosing dropped packets, and coding and SQL items; candidates do not say which stage each one comes from. Over the phone you have to explain mechanisms clearly in words, so practise answering aloud and sketching state diagrams from memory.

What to demonstrate

• Networking fundamentals such as TCP states, flow control and congestion control
• Linux system administration and operating system internals
• Socket-level programming and the system calls it relies on

How to prepare

• Recite the full socket call sequence for a server (socket, bind, listen, accept) and a client (socket, connect), then explain what each call changes inside the kernel.
• Practise the TCP handshake and four-way close with the state names on each side, including why TIME_WAIT exists and what a half-open connection is.
• Write a diagnostic order for a server dropping packets under load: NIC counters, receive queues and softirq processing, socket buffers, then application read speed.
• Rehearse the small coding answers from the reported list in case they come up here: a thread-safe packet queue, CIDR matching, an LRU cache and a linked-list reversal, plus the connection-log SQL queries.

Take-Home Test

Candidates report a hands-on take-home, with a few days to finish, in which you write a small working network tool; it is described as close to the real job. The reported advice is to hand in modular code with error handling, logging, unit tests, build and run instructions and written design decisions. Candidates also describe presenting it in a code review and defending data structure choices, especially memory use and time complexity under heavy concurrency.

What to demonstrate

• Robust, secure, well-structured code with clear documentation, which the reported advice stresses
• Code structure: modules, error handling, logging and unit tests
• Your reasoning about data structures, memory efficiency and complexity

How to prepare

• Pick the language you can write socket code in fastest and rehearse a small TCP or UDP utility end to end, including partial reads and writes, timeouts and shutdown.
• Write unit tests that run over loopback sockets and cover malformed input, a dropped peer and a slow reader.
• Write a README with the compile and run commands and a short design-decisions section naming each data structure and its complexity.
• Prepare a two-minute explanation of one trade-off you made, such as buffer sizing, blocking versus non-blocking I/O, or TCP versus UDP.

Where candidates lose points

Reciting the socket system calls in order without saying what the kernel does at each one

For each call, state the effect: socket() allocates the socket structure and file descriptor and sets the default buffer limits (the send and receive buffers are filled with sk_buffs only as data arrives), bind claims a local address, listen creates the pending-connection queues sized by the backlog, and accept takes a finished connection from that queue. Be ready for the blocking versus non-blocking variants of each.

Treating the TCP control (receiver) window and the congestion window as the same thing

Say that the receiver advertises its window for flow control based on its free buffer space, while the sender keeps a congestion window (slow start, congestion avoidance, reaction to loss) for network capacity, and that the amount in flight is limited by the smaller of the two.

Answering the dropped-packets question by naming a sysctl to raise

Locate the layer before tuning: NIC and driver counters, the ring buffer and softirq processing, socket receive buffer overflow, then whether the application reads fast enough. Say which counter or tool would confirm each guess and what you would change only after confirming.

A thread-safe queue that locks correctly but busy-waits, grows without bound or cannot shut down

Use a mutex with condition variables, re-check the predicate in a loop to handle spurious wakeups, choose a bounded ring buffer with a stated full-queue policy, and include a close or shutdown path that wakes blocked threads.

Submitting a take-home whose happy path works but ignores partial reads and writes, errors, logging and tests

Handle short reads and writes, timeouts and peer resets explicitly, log failures with enough context to debug, add loopback tests for the failure cases, and finish with run instructions and a short note on your design choices.

Questions & practice

Describe how virtual memory works and how it relates to process isolation in Linux.

Medium · Linux Systems & OS Fundamentals

Approach

1. Define the model: every process sees its own virtual address space, and the MMU translates virtual pages to physical frames through per-process page tables (four levels on x86-64 with 4 KiB pages). The TLB caches recent translations so most accesses skip the page-table walk.
2. Explain page faults and demand paging. A mapping created by mmap or malloc gets no physical memory until it is first touched. On the fault, the kernel allocates a zeroed frame, reads the page in from a file or swap, or delivers SIGSEGV if the address is not mapped or the access breaks its permissions.
3. Tie this to isolation. Each process has its own page-table root (CR3 is switched on a context switch, and PCIDs avoid a full TLB flush), so a process cannot even name another's memory. Permission bits enforce read/write/execute (NX) and user/supervisor access, which keeps user code out of kernel pages.
4. Cover controlled sharing. fork() copies the page tables and marks pages copy-on-write, so a frame is duplicated only when one side writes to it. Shared libraries and MAP_SHARED mappings point several processes at the same frames on purpose. Threads share one address space, so isolation exists between processes, not between threads.
5. Mention overcommit: Linux lets processes reserve more virtual memory than exists, which is why VSZ can far exceed RSS, and the OOM killer steps in when touched pages cannot be backed.

Follow-up

• What does fork() cost for a process with a large heap, and why is it much cheaper than copying the heap?
• Why can huge pages speed up a memory-heavy server, and what is the downside of transparent huge pages?
• A process's RSS keeps growing but heap profiling shows no leak. What else could account for it?

Implement a thread-safe queue in C or C++ that could be used for buffering incoming network packets.

Medium · Coding, Data Structures & Databases

Approach

1. Start with a bounded ring buffer protected by one mutex and two condition variables (not_empty and not_full). push() waits while the buffer is full and pop() waits while it is empty, each using wait with a predicate (or a while loop around pthread_cond_wait in C) so spurious wakeups are handled. Signal the other side after changing the count.
2. Keep it bounded and decide what happens when it fills, because a packet path usually cannot block the receiver: offer try_push() that returns false so the caller can drop the packet and count the drop, or overwrite the oldest entry. An unbounded queue just turns a traffic spike into memory exhaustion.
3. Get the API right in C++. A separate front() and pop() races between threads, so pop() should return std::optional<T> or a bool plus an out-parameter. Move packets in, or store pointers to buffers from a preallocated pool, so there is no malloc and copy per packet. Add close(): set a flag and notify_all so blocked consumers can exit cleanly at shutdown.
4. For one producer and one consumer, describe the lock-free version: a power-of-two array with atomic head and tail indices. The producer writes the slot and then stores tail with release ordering, and the consumer loads tail with acquire before reading the slot. Pad head and tail onto separate cache lines to avoid false sharing. Multi-producer lock-free queues need per-slot sequence numbers (Vyukov's bounded queue) and are much harder to get right.
5. State the costs. The mutex version is O(1) per operation but serialises all threads on one lock. Batching (popping many items per lock) or one SPSC queue per producer cuts contention. Name how you would test it: ThreadSanitizer, plus a stress test that checks every item arrives exactly once and in order per producer.

Follow-up

• Why must the condition-variable wait sit inside a loop, and what goes wrong with a plain if?
• How would you shut the queue down without losing or leaking packets that are still buffered?
• Where would false sharing appear in your design, and how would you show it in a benchmark?

How would you reverse a linked list, and what are the space and time complexities of your approach?

Medium · Coding, Data Structures & Databases

Approach

1. Give the iterative version with three pointers: prev = null and curr = head. Each step saves next = curr.next, sets curr.next = prev, then moves prev = curr and curr = next. When curr becomes null, prev is the new head. Check it by tracing a three-node list aloud.
2. State the complexity: O(n) time, because each node is visited once, and O(1) extra space, because the links are rewired in place.
3. Contrast the recursive version: reverse the rest of the list, then set head.next.next = head and head.next = null. It is also O(n) time but uses O(n) stack space, so a very long list can overflow the stack, which matters for systems code with small thread stacks.
4. Test the edge cases: an empty list, a single node, two nodes, and the old head's next becoming null. If the input might contain a cycle, the loop never ends, so either state that the list is acyclic or check first with Floyd's slow/fast pointers.
5. For a doubly linked list, swap each node's next and prev pointers and return the last node visited. Say which variant you are writing before you start.

Follow-up

• Reverse the list in groups of k nodes, leaving a final group shorter than k unchanged.
• Reverse only the nodes between positions m and n in one pass.
• Use reversal to check whether a singly linked list is a palindrome in O(1) extra space.

Write a SQL query to identify duplicate active user sessions from a connection log table.

Medium · Coding, Data Structures & Databases

Approach

1. Pin down the schema and the meaning of 'duplicate' first. Assume something like connection_log(session_id, user_id, device_id, started_at, ended_at), with ended_at NULL while a session is active. 'Duplicate' could mean the same session logged twice, or one user holding more than one active session at once. Say which one you are solving.
2. For concurrent active sessions per user: SELECT user_id, COUNT() AS active_sessions FROM connection_log WHERE ended_at IS NULL GROUP BY user_id HAVING COUNT() > 1. To return the rows themselves, use COUNT(*) OVER (PARTITION BY user_id) in a CTE and filter it to > 1. To flag all but the newest session, use ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY started_at DESC) > 1.
3. If 'active' means overlapping in time rather than still open, self-join on a.user_id = b.user_id AND a.session_id < b.session_id AND a.started_at < COALESCE(b.ended_at, now()) AND b.started_at < COALESCE(a.ended_at, now()). The < on session_id reports each pair once and stops a row matching itself.
4. If the log stores connect and disconnect events instead of sessions, rebuild sessions first: pair each connect with the next event for the same session using LEAD() and treat a connect with no later disconnect as active. Grouping raw events directly double-counts.
5. Cover NULLs and performance. Never compare ended_at = NULL; use IS NULL. COUNT(column) skips NULLs while COUNT(*) does not. In Postgres, a partial index on (user_id) WHERE ended_at IS NULL keeps the active-session query small even when the history table is huge.

Follow-up

• Write the DELETE that removes exact duplicate log rows and keeps one copy of each.
• How would you compute the maximum number of concurrent sessions each user ever had?
• The table has billions of rows and the query times out. What do you change: the index, partitioning, or the shape of the query?

Find version gaps and relay lag with window functions

Medium · window functions · event tables · frame semantics

outbox_event holds event_id, aggregate_type, aggregate_id, aggregate_version, event_type, payload, status ('pending','published','dead'), attempts, created_at, published_at. A projection is missing rows and you must decide whether the relay skipped events or the consumer dropped them. Write three queries over the last seven days: one listing every aggregate_id whose published aggregate_version sequence has a hole, one giving per-day counts with a running total, and one returning the newest published event per aggregate. For each, say where the window function is evaluated relative to WHERE and LIMIT. PostgreSQL 16.

Approach

1. Gaps: compute lead(aggregate_version) OVER (PARTITION BY aggregate_id ORDER BY aggregate_version) in a subquery, then filter next_version <> aggregate_version + 1 in the outer query. Window functions are evaluated after WHERE, GROUP BY and HAVING and before the outer ORDER BY and LIMIT, so the predicate cannot sit in the same WHERE clause and PostgreSQL 16 has no QUALIFY.
2. Say what the seven-day filter does to the answer: it truncates every partition, so the first row per aggregate has no predecessor inside the window and a hole spanning the boundary is invisible. Widen the window, or join to resource.version as the authority for the true maximum.
3. Running total: SELECT date_trunc('day', created_at) AS d, count() AS n, sum(count()) OVER (ORDER BY date_trunc('day', created_at) ROWS UNBOUNDED PRECEDING). An aggregate inside a window call is legal because grouping runs before windowing. The grouping key is unique per row here so ROWS and RANGE agree, but write the frame anyway — over ungrouped rows with tied timestamps the default RANGE frame pulls in every peer row and the total jumps.
4. Newest per aggregate: DISTINCT ON (aggregate_id) ... ORDER BY aggregate_id, aggregate_version DESC is the cheap PostgreSQL-only form when an index matches that order; row_number() OVER (PARTITION BY aggregate_id ORDER BY aggregate_version DESC) = 1 is the portable form and needs a subquery for the same evaluation-order reason as the gap query.

Follow-up

• Relay failover redelivers events. Does a duplicate break the gap query, and how would you detect one from this table alone?
• Turn the gap check into a continuous monitor rather than a query someone runs after an incident. What does it watch?

Explain the step-by-step process of how a TCP connection is established and terminated.

Medium · Network Protocols & Socket Programming

Approach

1. Walk the three-way handshake with sequence numbers: the client sends SYN with its initial sequence number x, the server replies SYN-ACK with its own ISN y and ack x+1, and the client sends ACK y+1. Name the states on each side: client CLOSED -> SYN_SENT -> ESTABLISHED, server LISTEN -> SYN_RECEIVED -> ESTABLISHED.
2. Say what the handshake negotiates besides sequence numbers. MSS, window scale and SACK-permitted are carried only on the SYN and SYN-ACK. Timestamps are agreed on the SYN but then carried on every segment for RTT measurement and PAWS, and SACK blocks appear on later ACKs once SACK is permitted. If a middlebox strips any of these options from the SYN, that feature is off for the whole connection.
3. Walk the four-way close: the active closer sends FIN (FIN_WAIT_1), the peer ACKs it and moves to CLOSE_WAIT while the closer moves to FIN_WAIT_2, the peer sends its own FIN (LAST_ACK), and the closer ACKs it and sits in TIME_WAIT for 2*MSL before reaching CLOSED.
4. Explain why TIME_WAIT exists: it lets the closer retransmit the final ACK if that ACK is lost, and it stops stray segments from an old connection being accepted by a new one on the same 4-tuple. Only the side that closes first pays this cost.
5. Cover the edge cases: simultaneous open and simultaneous close (the CLOSING state), abortive close with RST, half-close with shutdown(SHUT_WR), and half-open connections where one side vanished without a FIN, which only keepalive or a failed write will detect.

Follow-up

• A server shows thousands of sockets in CLOSE_WAIT. What does that tell you, and where do you look in the code?
• How do SYN cookies let a server survive a SYN flood, and what do they cost?
• Why are random initial sequence numbers used instead of starting at zero?

What is the difference between a TCP control window and a congestion window?

Medium · Network Protocols & Socket Programming

Approach

1. Read "control window" as the receive window used for flow control. The receiver advertises rwnd in every ACK, sized from the free space in its socket receive buffer, and this stops a fast sender from overrunning a slow receiver.
2. The congestion window cwnd is sender-side state that is never sent on the wire. It is the sender's estimate of what the network path can carry without loss, and it protects the routers in between rather than the receiver.
3. State the rule that ties the two together: the sender may have at most min(rwnd, cwnd) unacknowledged bytes in flight, so throughput is roughly that window divided by RTT. Whichever window is smaller is the bottleneck.
4. Describe how cwnd moves. In slow start it grows by about one MSS per ACK, doubling each RTT, until it reaches ssthresh. In congestion avoidance it grows about one MSS per RTT. Three duplicate ACKs trigger fast retransmit: Reno sets ssthresh to max(FlightSize/2, 2*MSS), roughly half the current cwnd, sets cwnd to ssthresh (plus 3 MSS during fast recovery), and then continues in congestion avoidance rather than slow start. A retransmission timeout drops cwnd to one segment. Linux defaults to CUBIC, which grows cwnd as a cubic function of time since the last loss.
5. Mention the edge cases. The 16-bit window field caps rwnd at 65,535 bytes unless window scaling is negotiated (shift up to 14, about 1 GiB). A zero window makes the sender use the persist timer and send window probes.

Follow-up

• On a high-latency link, how do you tell whether throughput is limited by rwnd or by cwnd, using ss -ti or a packet capture?
• What is the bandwidth-delay product, and how should it size the socket buffers?
• How does a loss-based algorithm such as CUBIC behave differently from BBR on a lossy wireless link?

How does DNS resolution work, and how would you optimize it for a low-latency application?

Medium · Network Protocols & Socket Programming

Approach

1. Trace one lookup end to end. The application calls getaddrinfo(), which checks nsswitch, /etc/hosts and any local cache before the stub resolver asks the recursive resolver in /etc/resolv.conf. On a cache miss the recursive resolver walks root -> TLD -> authoritative servers, following referrals, and caches every answer for its TTL.
2. Cover the transport details. Queries normally go over UDP port 53. A truncated response (TC bit) forces a retry over TCP, and EDNS0 allows larger UDP payloads. Negative answers (NXDOMAIN) are cached too, for a duration taken from the zone's SOA record.
3. For low latency, the main lever is avoiding round trips: run a local caching resolver close to the process (unbound, dnsmasq, systemd-resolved), pre-resolve or prefetch hot names before their TTL runs out, and reuse long-lived connections so most requests skip DNS entirely.
4. Fix the blocking problem: getaddrinfo() blocks the calling thread, so an event-loop server needs an async resolver (for example c-ares) or a resolver thread pool. Query A and AAAA in parallel and race the connections (Happy Eyeballs) so a broken IPv6 path does not add a timeout.
5. State the trade-offs. Long TTLs cut lookups but slow failover, and serve-stale answers keep things running during resolver outages at the risk of returning old data. In a VPN context, also say that DNS queries must go through the tunnel, or they leak the user's browsing to the local network. DoT and DoH encrypt them at the cost of a TLS handshake, which you pay back by reusing the connection.

Follow-up

• How would you measure how much of a request's latency is DNS, both on the client and in production?
• A record changed but some clients still hit the old IP an hour later. Which caches could be holding it?
• What is a DNS leak in a VPN client, and how would you test for one?

Describe the lifecycle of a network socket and the primary system calls involved in setting up a server socket

Medium · Network Protocols & Socket Programming

Describe the lifecycle of a network socket and the primary system calls involved in setting up a server socket.

Approach

1. Give the server sequence in order and what each call does in the kernel: socket() returns a file descriptor for a protocol endpoint, setsockopt(SO_REUSEADDR) lets a restart bind while old connections sit in TIME_WAIT, bind() attaches the local address and port, listen(backlog) turns it into a passive socket, and accept() returns a new descriptor for each completed connection while the listening descriptor stays open.
2. Explain the two queues behind listen(). Half-open connections wait in the SYN queue (sized by tcp_max_syn_backlog) and completed handshakes wait in the accept queue, capped at min(backlog, net.core.somaxconn). If the application calls accept() too slowly, that queue overflows and the ListenOverflows counter goes up.
3. Cover the data and teardown phase: read/recv and write/send can return short counts, so loop on them, and retry on EINTR. Writing to a peer that has closed raises SIGPIPE, which you avoid with MSG_NOSIGNAL or by ignoring the signal. shutdown() acts on the connection itself, while close() only drops one reference to the descriptor.
4. Contrast blocking and non-blocking mode. With O_NONBLOCK, accept() returns EAGAIN/EWOULDBLOCK when no connection is waiting, and a client's connect() returns EINPROGRESS. You then wait for the socket to become writable and read getsockopt(SO_ERROR) to learn whether the connect succeeded. accept4() with SOCK_NONBLOCK | SOCK_CLOEXEC sets both flags in one call.
5. Mention the client side (socket() then connect(), with an ephemeral port chosen implicitly) and SO_REUSEPORT, which lets several processes each bind their own listening socket so the kernel spreads incoming connections across them.

Follow-up

• What happens to a client's connection attempt when the server's accept queue is full?
• Why does a restarted server sometimes fail with EADDRINUSE, and what exactly does SO_REUSEADDR change?
• How would you find and fix a file descriptor leak in a long-running server built on these calls?

How does the Linux kernel handle incoming network packets at the physical and driver layers?

Medium · Linux Systems & OS Fundamentals

Approach

1. Start at the NIC. The driver gives the card a ring of receive descriptors pointing at pre-allocated memory, the NIC DMAs each frame into the next free buffer, and it raises an interrupt (MSI-X allows one per queue). If the ring is full, the frame is dropped in hardware.
2. Explain NAPI. The interrupt handler does almost nothing: it masks further interrupts for that queue and schedules a poll. The NET_RX softirq then calls the driver's poll function, which drains up to a budget of packets, wraps each in an sk_buff and hands it up. Under load this polling avoids an interrupt storm.
3. Follow the packet up the stack. GRO merges segments of the same flow, RSS (in hardware) or RPS/RFS (in software) spread flows across CPUs, and netif_receive_skb passes the packet to ip_rcv. From there it goes through the netfilter hooks and routing to tcp_v4_rcv or the UDP receive path, lands in the socket's receive queue, and wakes the reader blocked in read() or epoll.
4. Name the drop points and where each one shows up: a full RX ring (ethtool -S), the softirq running out of budget (the time_squeeze column in /proc/net/softnet_stat), the per-CPU backlog overflowing netdev_max_backlog (the dropped column), and full socket buffers (RcvbufErrors in netstat -su).
5. Mention XDP/eBPF, which runs a program in the driver before any sk_buff is allocated. That is where very high-rate filtering or forwarding is done, at the cost of working without the full stack.

Follow-up

• Why does Linux switch from interrupts to polling under load, and what latency does that cost at low load?
• What is the difference between RSS, RPS and RFS, and when would you enable RPS?
• Where would an XDP program run in this path, and what can it not see that a netfilter rule can?

What are the differences between user space and kernel space, and how do system calls bridge them?

Medium · Linux Systems & OS Fundamentals

Approach

1. Define the split. User code runs unprivileged (ring 3 on x86) and can only touch its own mapped pages. The kernel runs privileged (ring 0) with access to all memory and to devices. Kernel pages are marked supervisor-only, and KPTI, added after Meltdown, goes further and gives user mode a separate page table that maps almost no kernel memory.
2. Walk one system call on x86-64. The libc wrapper puts the syscall number in rax and the arguments in rdi, rsi, rdx, r10, r8 and r9, then executes the syscall instruction. The CPU switches to ring 0 at the entry point stored in an MSR, the kernel switches to its own stack, dispatches through the syscall table and returns with sysret. A negative return value becomes -1 with errno set.
3. Say how the kernel stays safe with user pointers: it never dereferences them directly. copy_from_user and copy_to_user check that the address range is valid and handle faults, which is why read() copies socket data into the caller's buffer rather than sharing it.
4. Quantify the cost honestly. The mode switch itself is cheap, but security mitigations and cache and TLB effects add up when a server makes millions of small calls. The fixes are fewer calls: batching (readv/writev, recvmmsg/sendmmsg), epoll instead of polling every descriptor, io_uring's shared rings, and the vDSO, which serves clock_gettime without entering the kernel at all.
5. Mention the tools: strace (or perf trace) shows which syscalls a process makes and how long they take, which is usually the first step when a server is mysteriously slow.

Follow-up

• Walk through what happens inside the kernel when a process calls read() on a socket that has no data yet.
• Why is clock_gettime() often faster than a normal system call?
• What does io_uring change about the user/kernel boundary compared with epoll plus read()?

How would you diagnose a production server that is dropping network packets under heavy traffic load?

Medium · Linux Systems & OS Fundamentals

Approach

1. Find where the packets are lost before changing anything: on the NIC, in the kernel, in the socket, or upstream. Compare counter deltas over a fixed interval rather than absolute totals, and confirm with a packet capture taken at both ends if the loss might be in the network.
2. Check the NIC and driver layer: ethtool -S for rx_missed/no_buffer/fifo counters (names vary by driver), ip -s link for RX dropped and overruns, ethtool -g for ring size, and ethtool -l for the number of queues. A full ring is fixed by growing it with ethtool -G or by adding queues.
3. Check the kernel and CPU layer: in /proc/net/softnet_stat, a growing second column means backlog drops (netdev_max_backlog) and a growing third column means the softirq ran out of budget (netdev_budget). mpstat -P ALL showing one core saturated in softirq points to poor interrupt distribution: inspect /proc/interrupts, then fix it with RSS, irqbalance or RPS.
4. Check the socket and netfilter layer: netstat -s or nstat for ListenOverflows/ListenDrops (accept queue full, so raise somaxconn and the backlog or accept faster), UDP RcvbufErrors (raise rmem_max and SO_RCVBUF, or read faster), and dmesg for 'nf_conntrack: table full', which means raising nf_conntrack_max or exempting the traffic from tracking.
5. Change one parameter at a time and re-measure the same counter. If the application cannot drain its sockets, kernel tuning only moves the drop point, and the real fix is in the consumer (more workers, batching, less work per packet).

Follow-up

• The NIC counters are clean but the client still sees retransmissions. Where do you look next?
• How would you tell drops caused by a slow application from drops caused by a softirq bottleneck?
• Which of these tuning changes cost memory or latency, and how would you roll them out safely across a fleet?

Given a stream of IP addresses, design a data structure that allows for fast lookup and CIDR block matching.

Medium · Coding, Data Structures & Databases

Approach

1. Clarify what has to be answered: exact-match membership of an address (a hash set gives O(1)), whether an address falls inside any stored block, or the most specific matching block (longest-prefix match, as a routing table needs). Also ask whether the blocks change while lookups run.
2. For longest-prefix match, use a binary trie keyed on address bits. Insert a /n block by walking its first n bits and storing the value at that node. On lookup, walk the address's bits and remember the last node that held a value. Both operations are O(W), where W is 32 for IPv4 and 128 for IPv6, regardless of how many blocks are stored.
3. Discuss the memory and speed trade-offs. Path compression (a Patricia or radix trie) collapses chains of single-child nodes. Multibit tries consume 8 or more bits per step, which means fewer memory accesses at the cost of more memory (DIR-24-8 resolves most IPv4 lookups in one or two table reads). Linux's IPv4 routing table uses an LC-trie. Another option is one hash table per prefix length, probed from longest to shortest: at most 33 probes for IPv4.
4. If only containment matters, convert each block to an integer range [network, broadcast], merge overlapping ranges, and binary-search the sorted list, giving O(log n) per lookup with very compact memory.
5. Cover the edge cases: normalise blocks whose host bits are set (10.1.2.3/8 becomes 10.0.0.0/8), handle /0 and /32, overlapping and nested blocks, and IPv4-mapped IPv6 addresses. For concurrent updates under a lookup-heavy stream, use RCU or copy-on-write snapshots so readers never take a lock.

Follow-up

• Estimate the memory for a million IPv4 prefixes in a plain binary trie versus a path-compressed one.
• How do you delete a block so that lookups still return the next-longest match?
• What changes when the same structure must hold IPv6 prefixes?

One customer endpoint stalls deliveries to every other destination

Medium · bulkhead · timeouts · saturation

The egress service delivers about 1.5k webhooks/second across 40,000 destinations, with a per-destination concurrency cap of 4 and a 10-second connect-plus-read timeout. Throughput falls to 300/second, queue depth climbs, and p99 delivery latency for unaffected destinations goes from 200 ms to minutes, while the error rate barely moves. One tenant holds 900 destination rows whose URLs share a hostname that now answers in 9.5 seconds. Explain the mechanism with the arithmetic, then give the containment in the order you would apply it.

Approach

1. Look at saturation before errors. A flat error rate with collapsing throughput says nothing is failing, things are waiting, so the first signal to pull is in-flight request count or pool wait time rather than the error counter. This is the distinction that decides the whole investigation.
2. Group in-flight work by resolved host, not by destination id. The cap is keyed per destination row, so 900 rows sharing one hostname buy 3,600 concurrent slots against a single host, each held for 9.5 seconds. The bulkhead was never a bulkhead for that host, and grouping by the wrong dimension is why the dashboard looked healthy.
3. Do the arithmetic in both directions. Required concurrency is arrival rate times latency, so 1.5k/second at 200 ms needs about 300 in flight, which is entirely consumed by 3,600 slow slots; conversely whatever concurrency is left sustains rate equals concurrency divided by 9.5 seconds, which is the 300/second you are seeing. Matching both numbers is what promotes this from a plausible story to the mechanism.
4. Explain why the circuit breaker never helped. It opens on consecutive failures, and a 9.5-second response inside a 10-second timeout is a success. Slow is not failing, so an error-rate breaker cannot see this; you need a slow-call ratio, a deadline propagated from the caller's remaining budget, or a concurrency limiter.

Follow-up

• The host recovers to 80 ms. How long does the queue take to drain, and what does the drain do to the recovered host?
• Where should the 10-second timeout number actually come from?

Suggested practice checklist

The plan follows the reported loop: protocol and socket mechanics for the phone interviews first, then the Linux path, coding and SQL, and last a miniature take-home so the build-and-defend stage feels familiar.

Day 1 — Map the loop and set a baseline

• Write a short account of your most relevant systems or networking projects and say which layer each one touched, for the recruiter conversation.
• List the reported topics: socket lifecycle, TCP states, flow and congestion control, system calls, kernel buffers, SQL on connection logs, data structures. Rate each from 1 to 5.
• Write three questions to ask the recruiter about the stages, the take-home and the language for your team.

Deliverable: A project account, a rated topic sheet with your two weakest topics marked, and a list of recruiter questions.

Day 2 — TCP and DNS mechanics

• Draw the three-way handshake and the four-way close with the state on each side (SYN_SENT, ESTABLISHED, FIN_WAIT, TIME_WAIT), and explain why TIME_WAIT exists.
• Explain the control (receiver) window versus the congestion window, covering slow start, congestion avoidance and what packet loss does to throughput.
• Walk through DNS resolution from stub resolver to authoritative server, then list ways to cut lookup latency for a latency-sensitive client.

Deliverable: A hand-drawn TCP state diagram plus recorded spoken answers to the TCP establishment and termination, window comparison and DNS questions.

Day 3 — Sockets, event loops and two reported scenarios

• Write a blocking TCP echo server in C or C++ using socket, bind, listen and accept, and note what each call does in the kernel.
• Rewrite it with non-blocking sockets and epoll, with per-connection output buffers and EPOLLOUT used only when data is pending.
• Test with many simultaneous clients and one deliberately slow reader, and record what the server does.
• Sketch two reported scenarios: a raw-socket client-server with manual retransmission (sequence numbers, ACK timers, retransmission timeout, CAP_NET_RAW), and a VPN user moving from Wi-Fi to cellular (session ID instead of the 4-tuple, as in WireGuard roaming or QUIC migration, no TCP-over-TCP restart, DNS re-resolved through the tunnel).

Deliverable: Working blocking and epoll versions of the echo server, a half-page note comparing select, poll and epoll, and one-page outlines for the raw-socket and Wi-Fi to cellular scenarios.

Day 4 — Linux internals and packet-drop diagnosis

• Trace a received packet from the NIC through the driver and kernel network stack to the socket buffer and your process's read() call.
• Explain user space versus kernel space, how a system call crosses the boundary, and how virtual memory supports process isolation.
• Write a runbook for a server dropping packets under load, with the counters and tools you would check at each layer, and a second one for a file descriptor leak in a daemon.

Deliverable: A one-page packet-drop runbook, a file descriptor leak checklist, and recorded answers on the kernel packet path, system calls and virtual memory.

Day 5 — Coding: queue, CIDR lookup, LRU, linked list

• Implement a bounded thread-safe packet queue in C or C++ with a mutex and condition variables, including a shutdown path, and test it with several producers and consumers.
• Design and code a structure for IP lookup with CIDR matching, such as a binary trie with longest-prefix match, and state its time and memory cost.
• Implement a cache that evicts the least recently used IP addresses, then reverse a linked list iteratively and recursively and state the space and time complexity of each.

Deliverable: Compiling code with small tests for the queue, the CIDR structure, the LRU cache and the list reversal, each with its complexity written beside it.

Day 6 — SQL on connection logs and the take-home build

• Write the query for duplicate active user sessions, and a second for daily active users and average session duration, using a composite index on (user_id, started_at) for the second.
• With two transactions inserting and closing one user's sessions, name each level's anomaly: read uncommitted (dirty read), read committed (non-repeatable read), repeatable read (phantom; write skew under snapshot isolation), serializable (none).
• Start a miniature take-home: a small TCP or UDP utility, such as a port forwarder or echo service with retry logic, split into modules with logging and error handling.

Deliverable: A SQL file with both queries, an index note and the isolation-level examples, plus a runnable first version of the utility with its module layout.

Day 7 — Finish the take-home and rehearse the conversation

• Add loopback tests for malformed input, a dropped peer and a slow reader, then write the README with build and run commands and a design-decisions section.
• Prepare to defend each data structure choice in the utility, covering memory use and complexity under many concurrent connections.
• Rehearse the behavioral prompts, including narrating a production incident you handled, and run a full spoken pass over the questions you rated weakest on day 1.

Deliverable: A finished utility with tests and README, a written defence of its data structures, and a list of rehearsed stories.

Stories and explanations to rehearse

No separate behavioral round is reported. The people-facing parts are the recruiter conversation about your background and interest in privacy software, and the presentation of your take-home. Candidates also describe being asked about technologies listed on their resume, so be accurate about your depth, and explain your reasoning when you are unsure of a specific setting. The requirements candidates describe name clear communication of trade-offs and collaboration, so prepare stories that show both with real technical detail.

1. Tell me about a production networking or performance problem you debugged: the symptom, what you measured, the cause you proved and the fix you shipped.
2. Why do you want to work on privacy and security software, and which of your projects shows that interest?
3. Describe a time you had to explain a technical trade-off, such as UDP versus TCP, to someone who disagreed with you.
4. Tell me about a time you said you did not know something in a technical discussion, and what you did next.
5. Walk through a low-level project where you owned the code, tests and documentation from start to finish.

How do you handle non-blocking I/O and multiplexing using system calls like select or epoll?

Medium · Network Protocols & Socket Programming

Approach

1. This is a technical question about event-driven I/O, so answer the mechanism first, and add a short story from a server or tool you built if you are asked about your experience.
2. Start with non-blocking mode. Set O_NONBLOCK with fcntl, or pass SOCK_NONBLOCK to socket or accept4. A read or write that cannot proceed then returns -1 with EAGAIN or EWOULDBLOCK instead of sleeping, and a non-blocking connect returns EINPROGRESS: wait for writability, then read SO_ERROR to learn whether it succeeded.
3. Compare the multiplexers on cost. select takes fd_set bitmasks, is capped by FD_SETSIZE (1024), and the sets are rebuilt and scanned on every call. poll takes an array of pollfd with no fixed cap but still walks every descriptor each call. epoll registers interest once with epoll_ctl (epoll_create1, then add, modify, delete), and epoll_wait returns only the ready descriptors, so its cost follows the number of events, not the number of connections.
4. Cover level-triggered and edge-triggered modes. Level-triggered reports a descriptor for as long as data remains. Edge-triggered reports a change only once, so you must use non-blocking descriptors and read or accept in a loop until EAGAIN, or the connection stalls with unread data. Mention EPOLLONESHOT when several threads wait on the same epoll instance.
5. Describe the real event-loop details: keep a per-connection output buffer and register EPOLLOUT only while it holds unsent bytes; accumulate partial reads until a full message is framed, because TCP is a byte stream; retry on EINTR; on a read returning 0, deregister and close the descriptor.

Follow-up

• Why does epoll scale better than select with tens of thousands of mostly idle connections, and when is the difference small?
• What goes wrong with an edge-triggered descriptor if you stop reading before EAGAIN?
• How would you use all CPU cores: one epoll instance per thread, SO_REUSEPORT listeners, or a single acceptor handing off descriptors?

Unblock an engineer without taking the keyboard

Easy · mentoring · leases · at-least-once

A teammate has spent two days on a job handler that occasionally writes duplicate rows. They are certain the queue is delivering twice by mistake. You suspect a lease expiring under a slow handler, so the job is running concurrently with itself. Describe how you have unblocked someone in this position: what you asked before offering a hypothesis, what you showed them rather than told them, and what you left them owning. Then say what you would do if their theory turned out to be the right one.

Approach

1. Ask before diagnosing, and ask for things answerable from data they already have: the attempt count on the job rows that produced duplicates, the handler's observed duration against its lease expiry, and whether the duplicate rows share a natural key that a unique constraint could have caught.
2. Teach the shape rather than the answer. A lease cannot distinguish a dead worker from a slow one, so a handler that outruns its lease is running twice by design, and deploys deliver the other half by killing handlers mid-run on every rollout. Both of their candidate theories produce identical duplicate rows, which is why the evidence has to come from timings rather than from argument.
3. Hand over a checklist they execute: a natural key on every write the handler performs so the second copy collides rather than appends, the record of intent written before any external effect, a lease heartbeat while running, and the metric that shows it working.
4. Keep ownership with them deliberately. Pair on the first write, then step back; if you finish it yourself you have closed one ticket and left the same person stuck on the next redelivery.

Follow-up

• How would you distinguish a genuine double-delivery from a lease expiry using only the data already stored?
• Their handler calls an external endpoint before recording that it did. What do you tell them to change first?

Narrate an outage you owned from page to postmortem

Hard · incident response · blast radius · postmortems

Pick an incident you personally drove, ideally one where writes were affected rather than reads. In six to eight minutes: state the symptom as it first appeared on a dashboard, the blast radius you established before you knew the cause, the mitigation you applied and when, the mechanism you eventually proved, and the follow-up that would prevent a repeat. Bring numbers: error rate, tenants affected, minutes to mitigate, minutes to resolve. If you cannot name what you measured, choose a different incident.

Approach

1. Open on the signal rather than the cause: which metric at which percentile moved, on which service, at what time, so the listener follows the same evidence you had rather than a conclusion you already reached.
2. Separate mitigation from diagnosis out loud. State what you did to stop the bleeding (flag off, shed traffic, drain a lease, roll back a deploy) and say plainly that you did it before the mechanism was known, because those are two jobs with different deadlines.
3. Establish blast radius in countable terms: how many tenants, how many writes, and crucially whether the effect was loss or only delay. An append-only revision table or a pending outbox row means the change survived and the projection was merely behind, which is a repair rather than a data-loss incident.
4. Prove the mechanism instead of asserting it. Name the trace span that grew, the plan that flipped to a sequential scan, the lease that expired, plus one alternative you ruled out and the signal that stayed flat while you ruled it out.

Follow-up

• What would you do differently in the first five minutes, given the same dashboard and no more information?
• Which follow-up action did you deliberately not take, and why was dropping it the right call?