Articles
NUMA Interview Questions: First-Touch Allocation and Remote-Memory Costs
Practice NUMA interview questions with first-write page ledgers, CPU versus memory binding, remote-access experiments, and measured placement decisions.
A server gains more worker threads but becomes slower after moving onto a machine with two NUMA nodes. The team pins half the workers to each node, yet performance barely changes. A useful NUMA interview answer asks who initialized the data before those workers started. CPU placement and physical page placement are separate decisions.
The useful starting point is to trace where pages live and where their users run before diagnosing remote-memory costs. A memory address remains accessible across a cache-coherent NUMA system; accessibility does not imply equal access cost from every processor.
Evidence boundary: Linked Linux kernel, Linux manual, and numactl project documentation establish official behavior. The page ledger and experiment designs are original exercises and engineering inference. They are not candidate reports, employer-frequency claims, or benchmark results from this host. Practice the diagnostic framing with Diagnose CPU Performance Problems on a Linux Server.
What makes memory access nonuniform?
Official fact: Linux represents NUMA resources as nodes. Access latency and effective bandwidth can vary with the relationship between the requesting CPU and the memory's location. Default local allocation can fall back when local resources cannot satisfy a request. See What is NUMA?.
Do not equate “two sockets” with exactly two NUMA nodes without inspecting the machine. Nor should an answer assume every node contains both CPUs and memory. Topology, firmware configuration, and heterogeneous memory can make the picture more complicated than one local DRAM bank per socket.
A local-versus-remote comparison needs a requester and a target. “This memory is remote” is incomplete until the answer names the CPU or device initiating the access. A worker can move and change that relationship even when the page stays in the same location.
Official fact: The kernel exposes NUMA performance relationships and, where the platform provides them, rated latency and bandwidth attributes. These are platform characteristics, not measured application response times. See NUMA memory performance.
Design inference: Use topology information to design an experiment, then measure the actual workload. A distance value or rated bandwidth does not incorporate your queueing, cache-hit rate, access pattern, or contention. It is evidence about the machine, not a complete performance model of the service.
When does first touch place a page?
The phrase “first touch” is shorthand that needs assumptions. For this article's exercise, assume fresh anonymous memory, ordinary demand allocation, default local policy, available local memory, and first writes by pinned workers. The allocating faults tend to place pages near the CPUs executing those faults.
Calling malloc() is not proof that a new physical page was allocated immediately on the caller's node. The allocator might reuse memory, reserve virtual address space, or initialize some pages itself. Reading untouched anonymous memory is also not a reliable way to create privately writable physical pages; use explicit writes when designing the exercise.
Official fact: Linux memory policy governs allocation with task and VMA scopes. Existing pages do not automatically move when a new policy is installed; migration is a separate operation. Task policies in a multithreaded process also have inheritance and installation scope that matter. See NUMA memory policy.
In an interview, say which operation establishes backing pages. “The main thread allocated the array” is weaker than “the main thread wrote every page while pinned to node zero, before the workers started.” The latter identifies the proposed cause and makes it testable.
If the array came from a reused pool, the original first writer might belong to an earlier request. If a file mapping is involved, its page-cache and copy-on-write behavior differs from the fresh anonymous allocation assumed here. Preserve these distinctions instead of extending one demonstration to all memory mappings.
Work through a two-node ownership ledger
Original exercise: Four workers process disjoint quarters of an array with 100 equal-sized pages. Workers A and B run on node zero; C and D run on node one. Each worker repeatedly accesses only its own quarter. Ignore migration, fallback, shared cache hits, and allocator reuse for this ledger.
| Initialization | Pages on node zero | Pages on node one | Worker C's 25 pages |
|---|---|---|---|
| Main thread on node zero writes all pages | 100 | 0 | Remote relative to C |
| Each pinned worker first-writes its own quarter | 50 | 50 | Local relative to C |
| Page-granular equal interleave across both nodes | 50 | 50 | Roughly half local under the stated layout |
Serial initialization creates a placement imbalance even though computation is evenly split across nodes. Parallel initialization aligned with ownership changes the ledger. Equal interleave balances capacity use but does not guarantee locality for each worker's partition.
The ledger predicts a remote-page proportion; it does not predict wall-clock speed. In the serial case, 50 of the 100 worker-owned pages are remote relative to their assigned workers. In the parallel case, that count becomes zero under the exercise's assumptions. It does not follow that execution time halves: caches, memory-level parallelism, and bandwidth saturation still matter.
Now change the workload so every worker scans the whole array. No single placement makes all pages local to both nodes. Partitioning, replication of read-only data, or a bandwidth-oriented interleave policy becomes more relevant than assigning a permanent owner to each quarter.
A precise answer updates its assumptions when the access pattern changes. It does not keep recommending parallel first touch after the workload has stopped being partitioned.
Does pinning threads move their memory?
Official fact: numactl distinguishes CPU binding from memory policy options. CPU-node binding chooses where execution is allowed; memory binding, preferred placement, and interleave address allocation behavior. See the numactl project's manual.
Original exercise: The main thread initializes all pages on node zero. You then pin worker C to node one. The CPU placement changes; the existing pages in this simplified trace remain on node zero. Pinning is not a retroactive initialization procedure.
Changing memory policy after the pages exist does not by itself establish that they moved. If migration is requested, verify its outcome rather than treating the API call as a complete observation. Moving pages costs work and can interact with application activity; moving a thread can also trade locality against CPU load balance.
Official fact: Administrative cpuset restrictions take priority over the application's requested memory policy. The available nodes and policy therefore need to be checked together. See memory policies and cpusets.
A container can see a broad machine topology while being allowed only a subset of CPUs or memory nodes. Explain the effective allowed set before suggesting a binding command. An apparently ineffective policy can be an environment mismatch rather than evidence that NUMA costs never matter.
What measurements distinguish NUMA from another bottleneck?
Original experimental matrix: Keep dataset, algorithm, worker count, payload, compiler settings, and request mix fixed. Use a fresh allocation for each placement treatment and record initialization separately from steady-state work.
| Treatment | Change | Evidence to collect |
|---|---|---|
| Local partition ownership | Workers initialize and process their own partitions | Actual page distribution and per-worker execution placement |
| Deliberately remote partitions | Same work; users run on the other node | Latency distribution and memory traffic change |
| Serial initialization | One pinned initializer faults pages | Whether the page ledger becomes concentrated |
| Interleaved allocation | Spread pages; retain worker placement | Bandwidth balance, local/remote traffic, and response-time effects |
| Shared writes | Retain placement; introduce shared hot cache lines | Whether coherence contention dominates the placement effect |
Official fact: /proc/<pid>/numa_maps reports memory-policy and page-location information for a process's mappings. It is useful for checking placement rather than inferring it from the source code. See proc_pid_numa_maps.
Capture placement after initialization and again after the measured phase. A policy or automatic migration mechanism can change the layout during a long run. The measurement should state whether it studies initial placement, convergence, or stable performance.
Use wall time and throughput alongside tail latency where the workload serves requests. Record warmup, page faults, CPU utilization, and node pressure. A comparison that includes page initialization in one treatment but excludes it in another cannot isolate steady-state remote-memory cost.
A pointer-chasing experiment emphasizes dependent-load latency; a streaming scan can emphasize bandwidth and parallel outstanding requests. Neither stands in automatically for the other. Choose the access pattern that tests the hypothesis rather than the one that produces the largest-looking difference.
Why can remote memory matter less than expected?
Engineering inference: A workload whose hot working set fits in cache may rarely fetch from the page's backing memory. Remote DRAM location remains real, but many accesses can be satisfied without visiting it. This explains why a placement change might produce little effect without disproving the topology.
Another workload may saturate one memory node's bandwidth even when every worker runs nearby. Spreading work and pages can improve throughput by using more controllers, despite adding some remote accesses. Optimize the measured objective: the fewest remote references and the highest service throughput are not always identical goals.
False sharing is a separate mechanism. Workers updating different variables in the same cache line can create coherence traffic. Putting all pages locally does not remove that sharing pattern. Conversely, separating hot counters onto different cache lines does not migrate a large read-only array to the worker's node.
To distinguish them, first retain disjoint page ownership and vary placement. Then retain placement and vary shared writes or padding. Inspect whether the result tracks the page relationship, the coherence pattern, or both. This is more defensible than labeling every multi-socket slowdown “NUMA.”
Choose a remedy that matches ownership
For stable partitioned work, align initialization and processing with the same ownership partition. Keep the partition boundaries at useful allocation granularity; two workers touching different elements of one page can still share its placement fate.
For widely shared read-only data, replication may improve locality when memory capacity and refresh complexity permit. Define how a new version is published consistently. For shared mutable data, replication also needs a coherence or application-level consistency design; it is not merely another allocation flag.
Interleave can be reasonable when access is broadly shared or distributing bandwidth matters more than keeping each worker's slice local. Strict binding can create pressure or allocation failures if the allowed nodes lack capacity. A preferred policy allows a different fallback contract. Describe the trade-off before choosing one.
If ownership shifts repeatedly, static binding may become stale. Consider whether the scheduler, automatic balancing, an application migration strategy, or redesigning the data flow better fits the workload. Do not promise automatic balancing as a universal cure; measure both its resulting placement and its overhead.
The ledger calculations for this article are executable arithmetic models, not hardware benchmarks. No native multi-node performance results were produced on this host. A real result should retain the topology, kernel and policy settings, actual page evidence, and repeated workload measurements with variability.
Five related questions to practice
The exercises below connect memory placement to operating-system diagnosis and performance modeling. Their inclusion does not establish NUMA-specific employer interview frequency.
| Question | NUMA reasoning to add |
|---|---|
| Explain Low-Latency Linux Networking and Shared-Memory Trade-Offs | State producer, consumer, and page-placement relationships. |
| Row-Wise vs Column-Wise Array Reads, L1/L2 Caches, and the TLB | Separate access pattern and caching from physical node placement. |
| Diagnose CPU Performance Problems on a Linux Server | Establish a controlled placement hypothesis. |
| Diagnose I/O, Memory, and CPU Constraints | Avoid treating every slowdown as remote memory. |
| Performance-Model a Sharded Matrix Multiply: FLOPs, Memory, Network and Roofline | Compare computation, memory traffic, and transfer limits. |
Continue with Row-Wise vs Column-Wise Array Reads, L1/L2 Caches, and the TLB. Explain its cache behavior first, then add the two-node page ledger. Keeping those explanations separate makes a proposed NUMA experiment easier to assess.