cswork / libraryOpen practice workspace

Articles

Java NIO Interview Questions: ByteBuffer State, Partial Reads, and Selector Loops

Practice Java NIO interview questions with ByteBuffer state traces, fragmented frames, partial writes, EOF handling, and selector-loop failure diagnosis.

Java NIO interview questions become easier when you track three things separately: the buffer's readable range, the connection's incomplete message, and the selector's interest in future work. A readable socket does not promise a complete application message, and one write does not promise that every queued byte was sent.

Official API behavior: flip() exposes the bytes just accumulated; compact() preserves unread bytes for another receive; nonblocking SocketChannel.read() can return zero, while -1 means end of stream. These distinctions follow Oracle's Buffer and SocketChannel contracts. The examples below are original practice exercises, not a claim about any employer's next interview.

What should your first answer establish?

NIO is Java's family of buffer, channel, and related I/O APIs. A ByteBuffer holds bytes with a cursor and a boundary; a channel transfers them; a selector reports readiness across registered selectable channels. Nonblocking operation lets an event loop retain unfinished work and return later.

Begin by defining the protocol. Is a message newline-delimited or length-prefixed? What byte order and maximum size apply? Is the connection already established? Can one read contain several messages? Those answers determine the parser and connection state before concurrency enters the discussion.

For this exercise, assume a connected nonblocking TCP channel, a four-byte big-endian signed integer header, and payloads of zero through four bytes. The receive buffer has capacity eight. This deliberately small contract makes fragmentation visible; it is not a general production frame-size recommendation.

Can you trace position, limit, and capacity?

The usable range is from position up to, but excluding, limit. Capacity is the storage boundary. Draw those indexes before describing a method call; naming a buffer “read mode” does not establish its actual state.

The following original ledger receives a header declaring three payload bytes, followed by ABC. First, only two header bytes arrive. Then the remaining header and A arrive. Finally, BC arrives.

OperationPositionLimitWhat the state means
Allocate eight bytes08Space for incoming bytes
Receive first two header bytes28Header is incomplete
flip()02Two bytes are available to inspect
No complete header; compact()28Preserve those two bytes
Receive remaining header plus A58Five accumulated bytes
flip(); peek header05Need seven bytes for the complete frame
Incomplete frame; compact()58Preserve header and A
Receive BC; flip()07Whole frame is readable
Consume header and payload; compact()08Ready for another frame

Explain why the header peek leaves position unchanged. Reading the header with relative getInt() and forgetting to restore position would discard the framing information when compacting an incomplete payload. An explicit saved position or absolute peek avoids that bug.

Why is compact different from clear or rewind?

Official behavior: compact() moves the unread range to the beginning, sets position to the number of preserved bytes, and restores limit to capacity. clear() resets boundaries without erasing storage; rewind() moves position to zero while keeping limit. See the ByteBuffer compact contract.

In the ledger, clearing after the five-byte fragment would allow the next read to overwrite the unfinished header and payload. Rewinding would expose already consumed bytes again. Calling flip() twice would make the second limit equal to the current position, which may hide every byte.

For an outbound buffer, retain its advanced position after a partial write. Rewinding before the next write would resend bytes. Compaction is useful when you intend to append more bytes to the same storage; a queue of already prepared outbound buffers can instead keep each buffer in its current readable state.

How do you parse a fragmented length-prefixed frame?

This Java sketch parses accumulated bytes after flip(). It validates the length before waiting for the payload, uses an absolute header read, and only advances past a frame when the entire frame is present.

static void parseAvailable(ByteBuffer in,
                           Consumer<byte[]> deliver) {
    while (in.remaining() >= Integer.BYTES) {
        int start = in.position();
        int length = in.getInt(start); // absolute peek
        if (length < 0 || length > 4) {
            throw new IllegalArgumentException("bad frame length");
        }
        if (in.remaining() < Integer.BYTES + length) {
            break;
        }
        in.position(start + Integer.BYTES);
        byte[] payload = new byte[length];
        in.get(payload);
        deliver.accept(payload);
    }
}

The snippet needs java.nio.ByteBuffer and java.util.function.Consumer. Keep the buffer big-endian for this protocol. The caller receives into a writable buffer, flips it, calls the parser, then compacts it after successful parsing. A malformed frame closes or rejects the connection under an explicit policy.

The callback must not perform slow work on the event-loop thread. In this sketch, a callback exception also aborts the connection; recovery from application-processing failures needs a separate delivery contract. This is reviewed interview code, not a complete or executed Java server.

Two concatenated frames require another loop iteration. A zero-length payload is a complete four-byte frame. A header advertising a size beyond the configured maximum is invalid immediately, even if its body never arrives. These cases establish progress and prevent unbounded accumulation.

What does a zero-byte read mean?

Keep three outcomes distinct. A positive count adds bytes to the parser's state. Zero means this call made no transfer; on a nonblocking channel, return to the event loop rather than spinning until a message arrives. Minus one means the inbound stream ended.

A zero result can also occur because the destination has no remaining space. Inspect the buffer before diagnosing the network. If an accepted frame cannot fit your fixed-capacity design, repeatedly reading into a full buffer cannot repair it; reject the size or implement a bounded growth strategy.

At EOF, first account for complete frames already received. If a partial header or payload remains, report truncation under this protocol. If none remains, inbound completion can be clean. Whether to drain pending responses before closing the whole connection is an application decision, especially when half-close behavior matters.

Do not transfer return conventions between APIs. The linked socket-reader exercise describes Python recv() returning empty bytes at EOF. Java SocketChannel.read() uses -1; its zero return is not that EOF signal.

Why can one write leave bytes unsent?

Use a second original trace. An outbound buffer contains seven bytes, with position zero and limit seven. Three attempts write two, zero, and five bytes. Its successive positions are two, two, and seven; its limit remains seven.

After the zero write, keep the same buffer at the queue head and enable write interest. When another writable event arrives, resume from position two. Remove the buffer only when it has no remaining bytes, then continue with the next queued buffer under a fairness budget.

Practice inference: keeping OP_WRITE enabled on an empty output queue can make an otherwise idle loop repeatedly wake for writable sockets. Use write interest while pending output needs another opportunity, and remove it once the queue is drained. This policy follows from the work your application actually retains.

How do you repair a selector loop that spins or stalls?

Official selector behavior: selected keys remain available until the application removes them; selection can update existing readiness information. A key's interest set describes operations to monitor, while its ready set describes observed readiness. Oracle documents those sets and wakeup() in the Selector API.

Iterate the selected-key set and remove each processed entry through the iterator. Check validity and handle cancellation. A single key can represent more than one ready operation, so avoid assuming that an else if chain always reflects the work to perform.

For a stall, inspect interest changes, queue ownership, and whether the loop is sleeping. If another thread enqueues a response, publish that change through a thread-safe handoff and wake the selector when needed. A simple design makes the event-loop thread the owner of connection buffers and applies registration or interest changes there.

Waking the selector does not make a queue thread-safe or complete a socket write. Likewise, cancelling a key is not the same as closing its channel. Give each operation a specific lifecycle responsibility instead of treating every cleanup call as interchangeable.

Which failure evidence should you collect?

Record connection identity, read or write count, buffer position and limit, bytes retained after parsing, output queue size, and interest transitions. For a high-CPU report, distinguish repeated zero transfers from repeated selection with no useful work. For a stall, establish whether pending output exists and whether write interest was enabled.

Create deterministic byte chunks rather than hoping localhost fragments a frame in a particular way. Exercise splits inside the header, inside the payload, and exactly between two frames. Add empty payloads, multiple frames in one chunk, negative lengths, oversized lengths, and EOF at each incomplete stage.

The parser should deliver the same ordered payloads for every chunking of a valid byte stream. A scripted writer that returns 2, 0, 5 should produce each outbound byte once. Then separately test actual sockets, scheduling, disconnects, and resource cleanup; parser tests alone do not prove the event loop correct.

How should you explain the design under follow-up pressure?

Start with the smallest invariant: every byte belongs either to an incomplete inbound frame, a delivered payload, or pending outbound work. Then show exactly which operation moves it between those states. If an interviewer changes the maximum payload or introduces concurrent workers, revise the affected boundary instead of rewriting everything.

For larger frames, explain how you bound memory per connection and across all connections. For fairness, limit work per ready key so one busy peer cannot monopolize the loop. For shutdown, decide what happens to incomplete input, queued output, worker results arriving late, and the channel itself.

One useful follow-up is: can this eight-byte receive buffer handle a valid stream longer than eight bytes? Yes, if the caller alternates receiving and parsing: completed frames release space, and each accepted individual frame fits the configured capacity. It cannot accept an arbitrary oversized frame simply because the overall connection is a stream. The per-frame bound remains essential.

Another follow-up is whether a thread-safe map makes the connection state safe. It protects operations on that map, not every mutable buffer stored inside it. If a worker reads or advances the same buffer while the selector thread compacts it, the parser invariant is broken. Copy the completed payload or establish exclusive ownership before transferring work, and keep the handoff bounded so slow consumers cannot accumulate unlimited responses.

Rehearse by solving the socket message reader with a state ledger and adversarial chunks. A convincing answer names the retained state, proves forward progress, and gives a concrete test for every boundary it relies on.

Sources and Further Reading