Articles
LSTM Interview Questions: Gates, State Shapes, and Variable-Length Sequences
Practice LSTM interviews with gate calculations, hidden and cell shapes, projections, padding counterexamples, packed sequences, and final bidirectional states.
An LSTM answer can contain the correct gate equations and still select the wrong sequence representation. The interview often becomes concrete when someone adds a second layer, turns on bidirectionality, or batches sequences of different lengths. At that point, the useful skill is tracing what each tensor represents, not memorizing that LSTMs “remember the past.”
Library behavior is identified explicitly; the worked examples and testing recommendations are independent preparation, not a claim about an employer’s exact interview.
What do the gates actually compute?
Official cell contract: a standard LSTM computes input, forget, and output gates using sigmoid activations, and a candidate update using tanh. The cell update combines retained old state with a gated candidate; the hidden state applies the output gate to the transformed new cell. PyTorch’s LSTMCell exposes one recurrence step. PyTorch LSTMCell
A compact mathematical description is:
i = sigmoid(input and previous-hidden affine terms)
f = sigmoid(input and previous-hidden affine terms)
g = tanh(input and previous-hidden affine terms)
o = sigmoid(input and previous-hidden affine terms)
c_new = f * c_old + i * g
h_new = o * tanh(c_new)
Here * means elementwise multiplication. The candidate g can be negative; it is not another sigmoid gate. The cell and hidden state serve different roles even when their dimensions coincide.
Explain the operation before giving a metaphor. The forget gate scales the old cell contribution, the input gate scales new content, and the output gate controls the exposed hidden representation. Calling the forget gate a binary switch hides the fact that its values are generally continuous. Calling h_new the cell memory confuses two distinct tensors.
Trace one cell with actual numbers
Our original scalar fixture chooses gate values i=f=o=0.5, candidate g=0.8, and previous cell c_old=0.4. One update gives:
retained old contribution = 0.5 * 0.4 = 0.2
new candidate contribution = 0.5 * 0.8 = 0.4
c_new = 0.6
h_new = 0.5 * tanh(0.6) ≈ 0.268525
A common mistake is reporting hidden state 0.3 by multiplying the output gate directly by the cell. Another is applying tanh before combining the two cell contributions. Both change the recurrence while preserving a superficially plausible shape.
This fixture is realizable: set input and recurrent weights to zero, sigmoid-gate biases to zero, and the total candidate bias to atanh(0.8). The nonzero candidate is then bias-driven. It is a deliberately simple diagnostic model, not a useful trained predictor.
Change one value at a time. If the forget gate approaches zero, the old cell contributes little; if the input gate approaches zero, new candidate content contributes little. Even when the old cell is retained, the output gate can suppress the visible hidden state. A classifier reading hidden state does not directly observe the entire cell state.
The scalar arithmetic was checked independently in NumPy. That checks the recurrence and the numerical counterexample below; it does not establish that a particular GPU kernel or PyTorch training job was executed.
Which tensor shapes belong to which objects?
Official API contract: with batched inputs, batch_first=True changes input/output layout but not hidden/cell layout. States use a leading layer-times-direction axis. Output contains last-layer hidden features at each time step. Projection changes hidden/output width while cell width remains hidden_size. PyTorch LSTM
For an original shape exercise, choose batch N=3, padded time T=5, input width I=4, cell width H=6, two layers, and two directions. Let hidden-output width be K: it is six without projection, or three when proj_size=3.
| Tensor or configuration | No projection | Projection width 3 |
|---|---|---|
| Batch-first input | [3, 5, 4] | [3, 5, 4] |
| Last-layer output across time | [3, 5, 12] | [3, 5, 6] |
| Initial or final hidden states | [4, 3, 6] | [4, 3, 3] |
| Initial or final cell states | [4, 3, 6] | [4, 3, 6] |
| Concatenated final top-layer directions | [3, 12] | [3, 6] |
Derive the leading state dimension: two layers multiplied by two directions produces four state slices. Derive output width: two directions multiplied by K. Time belongs to output, while the returned final states have no time axis.
Make the batch and layer-direction dimensions unequal in tests. A mistaken transpose can survive a fixture where both happen to equal four. A tensor with the expected total element count can still assign the wrong sequence’s state to a prediction.
For projection, do not initialize hidden and cell state with the same last dimension by habit. The projection idea separates recurrent representation width from cell capacity; the original projected-LSTM paper provides architectural context, rather than a guarantee of speed on your hardware. Sak, Senior, and Beaufays
Why are padded zeros not an instruction to stop?
The scalar fixture makes this visible without a training dataset. Because the candidate comes from bias, a zero input still advances the recurrence:
| Executed recurrence steps | Cell value | Interpretation in this fixture |
|---|---|---|
| Before any step | 0.4 | Supplied initial state |
| After step 1 | 0.6 | Correct final cell for a length-one sequence |
| After step 2 | 0.7 | Extra update if a padded step is processed |
| After step 3 | 0.75 | Final cell after three executed steps |
Batch two sequences with true lengths [3, 1], padding the second to three time steps. Processing every position produces cell 0.75 for both in this intentionally input-independent model. Stopping at each sequence’s valid boundary produces [0.75, 0.6]. The length-one hidden state should be approximately 0.268525, rather than the three-step value 0.317574.
The error is semantic, not a floating-point tolerance issue. Zeros are data values supplied to a recurrence. Without length-aware processing, the recurrence can still use biases, previous hidden state, and the previous cell. A loss mask applied after the recurrence does not rewind those extra state updates.
For a custom cell loop, one option is to compute a candidate update and retain the previous hidden and cell states for inactive batch elements. Apply the length mask to both states. If you update cell while freezing only hidden, the next active continuation can expose the corrupted memory.
For a bidirectional dense padded model, gathering the forward output at the last valid position does not generally repair the reverse direction: it may already have processed padding before reaching valid data. This is a reason to specify length-aware recurrence, not simply choose a different slice afterward.
How does packing use the true sequence lengths?
Official packing contract: pack_padded_sequence takes padded input plus lengths. With unsorted batches, enforce_sorted=False permits internal sorting. Length tensors must be on the CPU; lengths describe the valid portion and can truncate supplied input. Packing documentation
An illustrative PyTorch preparation pattern is:
packed = pack_padded_sequence(
x, lengths_cpu, batch_first=True, enforce_sorted=False
)
packed_out, (h_n, c_n) = model(packed)
This snippet is source-reviewed against the documented API; the local numerical checks use NumPy because PyTorch is not installed in the working environment. Verify it in your own pinned runtime before treating it as an executed library comparison.
The important inputs are the lengths, not just the padded tensor. An off-by-one length silently changes which observation belongs to the sequence. Validate that lengths match the actual sequence construction and that each training target is aligned with the intended valid position.
Official unpacking contract: pad_packed_sequence restores a padded tensor and lengths, reordering batch elements to the original packing input order. Its total_length argument can request a common padded extent. Unpacking documentation
If you sort batches manually, keep the targets, identifiers, initial states, and inverse permutation consistent. A valid shape does not prove that predictions still match their original examples. Test with distinct sequence identifiers and deliberately unsorted lengths rather than only a descending-length batch.
Why is the last output different from the final bidirectional state?
Official implementation note: PyTorch warns that a bidirectional h_n is not equivalent to the last output element. They place the reverse-direction summary at different sequence boundaries. PyTorch recurrent-module source
Reason through a three-token sequence a, b, c. The forward recurrence reaches its final state after c. The reverse recurrence processes c, b, a and reaches its final state after a. At the original time position of c, the forward component has consumed the whole sequence, but the reverse component has only begun its traversal. It is not the reverse summary after all three tokens.
For sequence classification using final directions, make the layout explicit:
h_n: [layers * directions, batch, K]
reshape to [layers, directions, batch, K]
choose the last layer
concatenate direction 0 and direction 1 along feature width
result: [batch, 2 * K]
This is an original indexing recipe based on the stated contract. h_n[-1] alone selects one slice, not automatically both directions or all layers. Verify the layer/direction interpretation before concatenating.
For token-level prediction, time-aligned outputs may be exactly what you need. The right representation follows the target. A whole-sequence bidirectional summary is unsuitable for a causal prediction made before future observations exist; clarify the information available at prediction time.
Which tests reveal silent sequence bugs?
Use independent controls with unequal dimensions, distinct sequence lengths, and distinguishable examples. A practical test set includes:
- The scalar cell trace, checking both
c=0.6andh≈0.268525. - A length-one sequence beside a length-three sequence, comparing valid final states with separate execution.
- A batch permutation, checking that outputs remain paired with the same identifiers.
- A two-layer bidirectional shape check, including both final directions.
- A projection case, checking that cell and hidden widths differ.
The biased-cell counterexample is especially valuable because an all-zero model can make an incorrect padding implementation appear correct. Choose controls that make the unwanted update observable.
For an actual library comparison, align weights, biases, dtype, initial states, valid lengths, mode, and direction convention. Compare values within justified numerical tolerance, not only shapes. A difference should be localized to the first step and state where it appears. Library acceleration and end-to-end latency require a separate benchmark; parameter count alone cannot establish them.
Connect the recurrence to the modeling decision
A strong answer moves from the target to the valid sequence boundary, then to the state representation and tests. Explain the gates numerically, state the tensor layout, and show why padding and direction change the result you feed downstream.
If you can identify the correct final state and the information available at prediction time, your architecture comparison has an implementation behind it.