cswork / libraryOpen practice workspace

Interview guides · Hajoca

Hajoca Software Engineer Interview Questions & Guide 2026

A prep plan for Hajoca software engineer candidates: answer first, read the solution second, repeat the misses later.

The 60-second brief

As a Software Engineer at Hajoca, you are joining a company that balances the stability of a leading wholesale distributor with the innovation of a modern, digital-first technology team. The role builds and maintains the platforms that support Hajoca's contractors and streamline complex supply chain workflows. By developing scalable, user-centric software, you directly influence the efficiency of Hajoca's business operations and the experience of its professional partners.

Find out what you will be typing into. A shared plain-text editor with no autocomplete, compiler or test runner changes what you have to hold in your head, and practising inside your own configured environment hides exactly that gap.

Focus areas

  • Make every write idempotent under client retries
  • Evolve APIs without breaking pinned SDK clients
  • Keep money in integer minor units

Overview

This role is both challenging and rewarding because it requires a blend of technical precision and business acumen. Whether you are working on a full-stack application or contributing to an AI-driven platform, your work will bridge the gap between traditional industry needs and cutting-edge software solutions. You will be expected to think critically about system architecture, cloud integration, and the long-term impact of your code, ensuring that Hajoca remains an industry leader in a competitive market.

You will be joining an organization that values both technical excellence and operational impact. Focus your preparation on how your engineering choices translate into business value for Hajoca's contractors.

The Hajoca interview process

Hajoca candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Screening Call

Half of this call is the part candidates treat as small talk: start date, notice period, work authorisation and its timing, location and time zone, on-call, and the number. Those are what kill offers late, after several engineers have each spent a day. Surfacing a hard constraint now costs you nothing and occasionally buys you something, since a loop compressed to fit a competing deadline can usually only be arranged if it is asked for early. The common failure is deflecting the compensation question twice, then discovering at offer stage that the band never reached your number.

What to demonstrate

  • Whether your hard constraints are compatible with the role before a loop gets booked: earliest start, notice period, what authorisation you hold and when it needs action, days on site, willingness to carry a pager
  • Whether you give a compensation range with something behind it, such as current total compensation or a competing timeline, rather than leaving the band untested
  • Whether your stated timeline is real, since a competing deadline raised now is something scheduling can sometimes work around and the same deadline raised at offer stage usually is not

How to prepare

  • Write each constraint down in one line before the call and state them as facts rather than negotiating them live under a question you were not expecting
  • Set your range from two or three current data points for that level and location, and name the structure you are quoting in, so the number is comparable to the one they are holding
  • If another process is running, say where it stands and by when, and ask directly whether this loop can be scheduled inside that window

Technical Assessment

The same problem is scored by two different mechanisms depending on the format, and preparing for one does not cover the other. With a person watching, partial progress is visible and a hint is a correction you can absorb; silence is the expensive failure, because nobody can read a half-written function. With an automated grader there is no partial credit for what you were about to do, nobody to ask, and the worked examples in the prompt are the entire specification. Read them as a contract, down to whether an empty result should be an empty list or no output at all.

What to demonstrate

  • In a live session, whether your commentary tracks what your hands are doing, and whether a hint redirects you or gets defended against
  • In an automated one, whether you cover the cases the examples do not show, since the hidden cases are where the score moves
  • Whether you manage the clock on purpose: abandoning an approach that is not converging while there is still time to write something simpler that finishes

How to prepare

  • Have someone hand you a problem and feed you one deliberately wrong hint. Practise testing it against a concrete case instead of accepting or rejecting it on authority.
  • Do one timed run a week in a plain browser editor with autocomplete, linting and your own snippets switched off, which is closer to what these environments give you
  • For the automated format, write the harness before the solution: a main that feeds the worked examples plus an empty and a single-element case and prints expected against actual, so a wrong submission is caught by you first

Second Round Interview

Because the format is not fixed, the first job in the room is classification. Listen to the opening question and decide what it is: a probe into work you have already described, a fresh problem to solve now, or a conversation about how you operate. Each wants a different register, and the common failure is forcing a rehearsed structure onto a question that did not ask for it. Running a full design ritual on a ten-minute debugging question reads as not listening. When you cannot tell which it is, ask how long they want to spend and answer at that depth.

What to demonstrate

  • Whether the shape of your answer matches the question, so a yes-or-no gets answered before it is justified and an open prompt gets a direction before a detour
  • Whether you check how much depth is wanted instead of deciding for them, and whether you stop when the answer is complete rather than continuing until someone interrupts
  • Whether you can be redirected in the middle of an answer without restarting it from the beginning
  • Whether a question outside your experience gets an honest boundary followed by reasoning from what you do know, instead of a confident answer with nothing behind it

How to prepare

  • Rehearse one project at three lengths, roughly thirty seconds, three minutes, and a full walkthrough at the depth of a design review, and practise switching between them when someone interrupts mid-telling
  • Have someone ask you five questions of deliberately mixed type in one sitting without telling you the types, and score only whether you identified each one correctly before you started answering
  • Draft the sentence you will use to check depth, along the lines of asking whether the short version is useful here or they want the detail, and use it in a real conversation this week so the day of the round is not its first outing

Third Round Interview

An unlabelled round is first an information problem, and the cheapest information is free. Whoever schedules it can usually tell you how long it runs, who will be in the room and what they work on, whether you will be writing code and in what environment, and whether anything is being sent beforehand. Ask in writing so the answer is on record, then prepare for the two or three formats those answers still leave open instead of betting on one. What separates a strong candidate is not guessing right; it is having an opening that works whichever one it turns out to be.

What to demonstrate

  • Whether you can start work from an ambiguous brief, since tolerating a vague scope without stalling is the same thing the job asks for
  • Whether the questions you asked beforehand were ones that change your preparation, such as duration, medium and who is joining, rather than ones whose answers you could not have acted on
  • Whether you adapt when the round turns out to be something other than what you were told, instead of spending the first ten minutes visibly recalibrating

How to prepare

  • Send one short scheduling message asking four things: how long, who is joining and what they work on, whether you will be writing code and where, and whether to prepare anything in advance. Treat a vague reply as real information, since it means the round is loosely structured and you will be shaping it yourself.
  • Write one opening that works in any of the formats still open: restate in your own words what you have been asked to do, then ask which of two directions is more useful to them. Say it aloud until it stops sounding recited.
  • Set up for the two most likely formats before the call starts, with a blank editor in the language you would choose and a shared document you can type into, so a format surprise costs you nothing in the first minutes

Where candidates lose points

Checking a quota with a select and then writing

Under read-committed isolation, two concurrent transactions both observe a count below the limit and both insert, so the limit is exceeded by exactly the concurrency. Repeatable read does not rescue it either: it provides a stable snapshot, and this is write skew, which snapshot isolation permits by design. The options are serialisable isolation, which detects the conflict and aborts one transaction with a serialisation failure and therefore obliges the caller to retry; a single statement with the predicate inside the write; or a constraint that makes the surplus insert fail outright. The reason this pattern survives review is that it is correct in every test that runs one request at a time.

Serialising a tenant's writes through select ... for update on a single counter row

It is the first change that makes a counter correct, and it caps that tenant's write throughput at roughly one divided by the lock hold time. A transaction that takes the lock, makes a network call and then commits holds it for the entire round trip: at 2 ms that is about 500 writes per second for the whole tenant, and the largest tenants are exactly the ones that exceed it. The damage then spreads, because every waiter holds a database connection while it queues, so one hot tenant drains the shared pool and the symptom presents as a site-wide latency incident rather than as a lock problem. The repairs are to shrink the critical section to a single statement, to shard the counter into per-(tenant, hour) or per-(tenant, bucket) rows and sum on read, or to batch in memory and flush periodically while accepting the bounded loss that batching implies.

Answering a debugging question with a guess instead of a bisection

Give a procedure that halves the search space at each step: confirm the symptom reproduces, establish the last known-good version, input or timestamp, then bisect over commits, over the data, or over the layers of the request path. A plausible cause with no way to confirm it is the same move whether it happens to be right or wrong, which is why it scores nothing.

Choosing a schema before the access patterns are known

Write the queries first, with their filters, sort orders, cardinalities and which ones sit on the latency-critical path, then design tables and indexes to serve them. An index nothing queries still costs write throughput and storage, and a hot query with no supporting index becomes a full scan that only hurts once the table is big.

Questions & practice

Find peak concurrent sandbox usage from run intervals

Medium · sweep-line · intervals · concurrency-caps · null-semantics

Given up to 5 million job_run rows for one tenant over one day, with run_id, started_at, finished_at, status and wall_clock_limit_seconds, report the maximum number of sandboxes running at once, the earliest instant that maximum is reached, and the first run_id that would breach a per-tenant cap of C. started_at is null while a run is queued; finished_at is null both for runs still executing and for runs in status lost. Treat a run as occupying [started_at, finished_at). Give the complexity and state how you handle each null.

Approach

  1. Turn each run into two sweep events, (started_at, +1) and (end, -1), then sort the 2n events by timestamp with -1 ordered before +1 at equal timestamps. That tie-break is what makes the interval half-open, so a run finishing at 10:00:00 and one starting at 10:00:00 never overlap.
  2. Decide each null out loud before sweeping, because each choice moves the answer. A null started_at means queued and contributes nothing. A null finished_at with status running or leased is clipped to the window end. Status lost has no observed end at all, so clip it at started_at + wall_clock_limit_seconds on the grounds that the supervisor owns the timeout, and record that you did. The table's check (finished_at is null or started_at is not null) guarantees you never see an end without a start.
  3. Sweep once, maintaining a running counter, the maximum, and the timestamp at which the maximum was first attained (update peak_at only on a strict increase, or you will report the last such instant instead of the earliest). Capture the first run_id whose +1 takes the counter to C+1 during the same sweep rather than in a second pass.
  4. Complexity: O(n log n) dominated by the sort, O(n) space. If rows already arrive ordered by started_at, a min-heap of end times gives O(n log k) time and O(k) space with k the peak concurrency, which is the better shape when the rows come from an index scan on (tenant_id, started_at).
  5. If second resolution is acceptable, counting-sort the endpoints into an 86,400-slot delta array and prefix-sum it: O(n + T) time and O(T) space, which beats the comparison sort at 5 million rows. It answers only at second granularity, so state which resolution the cap is defined in.

Worked solution (20 min)

  1. Write the null policy as three lines of prose first, one per case, and keep them beside the output.
  2. Emit 2n endpoint tuples (timestamp, delta, run_id) and sort on the key (timestamp, delta) so -1 precedes +1.
  3. Sweep, tracking cur, peak, peak_at updated only on a strict increase, and the first run_id whose +1 takes cur to C+1.
  4. Build a fixture with two runs where one ends exactly when the next starts, three genuinely overlapping runs, one run with a null finished_at and status running, and one with status lost and a 300-second wall_clock_limit_seconds.
  5. Re-run with every timestamp shifted by a constant and confirm the peak is unchanged while peak_at shifts by the same constant.

Expected result: Peak is 3, from the overlapping trio. The exact-handoff pair yields a peak of 1, not 2. peak_at is the start instant of the third overlapping run. The lost run occupies exactly [started_at, started_at + 300s) under the stated policy.

Verify it yourself

  • Setting peak_at on >= rather than > moves the reported instant to the last time the peak was attained. Flip it and watch the fixture change.
  • Sorting by timestamp alone, without the delta tie-break, raises the reported peak on the handoff fixture.
  • The deltas over the whole sweep sum to zero once every open-ended run has been clipped.

Follow-up

  • Now report peak concurrency per tenant for 10,000 tenants from one globally sorted stream. What changes about memory and about the sort?
  • The cap has to be enforced at dispatch rather than reported afterwards. What does the admission check look like, and where does it race?
  • How would you answer 'peak concurrency within any 5-minute window' without re-sorting?

Locate a billing reconciliation gap without rescanning ninety million events

Hard · reconciliation · dimensional-bisection · watermarks · hypothesis-testing

A tenant's sealed invoice total is 0.4% below the sum of its raw usage_event rows for the period. That tenant has 90 million events over 30 days in a table partitioned daily on ingested_at, and its rollups carry source_max_ingested_at, revision and sealed_at. Recomputing all 30 days from raw is correct, and you are not going to do it. Give the procedure that locates the divergent (workspace, sku, hour) cell, the cost of each probe, and the one query you run before any of it.

Approach

  1. Run the free query first. Sum raw quantity for the period restricted to ingested_at <= source_max_ingested_at of the sealed rollups, and compare that against the unrestricted sum. The rollup stores the watermark precisely so this can be answered without a scan. If the whole 0.4% sits above the watermark, nothing is broken: it is late data, it becomes an adjustment line, and the investigation ends in one query.
  2. Only if the gap survives that test do you bisect, and you bisect by dimension rather than by rows. Compare 30 per-day totals, then inside the offending day compare the 6 SKUs, then the workspaces, then the 24 hours. That is roughly 30 + 6 + W + 24 grouped probes, each an indexed range scan over one daily partition for one tenant, against O(N) per attempt for the naive re-fold.
  3. Quantify why naive is not merely slow but unusable mid-incident: at a generous 200,000 rows/second sequential, 90 million rows is about 7.5 minutes per attempt, you will want ten attempts, and every one competes for I/O on the same partitions live ingest is writing. The diagnostic worsens the backlog it is diagnosing.
  4. Before fetching each comparison, state what it would look like under each hypothesis. Two adjacent hours off by equal and opposite amounts is occurred_at versus ingested_at bucketing. A whole day offset by exactly N hours is a timezone applied at the wrong layer. A gap confined to one SKU in one workspace is an environment filter. The same (tenant_id, idempotency_key) present in two ingested_day partitions is the dedup horizon losing a retry that crossed midnight.
  5. Make the next bisection cheap by storing the aggregate you keep recomputing. A per-(tenant_id, ingested_day) count and quantity checksum turns step two from thirty probes into one read, and it is the same number the reconciliation job already produces.
  6. Whatever you find, the sealed period does not change value. The correction is an adjustment line pointing at the line it reverses, carrying its own source_rollup_watermark, because the original invoice is the evidence of what the customer was charged.

Follow-up

  • The gap is 0.4% in one direction on one day and 0.4% the other way the next day. What does that shape rule in, and what does it rule out?
  • How do you distinguish a duplicate from a restatement, given revision and recomputed_at on the rollup?
  • Ingest is still running while you investigate. What makes your two numbers comparable at all?

Schedule ordered webhook retries with a heap of subscription queues

Medium · heap · scheduling · backoff · head-of-line-blocking

Design the in-memory scheduler for webhook delivery. Up to 20 million rows sit in status pending or failed_retryable across 200,000 subscriptions, each row carrying next_attempt_at and attempt_count, and each endpoint having a circuit breaker. Deliveries for one subscription must be attempted in order, so at most one attempt per subscription may be in flight. Support due(now), complete(delivery, outcome) and insert(delivery) in O(log S), where S is the subscription count rather than the delivery count. Give the backoff formula you schedule retries with.

Approach

  1. Key the global heap by subscription, not by delivery. Each subscription owns a FIFO of its due deliveries in event order; the heap holds one entry per eligible subscription, keyed by its head's next_attempt_at. That is 200,000 heap entries instead of 20 million, and it makes the one-in-flight rule structural rather than a check somebody can forget.
  2. due(now): peek the minimum. If its key is in the future, sleep until then instead of spinning. Otherwise pop it, move the subscription into an in-flight set, and do not re-push it. A subscription absent from the heap cannot be dispatched twice, which is precisely how ordering is preserved.
  3. complete: on success, drop the head and re-push the subscription keyed by its new head, or leave it out when the queue empties. On a retryable failure, increment attempt_count and set next_attempt_at = now + uniform(0, min(cap, base * 2^attempt)), sampled uniformly across the whole interval. That is full jitter; deterministic backoff re-synchronises the herd you just created.
  4. Circuit breaker: park the subscription in a second heap keyed by its half-open time, so an endpoint dead for six hours costs one heap entry and zero attempts rather than consuming worker slots. Admit exactly one probe at half-open and close the breaker only on its success.
  5. Say the price of the ordering guarantee out loud. One in-flight attempt per subscription means an endpoint answering in 10 seconds drains at 0.1 deliveries/second however many workers you run, and its backlog grows until it recovers. If the customer does not need order, allow k in flight and document delivery as unordered; that is the trade, and it is a product decision.
  6. All three operations are O(log S) with O(S) resident heap memory and the queues themselves backed by the store. The database-backed equivalent is a partial index on (subscription_id, next_attempt_at) where status in ('pending','failed_retryable') claimed with FOR UPDATE SKIP LOCKED, and the write-back must be fenced on lease_token so a worker that stalled and resumed cannot overwrite a newer attempt.

Follow-up

  • One subscription has 4 million queued deliveries. What stops it from starving the other 199,999, and what does your heap look like under that load?
  • A customer requests redelivery of last Tuesday's events. Where do those rows enter your structure, and what keeps them from reordering live traffic?
  • The process restarts. How much state do you rebuild, and what stops every subscription from being attempted in the same second?

Decide which facts an invoice line copies instead of joining

Medium · denormalisation · immutable records · foreign keys · rounding

invoice_line_item already denormalises tenant_id, which is reachable through invoice_id, and stores amount_minor even though quantity times unit_price_micros would recompute it. A reviewer asks you to normalise both away, and separately asks whether the tenant's legal name and billing address should be copied onto the invoice header. Decide each case. For every field you keep denormalised, name the read pattern or the invariant that justifies it, the anomaly the copy can develop, and the mechanism that prevents that anomaly here.

Approach

  1. Split the question into two kinds of copy, because they fail differently. A copy of a currently mutable fact is a cache: it drifts and needs invalidation. A copy of a fact frozen at write time is not a cache at all, it is the record of what happened, and normalising it away destroys information the source no longer holds.
  2. Keep tenant_id on the line. It costs 8 bytes, it leads every index on the table so no read is ever accidentally cross-tenant, and it turns a wrong join into an empty result rather than another tenant's money. Prevent the drift structurally: a unique constraint on invoice (invoice_id, tenant_id) plus a composite foreign key from the line on (invoice_id, tenant_id) makes a mismatched pair impossible, so the database enforces agreement instead of a code review.
  3. Keep amount_minor. Rounding must happen exactly once, at a named site, with a stated mode (half-even here). If readers recompute from quantity and unit_price_micros, every reader owns a rounding decision, and half-up and half-even diverge systematically across thousands of lines rather than cancelling out. A check constraint can bound the stored value but deliberately cannot re-derive it.
  4. Copy the legal name and billing address onto the invoice header, written once and never updated. The statement must show what was true when it was sealed, and the tenant record will change afterwards. This is a snapshot for the same reason source_rollup_watermark is stored per line: without it, nobody can reconstruct what the customer was told.
  5. Name the read pattern that pays for all of it. Rendering, dispute response and export are per-tenant, per-period reads over thousands of lines that would otherwise join back to slowly changing dimensions that no longer hold the historical value. The write side is a once-per-period batch, so the extra columns cost nothing that matters.
  6. Concede the case where the reviewer is right: a mutable operational attribute such as the tenant's current plan name has no business on a line. If a report wants it, join. If a statement needs the plan as of the period, that is another snapshot and it belongs on the header with the rest.

Follow-up

  • Write the composite foreign key and the unique constraint it requires on the parent. What does it cost on every line insert, and what does it do to a bulk load?
  • A tenant is renamed after being invoiced. Which rows change, and what does the customer see on last quarter's PDF?
  • Where does currency live, and what breaks if a tenant's billing currency changes between two periods?

Migrate a live partitioned event table without blocking ingest

Hard · online migration · ddl locks · expand and contract · partitioning

usage_event is range-partitioned daily on ingested_at, holds roughly 250M rows per day across 400 live partitions, and is written at 10-40k rows/second. Two changes are required: quantity must move from double precision to numeric(20,6), and a new environment column must become NOT NULL with a default of 'production'. Ingest cannot stop. Give the ordered plan, naming for each step the lock it takes, what that lock blocks, and roughly how long it is held. Identify the one step that cannot be rolled back cleanly once traffic depends on it.

Approach

  1. Classify the two changes before planning anything. Adding a column with a non-volatile default has been metadata-only since PostgreSQL 11, so it is cheap. Changing double precision to numeric is not binary-coercible, so alter column ... type rewrites every partition under ACCESS EXCLUSIVE and rebuilds its indexes; on this volume that is hours of blocked ingest and is simply not an option, which is why the plan is expand-and-contract rather than one statement.
  2. Expand: add quantity_numeric numeric(20,6) and environment with its default on the parent. Both are catalogue-only but both take a brief ACCESS EXCLUSIVE that cascades to partitions, so run each with lock_timeout set to a second or two and retry on failure. A queued ACCESS EXCLUSIVE request blocks every reader behind it, which is how a metadata-only change turns into an outage.
  3. Dual-write: deploy producer code that populates both columns on every insert, and leave it running before anything reads the new column. This is the step that cannot be reverted cleanly. Once readers depend on quantity_numeric, reverting the writer leaves rows with a null there, and the gap is only discoverable by re-reading the old column, which the readers have stopped doing.
  4. Backfill older partitions in batches keyed on the primary key, oldest first, committing every few thousand rows with a pause between batches, and skipping the partition still receiving writes until it rotates. Each batch is an ordinary UPDATE taking row locks only. The cost is bloat and WAL rather than blocking, so watch dead tuples and let autovacuum keep pace instead of wrapping 400 partitions in one transaction.
  5. Make NOT NULL cheap with the three-step form: add constraint ... check (environment is not null) not valid (brief ACCESS EXCLUSIVE, no scan), then validate constraint (SHARE UPDATE EXCLUSIVE, scans while reads and writes continue), then set not null, which from PostgreSQL 12 uses the validated check and skips its own full scan. Do this per partition, then on the parent.
  6. Switch and contract: move reads to the new column behind a flag, verify over a full period that both columns agree on freshly written rows, drop the old column (metadata-only), and only then remove the dual-write. Any index on the new column goes on with CREATE INDEX CONCURRENTLY per partition, since CIC is not supported on a partitioned parent: create the parent index with ONLY, build each child concurrently, then ALTER INDEX ... ATTACH PARTITION until the parent index becomes valid.

Worked solution (45 min)

  1. On a scratch cluster, build 10 partitions of 2M rows each and run a writer at a few thousand inserts/second.
  2. Run the naive type change and measure how long writes stall and how far ingest lag grows before killing it.
  3. Run the expand step with lock_timeout = '2s' while the writer runs, and observe a clean lock timeout and retry instead of a pile-up of blocked readers.
  4. Backfill in 5k-row batches and chart dead tuples and WAL generated per batch.
  5. Run the not-valid, validate, set-not-null sequence and confirm from pg_stat_activity and timings that nothing held an exclusive lock through a full scan.
  6. Add an index with CIC per partition plus ATTACH PARTITION and confirm the parent index reports valid only after the last attach.

Expected result: An ordered expand, dual-write, backfill, switch, contract plan with the lock named per step: metadata-only ADD COLUMN under a brief ACCESS EXCLUSIVE taken with lock_timeout; batched UPDATE backfill under row locks; CHECK NOT VALID, then VALIDATE under SHARE UPDATE EXCLUSIVE, then SET NOT NULL; CREATE INDEX CONCURRENTLY per partition with ATTACH PARTITION; and the dual-write-to-read switch identified as the step that cannot be reverted cleanly.

Verify it yourself

  • No step in the plan holds ACCESS EXCLUSIVE for longer than the configured lock_timeout.
  • The writer's error rate stays at zero through the rehearsal apart from the deliberate lock timeout.
  • After VALIDATE, SET NOT NULL completes in milliseconds on a table where the naive form scanned every row.
  • Freshly written rows agree on both columns for a full verification period before the old column is dropped.

Follow-up

  • A CREATE INDEX CONCURRENTLY fails halfway through the partition list. What state is the table in, how do you detect it, and what do you run?
  • The producer computes quantity itself. What happens to a request already in flight when the dual-write deploy lands, and does it matter?
  • Give two queries that prove the backfill is complete: one cheap enough to run every minute, one authoritative.

How do you approach designing scalable features for a contractor-facing platform?

Medium · system design and architecture

Approach

  1. Name the read and write paths separately; they rarely have the same bottleneck.
  2. Choose a partition key and say what query it makes expensive.
  3. State the consistency you need, and where you are willing to be stale.

Follow-up

  • What breaks first when traffic grows ten times?
  • What would you drop to keep the system up under load?

Are you familiar with cloud-based infrastructure and support?

Medium · system design and architecture

Approach

  1. State the consistency you need, and where you are willing to be stale.
  2. Choose a partition key and say what query it makes expensive.
  3. Fix the scope first: who calls this, how often, and what they do when it fails.

Follow-up

  • What breaks first when traffic grows ten times?
  • How does this behave when that dependency is down for an hour?

How do you troubleshoot performance bottlenecks in a production-level application?

Medium

Approach

  1. Clarify what is being asked and what a complete answer contains.
  2. Work from the requirement backwards to the design.
  3. State your assumptions explicitly before working the problem.

Follow-up

  • How would you know your answer was wrong?
  • What assumption would you test first?

Walk me through your approach to the provided take-home assessment.

Medium

Approach

  1. State your assumptions explicitly before working the problem.
  2. Clarify what is being asked and what a complete answer contains.
  3. Say what you would check first and why it is the highest-information step.

Follow-up

  • How would you know your answer was wrong?
  • What assumption would you test first?

Metering ingest that survives a six-hour producer replay

Hard · idempotency · dedup horizon · partitioning · watermarks

metering-ingest consumes usage events at-least-once - 250M/day, 10-40k/second at peak - and folds them into usage_rollup_hourly keyed (tenant_id, workspace_id, sku, hour_start). usage_event is partitioned daily on ingested_at with unique (ingested_day, tenant_id, idempotency_key). A producer outage ends in a six-hour replay that re-sends events already ingested, some of whose originals crossed midnight. Design the consumer: partitioning, where the acknowledgement sits relative to the commit, the deduplication horizon and its storage cost, and how the rollup watermark advances. Nothing may be double-counted and nothing may be silently dropped.

Approach

  1. Choose the acknowledgement position deliberately and name what each choice costs. Acknowledging after the fold commits makes the consumer at-least-once: a crash between the two replays the batch and produces duplicates, which are ordinary and absorbable. Acknowledging first makes it at-most-once: a crash between the two drops revenue with no error raised anywhere and no way to detect it later. Take at-least-once and design everything downstream to absorb duplicates.
  2. Put the dedup and the fold in one transaction so there is no window between them. Insert the batch into usage_event with ON CONFLICT DO NOTHING, take the rows actually inserted, and fold only those into usage_rollup_hourly with an upsert on (tenant_id, workspace_id, sku, hour_start) bucketed by occurred_at, not ingested_at. A batch of about 2,000 rows is one round trip and one index probe per event.
  3. Attack the partition-key flaw head on: the unique index includes ingested_day because a unique index on a partitioned table must contain the partition key, so the same (tenant_id, idempotency_key) re-sent after midnight is a different index entry and passes. Deduplicate instead against a store keyed (tenant_id, idempotency_key) with no date component, whose horizon exceeds the producer's maximum retry window plus the longest replay you intend to support. At 14 days that is 250M x 14 = 3.5 billion keys, which is a dedicated key-value store, not a larger index on the same table. The alternative - partitioning usage_event on (tenant_id, occurred_day) so the natural key is stable - fixes dedup but loses pruning on ingest time and makes retention by dropping partitions awkward.
  4. Partition the consumer by hash of tenant_id so one tenant's replay stalls only its own partitions, and give replay traffic a separate lower-priority lane so live ingest keeps its latency. The cost is explicit: that tenant's watermark lags while the replay drains, and everything gated on the watermark waits for it.
  5. Define the watermark as a property of committed work, not of wall-clock time: per partition it is the largest occurred_at such that every event with a smaller occurred_at has committed, and the sealing decision uses the minimum across partitions. Record source_max_ingested_at on every rollup row so any number can prove what it did and did not include, and keep restatement legal only while status = 'open' - after sealed_at the value is frozen and a late event becomes an invoice adjustment instead.

Worked solution (40 min)

  1. Write the consumer loop in pseudocode with the acknowledgement after the commit, then annotate each line with what is lost or duplicated if the process dies exactly there.
  2. Size the dedup store: events/day x horizon_days keys, bytes per key including the tenant prefix, and the resulting memory or disk. Compare that cost against simply extending retention on the partitioned table and say why the latter does not fix the problem.
  3. Take one event ingested at 23:59:58 and replayed at 00:00:04 and work out its fate under (a) the partitioned unique index alone and (b) the separate dedup store.
  4. Write the per-partition watermark formula, then what the seal uses, then what a single stalled partition does to sealing.

Expected result: An at-least-once consumer acknowledging after the commit; deduplication on (tenant_id, idempotency_key) in a store whose horizon of at least 14 days exceeds max retry window plus supported replay length, sized at roughly 3.5 billion keys; tenant-hashed partitions with replay on a separate lane; a per-partition watermark with sealing gated on the minimum across partitions; source_max_ingested_at written on every rollup row.

Verify it yourself

  • The midnight-crossing retry must be rejected by whatever you chose. If the table's unique index is your only dedup, it is accepted and the invoice is wrong by exactly that event.
  • You should be able to name a distinct failure for acknowledge-before-commit and for acknowledge-after-commit. If both sound the same, the position was not actually chosen.
  • quantity_sum for any (tenant, workspace, sku, hour) must be reproducible by re-aggregating usage_event filtered to ingested_at <= source_max_ingested_at. If it is not reproducible, the rollup is an opinion rather than a projection.

Follow-up

  • The dedup store is lost entirely. What can you still guarantee, and how do you rebuild it from what remains?
  • A replay delivers events for an hour that is already sealed. Trace exactly what happens to them, row by row.
  • One partition is stuck on a poison message, so the minimum-across-partitions watermark never advances and no tenant can be sealed. What is your escape hatch and what does it cost in correctness?

Metering partition crash-loops and the sealing watermark freezes

Medium · poison message · at-least-once · dead-letter queue

One metering-ingest partition has stopped advancing. Lag grows linearly, the consumer restarts about every 40 seconds, and the same offset appears in every startup log while other partitions stay healthy. Events are committed in batches of a few thousand and the acknowledgement follows the commit. Sealing is six hours away and source_max_ingested_at for that partition's tenants is frozen. Give an ordered checklist, a containment action available within minutes, and the durable fix, saying what each does to exactly-once accounting.

Approach

  1. Distinguish a poison record from a capacity problem in one measurement: compare the offset and the exception across restarts. An identical pair every time is deterministic failure on one record, whereas a throughput problem still advances the offset between crashes.
  2. Read the record from a separate consumer group so the bytes can be inspected without perturbing the stuck consumer, then classify the defect: schema violation, a quantity failing the non-negative check, a null workspace, an unmappable SKU enum, or a payload past a size limit. That classification decides whether this is a producer bug or a missing consumer guard.
  3. Account for batch granularity before acting. With commits of a few thousand, one bad record fails thousands of good ones, so the blast radius is the batch. Halve the batch around the offset to isolate the record, or move to per-record error isolation so the radius becomes the record.
  4. Contain by diverting that record to a dead-letter store with its raw bytes and offset, then resume. This is safe here precisely because the acknowledgement follows the commit: the good records from the failed batch are re-consumed and absorbed by the uniqueness check on (tenant_id, idempotency_key) rather than counted twice.
  5. Make the fix durable with per-record error isolation, a bounded poison counter, and an alert on dead-letter rate rather than on lag alone, since lag only reveals this after the sealing margin has already been eaten.
  6. Check the horizon before replaying anything. The unique index lives on a daily-partitioned table and therefore includes the partition key, so it deduplicates within a day only; a replay landing on a later ingest day needs the separate dedup store or it double-counts into a tenant's bill.

Follow-up

  • Move the acknowledgement before the commit and describe exactly what is lost and what is duplicated in each of the two crash windows.
  • Sealing is in six hours and the partition will not drain in time. What do you seal on, and what does the invoice have to record so the difference is explainable later?
  • A producer replays two weeks of events next month. Which part of your fix stops holding, and what is the dedup horizon you would actually configure?

Suggested practice checklist

Day one measures instead of guessing, under a fixed rubric, and the remaining hours are allocated in proportion to the gaps before any studying begins. The allocation is deliberately not renegotiated midweek, because the area that feels worst on day three is usually the one that is moving.

Day 1 — Diagnostic, scored before you study anything

  • Sit a 110-minute diagnostic in four blocks: forty-five minutes on two coding problems, twenty-five on one design prompt taken to interface and data model, twenty of short-answer fundamentals, and twenty delivering two behavioural answers aloud.
  • Score each block from 0 to 3 on a fixed rubric where 3 is correct and fluent, 2 is correct but slow or prompted, 1 is partially correct and 0 is stuck, grading the artifact rather than how the attempt felt.
  • Allocate days two to five in proportion to 3 minus each block's score, write the allocation down, and commit to leaving it alone.

Deliverable: A scored rubric and a fixed hour allocation for the rest of the week.

Day 2 — Largest gap: find the boundary rather than the subject

  • Split the weakest area into named sub-skills and rate each separately. For coding those are restating the problem, choosing the structure, stating the invariant, turning the invariant into loop bounds, handling empty and single-element input, and accounting for complexity out loud.
  • Attempt three items positioned just above where the rating drops off, and for each write the first move you failed to make.
  • Re-attempt one of them from blank four hours later with nothing open.

Deliverable: A sub-skill map with the two blocking sub-skills circled.

Day 3 — Drill the blocking sub-skill by repeating the shape

  • Do eight short repetitions of the same shape rather than eight different problems, so what gets practised is the pattern and not the puzzle.
  • State the rule you now hold in one sentence, then test it against a case built to break it, a sliding window over an array containing negative values, or a cache-aside read path whose invalidation message is dropped.
  • Have someone else read your one-sentence rule and find the precondition you left out.

Deliverable: One rule statement with its preconditions attached and one counterexample that would have caught the incomplete version.

Day 4 — Second gap, plus maintenance on the strongest area

  • Run the same sub-skill decomposition on the second-largest gap in half the time.
  • Spend twenty-five timed minutes on the block you scored highest, choosing the hardest item you can still finish rather than a warm-up.
  • Write whether each area fails you on recall, on setup, or on execution, and set the fix accordingly: repetition for recall, a written checklist for setup, timed work for execution.

Deliverable: A second sub-skill map plus a one-line failure diagnosis for each area.

Day 5 — The gap that is not a skill

  • Record one technical and one behavioural answer, then count two things in the playback: seconds before your first clarifying question, and sentences you began without knowing where they would end.
  • Practise saying that you do not know, followed by how you would find out, without letting it soften into a guess, and practise stating a complexity or an estimate before being asked for it.
  • Redeliver one answer under a hard ninety-second cap, which forces structure ahead of detail.

Deliverable: Two recordings with a counted reduction in time-to-first-question.

Day 6 — Retest under day-one conditions

  • Sit the same 110-minute structure with new prompts of comparable difficulty and score it on the identical rubric.
  • For any block that did not move, change the method rather than adding hours: a block stuck at 1 usually means the practice was too varied, not too short.
  • Write down which single block you would still lose the offer on.

Deliverable: A second scored rubric placed beside the first, with one named remaining risk.

Day 7 — Full loop under interview conditions

  • Run a sixty-minute mock over the two blocks that moved least, with an interviewer briefed to interrupt and change direction mid-answer.
  • Write the recovery script for going blank: restate the question, state your assumption, name the first thing you would check.
  • Say every rule from the week aloud without reading it, and cut any you cannot state in a single sentence, since a rule you have to reconstruct mid-answer will not survive an interruption.

Deliverable: A one-page card holding the recovery script and only the rules you could state from memory.

Risky changes and the reversal path

For anything that touched live traffic, be ready to say how you would have undone it: a flag, a staged rollout, dual writes with the old path still authoritative. Once the old column is dropped or the source rows are overwritten there is no reverse, so name what you kept a copy of and for how long.

Describe a time you had to pivot your technical approach due to changing project requirements.

Medium · behavioural and engineering judgement

Approach

  1. State the situation in two sentences and spend the rest on the reasoning.
  2. Pick a story where you made the decision, not one where you watched it.
  3. Name the disagreement and how you resolved it with evidence.

Follow-up

  • What would you do differently if you ran that again?
  • What did you decide not to do, and why?

Can you describe your experience with full-stack development lifecycles?

Medium · behavioural and engineering judgement

Approach

  1. State the situation in two sentences and spend the rest on the reasoning.
  2. Give the blast radius: what could have broken, and what you measured.
  3. Name the disagreement and how you resolved it with evidence.

Follow-up

  • How did you know your change caused the improvement?
  • What would you do differently if you ran that again?

Unblock an engineer on a job run that finished twice

Easy · mentoring · fencing tokens · lease expiry · debugging method

An engineer two weeks into the team brings you a job_run row showing status succeeded with an exit_code written by a worker declared dead ten minutes earlier; the retry attempt also shows succeeded. They have spent a day adding logging and are no closer. You have twenty minutes and you do not want to take the keyboard. Describe how you unblock someone: the question you ask first, what you let them find themselves, the concept you name and when, and how you check the next day that they own the fix rather than having watched you produce it.

Approach

  1. Ask what they expect rather than what they see: which statement set status to succeeded, and what did it check before writing? That question points directly at the update's WHERE clause, which is where the answer lives, and it costs them nothing to answer, so it does not read as a test.
  2. Let them build the timeline themselves from the row: queued_at, started_at, leased_until, finished_at and worker_id, on both the original run and the retry. Two different worker_ids with a lease expiry between them tells the whole story, and they will see it before you say it.
  3. Name the concept once the evidence has earned it. A lease bounds time; it does not prevent a write. The store has to reject a stale writer, which means the update carries a fencing token the row compares — update job_run set status = 'succeeded' where run_id = $1 and lease_token = $2 and status = 'running' — and a long garbage-collection pause or a brief partition is enough to produce what they are looking at.
  4. Point at the second, less obvious half and let them decide it: 'lost' exists in the status enum precisely so a run whose worker vanished is not recorded as failed, because failed asserts an outcome nobody observed and the system then bills and retries on that assertion. Ask them what these two rows should have said.
  5. Leave them with the next step rather than the patch — a test that kills the first worker after the sandbox exits and before the row is written — and say when you are available again, so the offer is real rather than polite.
  6. Check ownership the next day by what they produced, not by asking if it went well: a test that reproduces the window proves they understood it; a test that only asserts the new WHERE clause proves they copied it. Ask them to explain it to a third person and listen for whether the explanation is theirs.

Follow-up

  • They propose a longer lease instead of a token. What do you say, and what breaks when legitimate runs last thirty minutes?
  • How can you tell whether your explanation landed or they simply deferred to you?
  • The same engineer hits a variant of this next month. What did you fail to teach the first time?