Interview guides · Arabelle Solutions
Arabelle Solutions Software Engineer Interview Questions & Guide 2026
API design practice for Arabelle Solutions software engineer candidates: compatibility, versioning, and changes you cannot take back.
The 60-second brief
As a Software Engineer at Arabelle Solutions, you sit at the intersection of complex industrial systems and cutting-edge digital control. This role is not merely about writing code; it is about engineering the logic that powers critical energy infrastructure. You will contribute to projects ranging from turbine control systems to assembly process automation, ensuring that Arabelle Solutions' technical solutions meet the demands of the global energy market.
Ask whether any round happens inside an existing repository instead of a blank file. Reading unfamiliar code, isolating a fault and making the smallest correct change is a different skill from writing a function from scratch, and it needs its own practice.
Focus areas
- Scope every query by tenant below application code
- Evolve schemas expand-contract across un-upgradable client deployments
- Make integration retries idempotent without target-side idempotency keys
Overview
The impact of your work is tangible and immediate. Whether you are optimizing a manufacturing process or refining the software architecture for a turbine control system, your contributions directly influence the reliability and efficiency of power generation assets. Candidates report that Arabelle Solutions favors engineers who understand both their technical domain and the broader physical systems their software controls. You will work in an environment where technical precision is paramount, and your ability to bridge the gap between abstract code and mechanical reality is essential.
The Arabelle Solutions interview process
Arabelle Solutions candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.
Initial Screening
The person on this call usually cannot evaluate your code and does not need to. They write a short paragraph, and that paragraph is what a hiring manager skims when deciding who to put on your loop. So the test is not whether your work was hard, it is whether a non-engineer can repeat it correctly. Name systems by what they did rather than by their internal codename, give each project a shape (what was breaking, what you changed, what happened after), and keep the whole walkthrough near ninety seconds. Depth that cannot survive a paraphrase reads as vagueness.
What to demonstrate
- Whether a non-engineer can restate your projects without distorting them, since their paraphrase is what travels to the hiring manager, not your sentences
- Whether each project has a shape rather than a stack list: the failure or constraint, the change you made, the result and how it was measured
- Whether you can say what was yours inside a team project without either inflating it or disappearing into the plural
How to prepare
- Rewrite each headline project as two sentences with no internal system names and no acronyms outside your company, then say them to someone outside engineering and have them repeat them back. Fix whatever came back wrong
- Attach one measured number to each project: the baseline, the change, and the window it was measured over. Where nothing was ever measured, say that plainly rather than reaching for a plausible percentage
- Time the background walkthrough against a clock. If it runs past two minutes, compress the earliest role to a single clause and spend the recovered time on the most recent one
Technical Deep-Dives
Most of the time lost in this format is not lost to thinking. It goes to a standard-library call you half-remember, an off-by-one in a loop bound, and a debugging loop that mutates code at random until something passes. When output is wrong, stop re-reading the whole function: take the smallest input that reproduces it and walk the state through by hand, printing intermediates if the environment allows. Guessing at a fix without a failing case you understand is how a five-minute bug becomes twenty, and the clock does not pause while you do it.
What to demonstrate
- Whether you reach the right structure without a detour, and can write it from memory rather than only recall that one exists
- Whether overflow is considered where the language has fixed-width integers, since a signed 32-bit value stops at 2,147,483,647 and then wraps in Java, is undefined behaviour in C++, and does not arise in Python, whose integers grow instead
- Whether recursion depth is treated as a constraint on large inputs, given that CPython's default limit is 1000 frames and a deep recursion can exhaust the stack in any language where an iterative version would not
- Whether a failing case is isolated and explained before any edit is made to the code
How to prepare
- From an empty file and with no references open, implement the pieces you lean on most: a heap push and pop, an iterative DFS with an explicit stack, and a binary search whose midpoint is written lo + (hi - lo) / 2, which avoids the overflow that (lo + hi) / 2 can hit in a fixed-width integer type
- Time yourself on the ten library calls you look up most, such as sorting with a custom comparator, splitting and joining strings, and finding the next key at or above a value in an ordered map, until the lookup is gone
- Take a solution you know is broken and, before touching it, write one sentence naming the input, the expected value and the actual value. Repeat until you do it without deciding to.
Behavioral Assessments
Many of these questions are about something that went wrong, and the grading sits mostly in the hours after you knew. Who found out first, whether that was you or an alert or a user, how long it took you to say it out loud, and whether the people who needed the news got it while they could still act on it. Engineers under-tell this part because it feels like confessing. The pattern it is looking for is the opposite: the quiet fix, an incident absorbed without telling anyone, after which nothing changed and the same failure is still available.
What to demonstrate
- How the problem was found, and whether that route was one you had built or one that happened to you, since a user reporting it first means your instrumentation did not cover that failure
- Whether time-to-detect and time-to-tell are separate numbers in your account and whether you know both, because a fast fix that nobody heard about until the retro is a different answer from a slow one that was announced immediately
- Whether the resolution left something durable behind, a check that fires or a default that changed, rather than depending on people remembering to be careful
- Whether you can say what the failure cost without either inflating it or waving it away
How to prepare
- Reconstruct one incident you were part of as a timeline with clock times: first bad request, first signal, first person who knew, first message outside the team, mitigation, permanent fix. The gaps between those entries are what gets asked about
- Look up the configuration of the signal that caught it, including its evaluation window and threshold. An alert defined on a five-minute aggregate cannot fire until the condition holds across that window, which puts a floor under time-to-detect that has nothing to do with how severe the failure was. Be able to say what that floor was and whether anyone had chosen it deliberately
- Prepare one story where you escalated early and the severity turned out to be smaller than you thought, including what it cost the people you pulled in. Without it, every answer you give about raising alarms is unfalsifiable
Where candidates lose points
Depending on a client field because it is always populated
It is populated by their operational process, not by a constraint in their database, and their process changes without telling you. When it stops being populated, a permissive pipeline writes nulls or defaults into derived data and the damage spreads for weeks before anyone notices. Validate at the boundary and fail the run with failure_class='schema_mismatch' plus a redacted sample, so the failure is loud, attributable to the client's change, and cheap to diagnose. A run that fails clearly is far cheaper than a backfill that has to reconstruct which rows were silently wrong.
Long-lived shared credentials for client systems
One static key used by the connector runtime, a debugging script and three engineers cannot be attributed to a person in an audit, cannot be expired at engagement close, and cannot be rotated without breaking an unknown number of consumers at once. The expensive part of the eventual incident is not the leak; it is that rotation requires finding every holder under time pressure, and nothing recorded who they were. Issue per-principal, per-target, short-lived grants from a broker so rotation is a no-op, attribution is automatic, and closure is a TTL rather than a search.
Naming no test cases at all
State what you would test before being asked: empty input, a single element, all elements equal, the maximum permitted size, and the input that exercises the branch you just wrote. It costs thirty seconds and is much of what separates someone who has shipped code from someone who has only solved puzzles.
Designing for a scale nobody asked for
Ask for request rate, data size, read-to-write ratio and expected growth, then size the simplest option first; one relational instance on current hardware covers a large share of real workloads. Reaching for shards, queues and a cache tier before any number has been quoted reads as pattern-matching rather than judgement.
Questions & practice
Resolve effective-dated engagement versions for two million access checks
Hard · binary search · effective dating · complexity budget
The engagement table holds 500,000 effective-dated rows: (engagement_id, version, effective_from, effective_to which is NULL for the open version). A database exclusion constraint guarantees that per engagement_id the tstzrange(effective_from, effective_to) values never overlap, and the ranges are half-open. You must answer 2,000,000 access checks of the form (engagement_id, at_time) in one process with no database, returning the version open at that instant or none. Also report, per engagement, any coverage gap between adjacent versions. Give the complexity of the scan-per-check approach and say why it is not an option.
Approach
- State the naive cost first: scanning the table per check is 2e6 x 5e5 = 1e12 range comparisons. Even at 1e9 simple comparisons per second that is roughly fifteen minutes of pure compare work, against a budget measured in seconds. It is correct, just unaffordable, and the fix is an index built once rather than a faster comparison.
- Build a hash map engagement_id -> array of versions sorted by effective_from, once. O(V log V) to sort, O(V) memory. The build is amortised over 2e6 lookups, which is what makes it worth doing at all.
- Answer each check by binary-searching for the last effective_from <= at_time, then confirming effective_to IS NULL OR at_time < effective_to. If the confirmation fails, the instant lies in a gap and the answer is none, not the neighbouring version. O(log v) per check, O(V log V + Q log v) total.
- Name the precondition: predecessor search is only valid because the ranges per engagement do not overlap, which the exclusion constraint enforces with btree_gist for the equality operator. Without that guarantee you need an interval tree returning all stabbed intervals, at O(log n + k). Use the same half-open convention as the database, where at_time equal to effective_to belongs to the next version.
- Find gaps with one O(V) pass over each sorted array comparing prev.effective_to against next.effective_from. The exclusion constraint forbids overlaps, not holes, so a hole is entirely possible and is where an access check fails closed for a reason nobody can see in the contract.
- If the checks can be buffered, sort them by (engagement_id, at_time) and merge against the version arrays: O(Q log Q + V log V), usually a large constant-factor win over 2e6 random binary searches because the merge walks memory in order.
Follow-up
- The open version has effective_to NULL. How do you keep both the sort and the comparison total without a sentinel that overflows your timestamp type?
- A gap exists for one engagement and a check lands in it. What should the access path return, and what should it emit so someone fixes the amendment?
- Amendments are being written while you answer checks. What read model keeps an answer self-consistent, and what does a check mean if the snapshot is a second stale?
Compute the minimal re-run set that preserves idempotency keys
Hard · interval complement · idempotency keys · watermarks
One connector binding must cover [span_start, span_end). You have all its connector_run rows: (idempotency_key, attempt_no, status, watermark_from, watermark_to), with half-open ranges, where idempotency_key is derived deterministically from the connector and the range, so re-running a range reproduces its key. Only status='succeeded' counts as covered. Produce the minimal re-run plan: every instant in the span ends up covered, any sub-interval some prior run already claimed is re-run at exactly that claim's range, and genuinely unclaimed sub-intervals become new claims cut at the binding's chunk size. Explain what merging ranges would cost.
Approach
- Merge the succeeded ranges into a maximal disjoint union: sort by start, sweep extending the open segment while the next start is not beyond the current end. O(n log n) time, O(n) space, and this is the only status that contributes coverage.
- Take the complement of that union inside [span_start, span_end) to get the uncovered gaps, in O(u) over the merged segments. Clip at both ends so the plan never reaches outside the span.
- Sweep the non-succeeded claims (ambiguous, failed, abandoned) against the gap list with a two-pointer over both sorted lists. Any claim intersecting a gap is emitted at its original watermark_from and watermark_to, byte for byte, so the derived idempotency_key reproduces and the retry rejoins the same ledger row. O(n) after sorting.
- Subtract the emitted claim ranges from the gaps; whatever survives was never claimed by any attempt and becomes fresh ranges cut at the chunk size. Only these get new keys. O(u + c).
- Accept that a re-emitted claim may overlap time that is already succeeded. Range bookkeeping is not what makes the retry safe; the record-level dedupe key is, and re-reading a covered window costs one extra fetch against the client's quota, which is far cheaper than a key you can no longer reproduce.
- Exclude anything still in status 'running' with a live lease. Re-claiming it races the in-flight attempt, doubles the quota spend against the client endpoint, and produces two ledger writers for one key. Total plan cost is O(n log n) time and O(n) space.
Follow-up
- An ambiguous run may have applied every record before the timeout. What do you read to settle it before re-running, and what is left if the client's schema offers no natural key?
- Retention pruned the ledger rows for one of these keys. What does the re-run do now, and what bounds how long the ledger must be kept?
- Two planners run concurrently against the same binding. What stops them emitting and executing the same plan twice?
Assign connector bindings to rollout waves and name the cycle
Medium · topological sort · cycle detection · dependency graph
Connector bindings declare ordering dependencies: an edge u -> v means v must not run in a wave earlier than one after u has succeeded. You have up to 50,000 bindings and 200,000 edges, and a misconfigured dependency may have created a cycle. Assign each binding the earliest wave number such that every predecessor sits in a strictly earlier wave, return the total wave count, and if a cycle exists return the bindings on one concrete cycle rather than a boolean. The implementation must be iterative: a dependency chain can be 50,000 deep.
Approach
- Build an adjacency list and an indegree array, then run Kahn's algorithm from every indegree-zero node. Bindings with no predecessors get wave 0. O(V+E) time, O(V+E) space.
- Relax waves as you decrement: when popping u, set wave[v] = max(wave[v], wave[u]+1) for each successor before enqueueing v at indegree zero. Because v is only enqueued after its last predecessor is popped, wave[v] is 1 + max over predecessors, which is the earliest legal wave. Total wave count is max(wave)+1, the longest path in nodes.
- If Kahn emits fewer than V nodes, the residual subgraph is exactly the nodes lying on or downstream of a cycle. Every residual node still has an in-edge inside the residual, otherwise it would have been popped, so walking backwards along in-edges cannot dead-end and must revisit a node within |residual| steps; the segment between the two visits is a concrete cycle. O(V) with a visited-position map.
- Use an explicit stack or a queue rather than recursion. Recursive DFS on a 50,000-node chain exceeds the default stack in most runtimes, and the failure looks like a crash rather than a cycle.
- At this size, recompute the whole plan whenever an edge changes. Incremental topological order maintenance costs far more code than the O(V+E) rebuild it saves.
Worked solution (25 min)
- Take seven bindings A,B,C,D,E,F,G with edges A->C, B->C, C->D, F->G, G->F. Indegrees: A0 B0 C2 D1 E0 F1 G1.
- Kahn from [A,B,E]: pop A (wave 0), relax C to wave 1 and indegree 1; pop B (wave 0), relax C to wave 1 and indegree 0, enqueue C; pop E (wave 0).
- Pop C (wave 1), relax D to wave 2 and indegree 0, enqueue D; pop D (wave 2). Five of seven nodes emitted.
- Residual is {F,G}. Walk backwards from F: F's in-edge comes from G, G's in-edge comes from F, so F repeats and the cycle is F -> G -> F.
- Verify the wave assignment against every edge before reporting.
Expected result: wave(A)=wave(B)=wave(E)=0, wave(C)=1, wave(D)=2, wave count 3. Cycle reported as the concrete node list [F, G]; F and G receive no wave.
Verify it yourself
- For every edge u -> v that was emitted, wave[u] < wave[v] strictly.
- The wave count equals the number of nodes on the longest emitted path (A -> C -> D = 3).
- Kahn emitted 5 nodes and V = 7, so the residual size 2 equals the reported cycle size.
- Removing edge G->F and rerunning emits all 7 nodes with wave(F)=0, wave(G)=1 and no cycle.
Follow-up
- Two bindings land in the same wave but hit the same client endpoint, whose rate budget is shared. How does the wave plan have to change, and what problem class does that turn into?
- One dependency holds only for overlapping watermark ranges rather than globally. Is the dependency graph still a DAG over bindings, and what is the right vertex if not?
- Report the critical path so an operator can see which chain sets the wave count, not just that it is five.
Explain why the failed-run backlog query ignores three indexes
Medium · indexing · query planning · partial indexes
connector_run holds roughly six million rows over three years. Its indexes are (connector_id, started_at DESC); (started_at) WHERE status = 'failed'; and ((finished_at AT TIME ZONE 'UTC')::date). The operator backlog query is SELECT run_id, connector_id, failure_class, started_at FROM connector_run WHERE environment_id = $1 AND status = ANY($2) AND started_at >= now() - interval '7 days' ORDER BY started_at DESC LIMIT 50, with $2 bound to {failed, ambiguous}. It sequentially scans. Explain index by index why each is unusable, then give the one index you would add.
Approach
- Index one leads on connector_id, which this query never constrains. A B-tree can only be probed from a prefix of its key columns, so with no equality on connector_id it degenerates into a full index scan plus heap fetches, which the planner will rightly cost above a sequential scan. PostgreSQL 18 added B-tree skip scan for an unconstrained prefix, so state your server version before assuming this is fatal; it helps only when that prefix has few distinct values, and hundreds of connectors is already too many to rely on it.
- Index two is partial on status = 'failed', and the rows the query wants are partly not in it at all. Predicate implication runs one way: the query's predicate must imply the index's. status IN ('failed','ambiguous') is weaker than status = 'failed' and implies nothing, so no amount of inlining the parameter helps. It would serve only the failed branch of a UNION ALL.
- Index three is an expression index on a function of finished_at, while the query filters started_at as a plain range. Even on the right column, a range predicate on the bare column cannot use an index on a function of it. Note also why the AT TIME ZONE 'UTC' wrapper is present: finished_at::date on a timestamptz is STABLE rather than IMMUTABLE because it depends on the session TimeZone, so PostgreSQL refuses to index it without a fixed zone.
- Add (environment_id, started_at DESC) WHERE status IN ('failed','ambiguous'): equality on the scoping column first so the scan starts in the right place, the ordering column second so ORDER BY started_at DESC LIMIT 50 stops after fifty index entries instead of sorting the week, and partial on the status set so the index covers the small failing minority rather than six million rows.
- Confirm the partial predicate is provable for the query's parameterised form. Under a custom plan the array parameter is substituted as a constant and the prover handles the ScalarArrayOpExpr; under a generic plan it stays a Param and the proof fails, so check plan_cache_mode or the sixth execution of a prepared statement before declaring victory.
- Say what you deliberately did not do. Adding INCLUDE (run_id, connector_id, failure_class) to chase an index-only scan pays only while the visibility map is current, which ties the query's latency to whether autovacuum is keeping up with the write rate — a dependency worth accepting consciously, not by accident.
Follow-up
- The same query is also run without the environment filter, for an estate-wide backlog. Does your index still help, and what would you add instead?
- How does row-level security on this table interact with your index, given that policy quals are applied before non-leakproof user quals?
- Six months from now the partial index covers 30 percent of the table. What signal tells you to change it?
Fix a billing report that double-counts through a join
Medium · join fan-out · aggregation · reporting
A monthly report joins engagement to work_record (engagement_id, principal_id, work_date, minutes, is_billable, status) and to invoice_line (engagement_id, period_start, period_end, amount_minor, line_type, status) in a single FROM clause, then aggregates sum(minutes), count(DISTINCT principal_id) and sum(amount_minor) grouped by engagement. Finance reports the minutes are roughly triple the timesheets while the headcount column looks correct. Explain the arithmetic, then write the corrected query for one client and one period, returning zero rather than NULL for engagements with no work.
Approach
- Name the arithmetic before touching SQL: joining two independent one-to-many children of the same parent produces the Cartesian product per engagement, |W| x |L| rows. sum(minutes) is therefore multiplied by the invoice-line count and sum(amount_minor) by the work-record count. Three lines per engagement — fees, expense, credit note — is exactly the factor of three finance reported.
- Explain why the headcount column looked fine: count(DISTINCT principal_id) collapses the duplication, so it is correct by accident. That is precisely why the bug survived review, and it is the reason a column agreeing with expectations is not evidence that a join is at the right grain.
- Pre-aggregate each branch to the join grain first: one subquery per fact table, grouped by engagement_id, each producing at most one row per engagement. Join those results, which are now one-to-one, so no multiplication is possible by construction rather than by care.
- Drive the outer query from engagement with LEFT JOINs and wrap each aggregate in COALESCE, because sum over zero rows is NULL rather than 0. For counts, count the key column and not count(*), which returns 1 for a non-matching LEFT JOIN row.
- Keep credit notes inside the amount sum deliberately: amount_minor is negative for line_type = 'credit_note', so filtering them out overstates what was billed. Exclude status = 'void' instead, and state that choice in the query as a comment because the next reader will assume the opposite.
- Compare cost: the fan-out plan materialises |W| x |L| intermediate rows before aggregating, while pre-aggregation reads each table once under its own index and hash-joins two small results. The base-table I/O is identical; the intermediate work differs by orders of magnitude on a large engagement.
Worked solution (25 min)
- Build a fixture: one engagement with two work records of 60 and 90 minutes and three invoice lines.
- Run the broken join and confirm sum(minutes) returns 450 rather than 150, and that count(DISTINCT principal_id) is unaffected.
- Rewrite with one aggregating CTE per fact table and LEFT JOIN both onto engagement.
- Add an engagement with invoice lines but no work records and confirm it returns 0 rather than NULL or a missing row.
Expected result: WITH w AS (SELECT engagement_id, sum(minutes) AS billable_minutes, count(DISTINCT principal_id) AS people FROM work_record WHERE work_date >= $2 AND work_date < $3 AND is_billable AND status IN ('approved','invoiced') GROUP BY engagement_id), l AS (SELECT engagement_id, sum(amount_minor) AS billed_minor FROM invoice_line WHERE period_start >= $2 AND period_end <= $3 AND status <> 'void' GROUP BY engagement_id) SELECT e.engagement_id, COALESCE(w.billable_minutes, 0) AS billable_minutes, COALESCE(w.people, 0) AS people, COALESCE(l.billed_minor, 0) AS billed_minor FROM engagement e LEFT JOIN w USING (engagement_id) LEFT JOIN l USING (engagement_id) WHERE e.client_id = $1 AND e.effective_to IS NULL ORDER BY e.engagement_id;
Verify it yourself
- On the fixture the corrected query returns 150 minutes where the broken one returned 450.
- Totals reconcile against SELECT sum(minutes) FROM work_record with the same predicates run on its own.
- An engagement with no work records appears once with zeros, not absent and not NULL.
- Adding a fourth invoice line changes billed_minor and leaves billable_minutes untouched.
Follow-up
- The report now needs a per-principal breakdown alongside the per-engagement totals. Does your shape survive, or do you need a different grain entirely?
- How do you stop the next analyst reintroducing this — a view, a naming convention, or a test that fails on a fixture with two children?
- Why does sum(DISTINCT minutes) not fix it, and what does it actually compute?
How would you interpret a technical drawing or a complex process diagram?
Medium
Approach
- State your assumptions explicitly before working the problem.
- Work from the requirement backwards to the design.
- Say what you would check first and why it is the highest-information step.
Follow-up
- How would you know your answer was wrong?
- What assumption would you test first?
What is the principle of centrifugal force?
Medium
Approach
- State your assumptions explicitly before working the problem.
- Say what you would check first and why it is the highest-information step.
- Clarify what is being asked and what a complete answer contains.
Follow-up
- How would you know your answer was wrong?
- What assumption would you test first?
Can you explain the underlying logic of the projects you have previously led?
Medium
Approach
- Clarify what is being asked and what a complete answer contains.
- Work from the requirement backwards to the design.
- State your assumptions explicitly before working the problem.
Follow-up
- What assumption would you test first?
- How would you know your answer was wrong?
Mint short-lived client credentials that expire with the engagement
Easy · credentials · ttl · audit · partitioning
The Access Broker mints credentials for client-owned systems. Its callers are connector jobs and operator sessions, roughly 400 mints a minute at peak against a 150 ms p99 budget. Every mint appends to credential_grant (grant_id, engagement_id, principal_id, target_system_id, scopes, issued_at, not_after, issued_by_request_id), and those rows stay queryable per engagement for a contracted retention period measured in years. Design the mint path and the audit store: how not_after is computed, what happens when the audit write fails, and how an auditor answers who held access to one client system last March.
Approach
- Compute not_after server-side as min(requested_ttl, engagement.ends_on + access_grace_days - now) and refuse the mint when that is already non-positive. The requested TTL is a hint; the engagement window is a bound. Then subtract the verifier's clock-skew allowance: a validator that tolerates 60 seconds of skew accepts a token 60 seconds past not_after, so effective expiry is not_after plus that leeway and it has to land inside the grace window rather than just outside it.
- Write the audit row in the same transaction that reserves grant_id, before the secret material leaves the process, and treat a failed audit write as a failed mint. A mint that failed is retryable; access that cannot be attributed to a named principal is permanent, and the asymmetry decides the ordering.
- Make the mint idempotent on issued_by_request_id with a unique index, so a caller retrying after a timeout does not leave a second untracked credential alive. On 23505, read the row back. Then choose deliberately between a short replay window that returns the same material and a conflict that forces a fresh mint, and justify it: replay means the secret crosses the wire twice, conflict means a timeout costs the caller a new credential.
- Range-partition credential_grant by issued_at monthly and index (engagement_id, issued_at DESC). The auditor's question then reads one or two partitions instead of the full history, and expiring a retention window becomes DETACH plus DROP - a metadata operation - rather than a delete of millions of rows that then has to be vacuumed.
- Note that the cross-table rule cannot be a CHECK constraint, because a PostgreSQL CHECK cannot reference another table. Either denormalise the engagement's end into the grant row and CHECK against that local column, or enforce it in a trigger, and run a scheduled query that proves no live grant outlives its engagement.
Worked solution (20 min)
- Implement the mint with not_after = min(requested, ends_on + grace - now) and enforce the same rule in the database with a trigger or a denormalised column.
- Request a 24-hour TTL against an engagement ending in 90 minutes and assert the returned not_after.
- Force a timeout on the first attempt, retry with the same issued_by_request_id, and count rows in credential_grant.
- Create 24 monthly partitions, load 10 million grants, and time the per-engagement auditor query for one month with and without partitioning.
Expected result: The 24-hour request returns roughly 90 minutes minus the skew allowance, never the requested value, and the database rejects the row if the application computes it wrong. The retry yields exactly one row. The partitioned auditor query reads one partition and returns in milliseconds while the unpartitioned equivalent scans the whole table.
Verify it yourself
- No row in credential_grant has not_after beyond its engagement's ends_on plus access_grace_days, asserted on a schedule rather than only at insert.
- Every grant carries a non-null issued_by_request_id that appears in the calling service's request log.
- Dropping a retention-expired partition touches no live rows and requires no vacuum of the parent.
Follow-up
- The connector caches a credential for its full lifetime and the engagement is terminated ten minutes later. What is actually true about access, and for exactly how long?
- An operator needs a week of access to a client system but the engagement ends in two days. What does the system do, and who is allowed to change that answer?
- How would you prove to an auditor that no credential was ever minted against a closed engagement?
One connector burns a client's entire rate budget
Easy · poison message · retry loop · rate limiting
A client reports that your integration is exhausting their API quota, and every connector binding for that client now fails with client_429. For one binding, connector_run holds 300 rows sharing one idempotency_key with attempt_no incrementing, all with status 'running', finished_at NULL and failure_class NULL, created exactly 120 seconds apart over the last ten hours. The run lease is 120 seconds. Workers in that pool show restart counts in the hundreds. Other clients are unaffected. Diagnose in order, say who is at fault, and state what stops one range doing this again.
Approach
- Split client fault from your fault before anyone contacts the client. Group the last hour's failures by client and by binding: every binding for one client failing while other clients are clean means a resource shared with that client is exhausted, and the only thing you share with them is their quota.
- Read the ledger's shape, which is the entire diagnosis. Three hundred attempts under one idempotency_key, none with a terminal status and none with a failure_class, spaced at exactly the lease interval. A handled failure writes failure_class and finished_at; writing nothing at all means the worker was killed before it could classify. The exact spacing names the killer as lease expiry, not the client endpoint.
- Quantify the damage from the same rows: records_applied is zero across all 300, while each attempt got far enough to pull pages. The binding consumes quota continuously and applies nothing, which is why the other bindings for that client see 429 and the faulty one does not.
- Reproduce offline. watermark_from and watermark_to are recorded, so replay that exact range against a captured payload with the lease disabled and find what exceeds 120 seconds: a page far larger than the others, a validator input that degrades badly, or an unbounded in-memory accumulation over the range.
- Fix the loop before the payload. Cap attempt_no, move the range to status 'abandoned' with an alert, and hold the watermark rather than advancing past it, because a silent gap is data loss and skipping must require a human decision. Then make the work fit the lease: stream and chunk the page, renew the lease only while progress is provable, and bound the page size at the boundary.
- Add the detector this evaded. Alert on attempts that never reach a terminal state, not only on failure counts, because a run that never finishes never increments any failure metric.
Follow-up
- The client asks how much of their quota you consumed. Which rows answer that, and what would you have to add to answer it exactly?
- The abandoned range contains records you still need. What is the safe way to get past it without breaking the at-most-once guarantee?
- Lease renewal introduces a new failure: a worker that renews steadily but makes no progress. How do you catch that one?
Suggested practice checklist
For a candidate senior enough that the loop turns on design and judgement rather than on whether the coding round gets finished. Five days build one system properly and then stress it; coding gets a single maintenance day, on the assumption that the risk at this level is an unexamined tradeoff rather than a missed algorithm.
Day 1 — Numbers before diagrams
- Build your own reference card of the figures you will re-derive all week: bytes for a realistic record, requests per second implied by a given daily active count, and the storage that a year at a given write rate produces. Derive each one rather than copying it, because the derivation is what survives a follow-up.
- Turn one product statement into capacity requirements. From ten million daily users at four writes and forty reads each, state the peak-to-average factor you are assuming and why, then produce peak write QPS, peak read QPS and a year of storage.
- Write the two numbers whose order of magnitude changes the design, the read-to-write ratio and the working-set size against memory per node, and state the threshold at which each one flips your answer.
Deliverable: A one-page numbers card and one worked capacity estimate with every assumption written down.
Day 2 — One system, from requirements to schema
- Spend the first ten minutes producing only functional requirements, non-functional targets with numbers attached, a p99 latency, a durability expectation, a consistency requirement, and an explicit out-of-scope list.
- Define the interface before the boxes: the three or four endpoints, their parameters, what each returns, and which of them are idempotent.
- Write the data model, then write the single access pattern that justifies it, and state what the schema would have to become if the dominant access pattern were the other one.
Deliverable: One design carried to endpoint-and-schema depth, with non-functional targets expressed as numbers and a written out-of-scope list.
Day 3 — The consistency you are actually buying
- Write out what a client sees under asynchronous replication when its write commits on the leader and its next read is served by a lagging follower, then write the two fixes, pinning that session's reads to the leader for a bounded window or carrying a version token the replica must reach, and the cost of each.
- Work the quorum arithmetic on paper for N of three with W and R of two, and separate what R + W > N does guarantee, that any read set intersects any write set, from what it does not: on its own it is not linearizability, and a sloppy quorum that accepts writes on nodes outside the preference list breaks even the intersection.
- Take two storage choices with different defaults, a single-leader relational store committing synchronously and a quorum-replicated store that converges eventually, and write the specific product behaviour that would be wrong under each, rather than a general statement about which is stronger.
Deliverable: A page separating what quorum overlap guarantees from what it does not, with one concrete product misbehaviour attached to each gap.
Day 4 — Failure is the design
- For one write path, work through the case where the client times out after the server has already committed, then design the idempotency key: who generates it, how long it is retained, and what the duplicate request returns.
- Express the retry policy as parameters rather than as a word: maximum attempts, base delay, backoff factor, jitter, and which error classes are retried at all. Then state why retrying a non-idempotent write without a key is a correctness bug and not merely waste.
- Compute the fan-out effect on tail latency. If a request waits on ten backends and each independently exceeds its p99 one percent of the time, the chance at least one is slow is 1 - 0.99^10, about ten percent. Then write why independence is the optimistic assumption and what correlates them in practice.
- Name the backpressure mechanism for one queue or one dependency in the design, a bounded queue with shedding or a concurrency limit, and write what the caller is told when it engages.
Deliverable: One write path with an idempotency design, a parameterised retry policy, and a written tail-latency calculation with its assumption named.
Day 5 — Scaling the hot path
- Choose cache-aside or write-through for one read path and write the staleness window each produces, then name the invalidation event and what the system does when that event is lost.
- Design against the stampede: either coalesce requests so only one recomputes a missing key, or refresh early with jittered expiry, and write why identical TTLs on keys populated in the same moment produce a synchronised expiry and a thundering herd.
- Shard one table by a key you choose, then answer the two questions that break the choice: which queries now require a scatter-gather, and what happens to the distribution when one tenant is a hundred times larger than the median.
- Write the cost of adding a node under plain modulo placement, where nearly every key moves, against consistent hashing, where roughly one key in n+1 moves, and state what virtual nodes are for.
Deliverable: A caching and sharding decision for one path, each with its failure mode and its rebalancing cost written beside it.
Day 6 — Keep the coding hand in, at the bar that applies to you
- Solve one medium problem in thirty minutes, then spend twenty more making it production-shaped: named invariants, validation at the boundary, and errors that distinguish a caller mistake from an internal fault.
- Write the tests you would require of a colleague's version of that function: one for empty input, one for the boundary, and one for the case the implementation is most likely to get wrong.
- Read a piece of your own code from six months ago and write the change you would ask for, phrased as you would actually phrase it in review.
Deliverable: One problem hardened to review standard, with its test list and one written review comment.
Day 7 — Defend it while being interrupted
- Run a forty-five-minute design mock with an interviewer briefed to change a requirement halfway, a tenfold traffic increase or a new strict consistency requirement, and to push on one number you estimated.
- Rehearse the two sentences a senior loop is listening for: naming the tradeoff you are choosing against and why, and saying what you would measure to learn that the choice was wrong.
- Prepare the design you regret: a real decision, the constraint that produced it, what it cost, and what you changed afterwards.
Deliverable: Mock notes recording how the design changed under the new requirement, plus a written account of one regretted decision.
Bounding a vague ticket before writing code
When the requirements were thin, the interesting part is how you fenced the problem off: the assumption you wrote down, who you got to confirm it, the narrow version you shipped first so the rest stayed cheap to change. Guessing and being right is luck. Guessing in writing, where someone could correct you, is method.
Are you comfortable working in an international environment?
Medium · behavioural and engineering judgement
Approach
- State the situation in two sentences and spend the rest on the reasoning.
- Give the blast radius: what could have broken, and what you measured.
- Name the disagreement and how you resolved it with evidence.
Follow-up
- How did you know your change caused the improvement?
- What would you do differently if you ran that again?
Is your level of English sufficient for daily professional communication?
Medium · behavioural and engineering judgement
Approach
- State the situation in two sentences and spend the rest on the reasoning.
- Pick a story where you made the decision, not one where you watched it.
- Close with what you would do differently, concretely.
Follow-up
- What would you do differently if you ran that again?
- How did you know your change caused the improvement?
How do you handle collaborative projects involving cross-functional teams?
Medium · behavioural and engineering judgement
Approach
- Pick a story where you made the decision, not one where you watched it.
- Give the blast radius: what could have broken, and what you measured.
- Name the disagreement and how you resolved it with evidence.
Follow-up
- What would you do differently if you ran that again?
- What did you decide not to do, and why?