cswork / libraryOpen practice workspace

Interview guides · AUO Mobility Solutions Corporation America

AUO Mobility Solutions Corporation America Software Engineer Interview Questions & Guide 2026

AUO Mobility Solutions Corporation America software engineer prep on partial failure: a timed-out request has an unknown outcome, not a failed one.

The 60-second brief

The Software Engineer role at AUO Mobility Solutions Corporation America is a pivotal position focused on driving innovation within the mobility sector. You will be responsible for developing, maintaining, and optimizing complex software solutions that integrate with AUO Mobility Solutions Corporation America’s advanced hardware platforms. Your work directly impacts the efficiency and reliability of mobility systems, contributing to the cutting-edge technology that defines the company’s reputation.

If the seat owns a service boundary, scope your preparation toward failure behaviour rather than topology. Retrying over an at-least-once channel produces duplicates by construction, so a retry policy is only as safe as the idempotency key underneath it.

Focus areas

  • Diagnose failures inside environments you cannot instrument
  • Bound credential lifetime by the engagement end date
  • Evolve schemas expand-contract across un-upgradable client deployments

Overview

This role requires a blend of technical precision and collaborative spirit. You will often work within specialized domains such as SMT (Surface Mount Technology) or FMS (Flexible Manufacturing Systems), ensuring that software architectures are robust and scalable. Whether you are working on HMI (Human-Machine Interface) technologies or core R&D software platforms, you are expected to deliver high-quality code that meets the rigorous standards of the automotive and mobility industries.

The environment is characterized by professional rigor and a focus on organic growth. You will find yourself working alongside experts who value technical depth and effective communication. The work is challenging, requiring you to be both a skilled individual contributor and a collaborative team member who can translate complex requirements into seamless digital experiences.

The AUO Mobility Solutions Corporation America interview process

AUO Mobility Solutions Corporation America candidates report 5 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

The person on this call usually cannot evaluate your code and does not need to. They write a short paragraph, and that paragraph is what a hiring manager skims when deciding who to put on your loop. So the test is not whether your work was hard, it is whether a non-engineer can repeat it correctly. Name systems by what they did rather than by their internal codename, give each project a shape (what was breaking, what you changed, what happened after), and keep the whole walkthrough near ninety seconds. Depth that cannot survive a paraphrase reads as vagueness.

What to demonstrate

  • Whether a non-engineer can restate your projects without distorting them, since their paraphrase is what travels to the hiring manager, not your sentences
  • Whether each project has a shape rather than a stack list: the failure or constraint, the change you made, the result and how it was measured
  • Whether you can say what was yours inside a team project without either inflating it or disappearing into the plural

How to prepare

  • Rewrite each headline project as two sentences with no internal system names and no acronyms outside your company, then say them to someone outside engineering and have them repeat them back. Fix whatever came back wrong
  • Attach one measured number to each project: the baseline, the change, and the window it was measured over. Where nothing was ever measured, say that plainly rather than reaching for a plausible percentage
  • Time the background walkthrough against a clock. If it runs past two minutes, compress the earliest role to a single clause and spend the recovered time on the most recent one

Technical Discussions

Input bounds are the part of the prompt most often skimmed, and they usually contain the answer. They tell you which complexity class is admissible, which narrows the search before you have thought about the problem itself. As a rough planning figure, a compiled language does on the order of 10^8 simple operations per second and an interpreted one roughly an order of magnitude less. So n up to about twenty admits enumerating subsets, a few thousand admits a quadratic pass, and a million admits neither: you need near-linear, or linear with a log factor. If the bounds are missing, ask for them.

What to demonstrate

  • Whether the approach is justified by the stated input size rather than by whichever pattern you recognised first
  • Whether you ask about the properties that change the algorithm: whether the input arrives sorted, whether duplicates occur, whether values are bounded integers, whether it all fits in memory
  • Whether you can name the bottleneck in your own solution and what would remove it, even when you deliberately leave it in place
  • Whether a claimed speedup is real, since memoising a recursion only helps when subproblems genuinely overlap and the state can be keyed cheaply

How to prepare

  • For each algorithm you rely on, write down the largest n it handles in roughly a second, then check two of those figures by timing them in the language you will actually type in
  • For two weeks, write one line naming your target complexity and the bound that justifies it before you write any code, then compare that line with what you ended up submitting
  • Practise the conversion backwards: given a required O(n log n), list the mechanisms that get you there (sorting, a heap, an ordered map, divide and conquer) and choose by what the problem needs to query, not by what you used last

Leadership Discussions

When a round has no standard shape, it is often there because something is still open: an area no earlier conversation reached, a round where the signal came out mixed, or a decision someone is not ready to make alone. Work out which by going back over what each earlier round actually covered rather than how it felt, and arrive able to give evidence on that point without being asked twice. Weak answers replay the loop's earlier material at the same depth. Strong ones go a level deeper and stay consistent with what you already said.

What to demonstrate

  • Whether your account of a project matches the one you gave earlier in the loop, since what you said before may be available to whoever runs this round
  • Whether you can go a level deeper on something already covered, reaching the decision and its alternatives rather than repeating the summary
  • Whether you state your own uncertainty accurately, including parts of a system you did not build and decisions you inherited, instead of claiming even ownership across all of it
  • Whether you can answer a question you handled poorly earlier by naming what you missed, rather than delivering a polished second version as if the first had not happened

How to prepare

  • Reconstruct the loop on one page: for each round, the questions you were asked and the answer you actually gave, not the better one you thought of afterwards. The gaps on that page are your best available guess at why this round exists.
  • Take the two claims you made earlier that carry the most weight and assemble the backing for each: the measurement, the date, what broke, the decision you would make differently now.
  • Write down the three facts about your work that must not drift between tellings, such as team size, timeline and your own role, and check your stories against that list rather than trusting recall under pressure

Panel Interviews

A day like this is several different games in a row, and the expensive mistake is carrying the previous one into the next room. Coding rewards narrow precision and finishing inside a timer. Design rewards breadth, stated assumptions and naming what you are deliberately not building. Behavioural rewards specificity about people and decisions. Candidates who over-engineer a coding problem they were supposed to finish, or who start sketching class hierarchies before anyone has agreed what the system has to do, are usually still playing the last round. Between rooms, name out loud which game the next one is.

What to demonstrate

  • Whether the coding round ends with something that runs and has been traced against a degenerate input, rather than an extensible design that was never finished
  • Whether a design discussion opens by agreeing on traffic shape, read-to-write ratio and what is allowed to be stale, instead of proceeding from an architecture you arrived with
  • Whether a behavioural answer names a person, a disagreement and what you did about it, rather than describing the system the story happened inside
  • Whether the opening habits still appear late in the day: restating the problem, asking for constraints, saying the plan before typing

How to prepare

  • Book three mocks of different types back to back on one afternoon and ask each interviewer afterwards which round you answered in the wrong mode
  • Write a three-line opening script per round type — coding: restate, name the approach and its cost, then type; design: ask for scale, read-write mix and what must not break; behavioural: name the person, the stakes and the decision — and run it off a card so the switch is mechanical rather than remembered
  • Practise coding with a timer you do not extend, stopping when it stops, so the trained reflex is to finish a correct solution rather than to keep improving one

Final Decision

Coding rounds mostly set a floor. They decide whether you clear the bar, not where you land on the ladder. Level tends to come out of the design discussion and the ownership stories, so the question worth auditing beforehand is whether the scope you describe matches the scope of the job. Work that stops at your own service, or a story whose hard part was writing the code rather than getting several people to agree on an interface, reads a level below where you think you are interviewing, and that gap is usually resolved downwards.

What to demonstrate

  • Whether the largest thing you describe owning ran end to end — the decision, the migration path, the rollout, and what you did when it went wrong — or stopped at the change you merged
  • Whether design answers include what you would not build, what you would defer, and what you would measure before committing, rather than only what the boxes are
  • Whether a disagreement in a story was settled with something checkable — a benchmark, a prototype, a written proposal — instead of by seniority or by waiting it out
  • Whether you can say which calls you made alone and which you escalated, and why the line sat where it did

How to prepare

  • Write your largest piece of owned work as a timeline of decisions — who decided what, when, and what you did when the plan broke — then delete every sentence whose subject is "we" and see how much survives
  • Take one system you know well and drill the migration answer: how old and new paths run side by side under live traffic, how you compare their outputs, what the rollback is once writes are going to both, and which step you would not automate
  • Map each line of the ladder in the job posting to a specific thing you have done, find the line you cannot support, and prepare the closest evidence you have plus an honest account of the gap

Where candidates lose points

Tenant context leaking across a connection pool

Setting the tenant on a connection and relying on it for the rest of the request looks correct in every test that runs one request at a time. Under transaction pooling the connection goes back to the pool with that session state still attached, and the next checkout, serving a different client, inherits it; row-level security then enforces the previous tenant's policy flawlessly and returns the wrong rows. It reproduces only under concurrency, which is the load profile your integration tests do not have. Use a transaction-scoped setting, and assert that the setting matches the request's tenant immediately before the first query rather than trusting that it was set correctly upstream.

Validating against a client sandbox and assuming production parity

Sandboxes typically carry smaller data, looser or absent rate limits, a schema version behind production, and sometimes synchronous behaviour where production is asynchronous. Code that passes there fails first in the client's production, during a change window you do not control and often cannot get a second one of. The mitigations are specific: assert the observed schema version at the boundary of every run, measure the production rate limit empirically rather than reading it from a document, and design the first production run to be a bounded, reversible slice rather than a full backfill.

Designing for a scale nobody asked for

Ask for request rate, data size, read-to-write ratio and expected growth, then size the simplest option first; one relational instance on current hardware covers a large share of real workloads. Reaching for shards, queues and a cache tier before any number has been quoted reads as pattern-matching rather than judgement.

Optimising an axis nobody named

Ask which resource is actually scarce here: wall-clock latency, throughput, memory footprint, cost per request, or engineering time. Shaving a constant factor off an in-memory step is wasted effort when the same function makes a blocking remote call inside the loop.

Questions & practice

Collapse connector retry attempts into logical jobs and failure counts

Easy · hash aggregation · idempotency · retries

You are given one UTC day of connector_run rows: (connector_id, idempotency_key, attempt_no, status, failure_class, records_read, records_applied), unique on (connector_id, idempotency_key, attempt_no), up to 5,000,000 rows in arbitrary order. Retries of one logical job share the idempotency_key; the job's outcome is the status of its highest attempt_no. Return, per connector_id: the number of logical jobs, a count of terminally failed jobs by failure_class, and the dedupe hit total, meaning sum(records_read - records_applied) over terminal-succeeded attempts only. One pass over the input; state your memory bound.

Approach

  1. Key a hash map on (connector_id, idempotency_key) and keep only the highest attempt_no seen so far plus that row's status, failure_class and record counts. One pass, O(n) time, O(d) space where d is the number of distinct logical jobs, not O(n).
  2. Treat a repeated (key, attempt_no) as corrupt input and raise, since the table's uniqueness constraint says it cannot happen; silently overwriting hides a double-insert in the runtime.
  3. Fold the d surviving entries into per-connector counters in a second pass over the map, not over the rows. The failure_class histogram is a small fixed-width map per connector because failure_class is an enum.
  4. Compute dedupe hits only from terminal-succeeded attempts. A failed attempt's records_read is work attempted, not work deduplicated, and adding it counts the same source records once per retry.
  5. If d does not fit in memory, partition the input by hash(connector_id, idempotency_key) into p files and aggregate each partition independently. Same O(n) total work, p sequential passes, memory traded for I/O; the partition function must use the full key or a job's attempts split across files.

Follow-up

  • A job's highest attempt is 'cancelled' but an earlier attempt succeeded. What is the job's outcome, and what does that say about who writes the cancel?
  • Produce this incrementally as runs land instead of as a daily batch: what state do you keep per key, and what happens when an attempt arrives out of order?
  • records_applied is written by your runtime, not by the client. What would you reconcile it against before anyone trusts the dedupe-hit number?

Compare release versions and report the live estate spread

Easy · parsing · version skew · ordering

Each environment row carries reported_release as TEXT: either NULL, or a version of the form MAJOR.MINOR.PATCH with an optional -rc.N suffix. Up to 400 environments. Ordering rule: compare major, then minor, then patch numerically; a -rc.N build precedes the same MAJOR.MINOR.PATCH with no suffix, and rc numbers compare numerically. Return the lowest and highest reported versions, the minor-version spread between them, and the count of environments whose reported_release is NULL. NULL means the environment's version is unknown, not that it converged.

Approach

  1. Parse each string once into a comparable tuple: (major, minor, patch, is_final, rc), where is_final is 1 for a plain release and 0 for a pre-release, and rc is 0 when absent. Tuple comparison then yields the stated order for free, including 3.10.0-rc.1 < 3.10.0. O(L) per string, O(n*L) total.
  2. Track min and max in one linear scan rather than sorting: O(n) time, O(1) extra space, and n is only a few hundred so the parse dominates either way.
  3. Fail loudly on a string that does not match the grammar instead of defaulting it to 0.0.0 or to the newest release. An unparseable reported_release is a reconciliation-ingest bug, and either default hides it in the direction that looks safe.
  4. Compute the minor spread only when min and max share a major version. Across majors the difference of minor numbers is meaningless, so return the pair of versions and flag the condition rather than a number.
  5. Report the NULL count as its own output line. Folding unknowns into 'converged' is how an install ends up running a release nobody knew was still live.

Worked solution (15 min)

  1. Input reported_release values: '3.9.0', '3.10.2', '3.10.0-rc.1', NULL, '3.9.0'.
  2. Parse to tuples: (3,9,0,1,0), (3,10,2,1,0), (3,10,0,0,1). The NULL is diverted to the unknown counter before parsing.
  3. Scan for min and max by tuple: min is (3,9,0,1,0) = 3.9.0, max is (3,10,2,1,0) = 3.10.2. Confirm 3.10.0-rc.1 sits strictly between them.
  4. Same major (3), so the minor spread is 10 - 9 = 1. Unknown count is 1.

Expected result: lowest='3.9.0', highest='3.10.2', minor_spread=1, unknown_count=1. A lexicographic sort of the same list would have returned highest='3.9.0' and lowest='3.10.0-rc.1'.

Verify it yourself

  • The parsed tuple for '3.10.0-rc.1' compares less than the tuple for '3.10.0' and greater than the tuple for '3.9.0'.
  • unknown_count plus the number of parsed versions equals the number of environment rows, so nothing was dropped.
  • Swapping the input order does not change any of the four outputs.
  • Adding a second environment already at '3.9.0' leaves the spread at 1, since spread is over distinct versions present, not over row counts.

Follow-up

  • Two majors are live at once. What does 'spread' mean now, and what single number, if any, would you still publish?
  • reported_release is a claim from the environment, and last_heartbeat_at may be days old for a client-managed install. At what staleness does this figure stop being usable?
  • How would you enforce the supported window when the rollout plan is built, instead of reporting a violation after it already exists?

Top-k longest connector runs from a stream too large to sort

Medium · heap · top-k · streaming

A day of connector_run rows sits in object storage: 200,000,000 rows, unsorted, far larger than memory, and streamable once. Each row has run_id, connector_id, started_at and finished_at, where finished_at is NULL for runs that never reached a terminal state. Return the 200 longest terminated runs by (finished_at - started_at), ties broken in favour of the lower run_id, plus the count of rows with a NULL finished_at. You may not sort the input or hold it in memory. State the time and space complexity you achieve.

Approach

  1. Keep a bounded min-heap of capacity k=200 ordered on (duration, -run_id). The root is then the weakest retained entry: smallest duration, and on a duration tie the largest run_id, which is exactly the one the stated tie-break should evict.
  2. Per row: if finished_at is NULL, increment a counter and skip; otherwise compute the duration and push when the heap holds fewer than k, else compare against the root and replace only if strictly better. O(n log k) worst case, O(k) space, and after the heap warms nearly every row costs a single comparison against the root.
  3. Reject the alternatives explicitly. A full sort is O(n log n) and at 2e8 rows means an external merge sort over hundreds of gigabytes to produce 200 rows. Quickselect is O(n) average but needs the whole array resident, which the constraint forbids.
  4. Handle the NULL as data, not as an edge case: depending on the language, subtracting from NULL either throws or yields a sentinel that lands at the top of the list. Count those rows and report them, because a spike of unterminated runs is its own signal.
  5. To parallelise, give each of p shard readers its own size-k heap and merge the p results. The union of per-shard top-k sets contains the global top-k, because an element beaten by fewer than k rows globally is beaten by fewer than k rows in its own shard, so the merge is exact at O(pk log k).

Follow-up

  • Now return the top 20 bindings by total run time instead of the top runs. What changes, and how many distinct keys must you hold to make it exact?
  • The key space is too large to hold an exact aggregate. What does a Misra-Gries summary with m counters actually guarantee about the counts it returns?
  • Rows now arrive continuously and you want a rolling one-hour top-k. Does the bounded heap still work, and if not, what breaks?

Return the latest attempt per connector idempotency key

Easy · distinct-on · indexing · idempotency

connector_run stores one row per attempt: run_id, connector_id, idempotency_key, attempt_no SMALLINT, status (queued, running, succeeded, failed, ambiguous, cancelled, abandoned), failure_class, watermark_from, watermark_to, started_at, finished_at, unique on (connector_id, idempotency_key, attempt_no). For one connector_id, write the query returning the highest-numbered attempt per idempotency_key, keeping only those keys whose latest attempt is not succeeded, ordered by watermark_from. Give the index that serves it, and say what the plan does without that index.

Approach

  1. Use DISTINCT ON (idempotency_key) with ORDER BY idempotency_key, attempt_no DESC. PostgreSQL requires the ORDER BY to lead with exactly the DISTINCT ON expressions, and that requirement is what makes the row picked deterministic rather than whichever the scan happened to reach first.
  2. Apply the not-succeeded filter outside, in a wrapping SELECT. Putting it in the inner WHERE changes the question from 'keys whose latest attempt failed' to 'keys that have any non-succeeded attempt', which reports a key that failed twice and then succeeded, and is the single most common wrong answer here.
  3. Know the portable alternative: row_number() OVER (PARTITION BY idempotency_key ORDER BY attempt_no DESC) in a subquery, filtered = 1 outside. A window function cannot be filtered in WHERE because WHERE is evaluated before the window, and PostgreSQL has no QUALIFY clause, so the subquery is mandatory rather than stylistic.
  4. Index (connector_id, idempotency_key, attempt_no DESC). With connector_id fixed by equality, the index supplies the exact order DISTINCT ON needs and the Sort node disappears. It does not reduce rows read: through PostgreSQL 17 there is no loose index scan, so the executor still walks every attempt in range rather than skipping to each next distinct key.
  5. Give the complexity honestly: without the index the plan is a scan plus an O(n log n) sort over the connector's attempts; with it, O(n) ordered reads and constant working memory, where n is attempts in range, not distinct keys. If the distinct keys are a tiny fraction of attempts, a recursive skip-scan is the next step.

Follow-up

  • attempt_no is not monotonic in time because a retry was queued out of order. Does your query still pick the right row, and which column should it actually order by?
  • How would you return the latest attempt per key across all connectors in one environment without the query degrading to a full sort?
  • An ambiguous latest attempt means the target may or may not have been written. What does this query feed, and should ambiguous be treated as failed here?

Soft-delete environments without breaking uniqueness or audit history

Easy · soft deletes · partial indexes · audit history

environment holds one row per provisioned deployment: environment_id, client_id, engagement_id, env_name, region, status, destroyed_at TIMESTAMPTZ NULL. Decommissioned environments stay readable for seven years, and env_name must be unique per client among live environments only, because a client may reuse a retired name. Design the constraint set and the history table, write the query that lists a client's live environments, and say what your design does about connector_run rows whose foreign key points at a destroyed environment.

Approach

  1. Keep the tombstone as a nullable timestamp rather than a boolean: destroyed_at carries when as well as whether, and enforce live-only uniqueness with a partial index, CREATE UNIQUE INDEX env_live_name ON environment (client_id, env_name) WHERE destroyed_at IS NULL, which indexes only the live subset and so stays small.
  2. Get the NULL semantics right before you pick a spelling. In a B-tree unique index two NULLs compare distinct by default, so plain UNIQUE (client_id, env_name, destroyed_at) enforces nothing at all among live rows — every live row is unique on the third column purely by virtue of being NULL. PostgreSQL 15's UNIQUE NULLS NOT DISTINCT (client_id, env_name, destroyed_at) does satisfy the requirement: the NULL live rows now compare equal and collide with each other, while retired rows carry distinct destroyed_at timestamps and stay distinct, so a name can be retired and reused any number of times.
  3. Prefer the partial index anyway, and be able to say why rather than asserting it. It indexes only the live subset — tens of rows per client — instead of seven years of tombstones; it needs no version floor; and it keeps destroyed_at out of the key, so it cannot be defeated by two rows retired at the same instant. That last case is reachable: now() is transaction-start time, so retiring a name, recreating it and retiring it again inside one transaction writes two identical destroyed_at values and raises a duplicate-key error under the constraint form. clock_timestamp() avoids it if you take that route.
  4. Put history in a separate append-only table written by an AFTER INSERT OR UPDATE OR DELETE trigger holding environment_id, changed_at, changed_by, op and the before and after row images. PostgreSQL has no system-versioned tables, so this is a trigger or a logical-decoding consumer, not a DDL flag, and the soft delete arrives at that trigger as an UPDATE rather than a DELETE.
  5. State the foreign-key consequence rather than leaving it implicit: a soft delete leaves every referencing connector_run row resolvable, which is exactly what the audit requirement wants, but it means no read is protected by the database. Give the predicate one home, a view or an RLS-backed view that every caller uses, instead of repeating destroyed_at IS NULL across hundreds of call sites.
  6. Size the retention: seven years of history plus tombstoned rows in the hot table. Partition the history table by month so expiry is a DETACH PARTITION rather than a DELETE that has to be vacuumed, and decide separately whether a residency-driven erasure request can be honoured against an append-only store at all.

Worked solution (20 min)

  1. Write the DDL with destroyed_at TIMESTAMPTZ NULL plus the partial unique index on (client_id, env_name) WHERE destroyed_at IS NULL.
  2. Insert two live rows sharing (client_id, env_name) and confirm the second is rejected; set destroyed_at on the first and confirm the insert now succeeds.
  3. Build the same table a second time under UNIQUE NULLS NOT DISTINCT (client_id, env_name, destroyed_at) and confirm it passes the same two tests, so you are choosing between two working designs rather than rejecting a broken one.
  4. Add environment_history (history_id, environment_id, changed_at, changed_by, op, row_before JSONB, row_after JSONB) and the AFTER trigger that populates it.
  5. Write the live-listing query: SELECT environment_id, env_name, region, status FROM environment WHERE client_id = $1 AND destroyed_at IS NULL ORDER BY env_name.

Expected result: A partial unique index covering only live rows, a separate append-only history table with its own partitioned retention, and a listing query whose predicate matches the index predicate exactly so the planner can use it.

Verify it yourself

  • Run retire-and-recreate three times for the same name: three tombstoned rows and one live row coexist without violating the index.
  • On the NULLS NOT DISTINCT variant, retire and recreate the same name twice inside one transaction and watch it fail on duplicate destroyed_at, since now() does not advance within a transaction; the partial index has no such failure mode.
  • EXPLAIN the listing query and see the partial index; widen the predicate to destroyed_at IS NULL OR destroyed_at > now() and watch the index drop out, because that predicate no longer implies the index's.
  • Soft-delete a row and confirm the history trigger fired; a DELETE-only trigger records nothing, which is the common way this audit trail ends up empty.

Follow-up

  • What stops the next endpoint from forgetting the destroyed_at IS NULL predicate, and does that mechanism survive a developer writing raw SQL?
  • The history table stores row images as JSONB and the base table gains a column. What do old rows now mean, and how do you read them six years later?
  • A residency erasure request requires a genuine hard delete for one client. What breaks, and what do you keep?

Can you explain your approach to debugging complex code in a production environment?

Medium

Approach

  1. State your assumptions explicitly before working the problem.
  2. Work from the requirement backwards to the design.
  3. Clarify what is being asked and what a complete answer contains.

Follow-up

  • What assumption would you test first?
  • How would you know your answer was wrong?

What are the core concepts of the C programming language?

Medium

Approach

  1. State your assumptions explicitly before working the problem.
  2. Clarify what is being asked and what a complete answer contains.
  3. Say what you would check first and why it is the highest-information step.

Follow-up

  • How would you know your answer was wrong?
  • What assumption would you test first?

How do you manage memory and performance in embedded or low-level systems?

Medium

Approach

  1. Clarify what is being asked and what a complete answer contains.
  2. Work from the requirement backwards to the design.
  3. Say what you would check first and why it is the highest-information step.

Follow-up

  • How would you know your answer was wrong?
  • What assumption would you test first?

Keep delivery running when the control services go unreachable

Hard · consistency vs availability · partition · credential ttl · bounded exposure

A network partition cuts one region's workers off from the Engagement and Entitlement Service and the Access Broker for 90 minutes. Residency forbids failing over to another region's copy of that data. Inside the region, connector runs hold cached credentials with not_after values minutes to hours out, and operator sessions are mid-task. The entitlement service fails closed by design. Decide what continues and what stops, give a stated exposure bound for each choice, and name the one thing that must never continue however long the partition lasts.

Approach

  1. Classify each operation by whether it widens authority, because that is what decides the answer. Continuing on a credential already minted extends access that was already granted; minting a new one grants access you currently cannot verify. A blanket fail-closed that stops both is as unconsidered as a blanket carry-on, and neither tells you what your exposure was afterwards.
  2. Let in-flight work continue on already-issued credentials. Their exposure is already bounded by not_after, which is the reason you capped it at issue, and the worst case is the longest outstanding not_after at the moment the partition began. That number is not fate: capping connector TTLs at the expected run duration rather than at the engagement window shrinks it in advance, which is the only time it can be shrunk.
  3. Refuse everything that widens authority - new mints, scope increases, the first run of a binding that has never run, an operator establishing new access. Return those as readable refusals naming the cause rather than as transport errors, or every operator and automated agent in the region spends 90 minutes guessing whether the system is broken or is saying no.
  4. Name the operation that must never continue: work against an engagement whose ends_on has already passed. Expiry is arithmetic over data every worker already holds and needs no network, so a partition is no excuse for it. This is the concrete payoff of never materialising an active flag - the one check that has to survive isolation is the one that is pure computation on a date.
  5. Time-box the degradation by construction: refuse to serve from a snapshot older than the staleness bound, so the window ends on its own rather than when somebody notices. On reconnect, reconcile rather than resume - re-read entitlements, revoke grants for engagements that closed during the window, and resolve runs that ended ambiguous against their dedupe keys, since a partition is the single most likely producer of ambiguous outcomes.
  6. State the cost plainly. This chooses availability for already-authorised work and consistency for every change of authority, and pays with a window in which a closure decided elsewhere is not yet honoured. That window is min(longest remaining TTL, staleness bound), it is a number you can put in front of a client, and it is shorter than the partition.

Worked solution (45 min)

  1. Write the region's operation inventory: for each operation, whether it widens authority, what it reads, and whether that read can be served from cached state.
  2. For each one, state the exposure if it proceeds on data 90 minutes stale, as a duration and a scope rather than as a risk rating.
  3. Simulate the partition with a firewall rule and run two bindings: one whose credential has 40 minutes left, one whose engagement ended yesterday.
  4. Restore connectivity and run the reconciliation, recording which grants are revoked, which runs are reclassified, and which watermarks are re-checked.

Expected result: The binding with a live credential finishes its range and then stops at expiry without entering a mint retry loop. The binding on the ended engagement refuses immediately and locally, citing ends_on, and never touches the network. Mints and scope changes refuse for the whole window. After reconnection, grants for engagements closed during the partition are revoked, ambiguous runs resolve against dedupe keys instead of blind retries, and the exposure actually realised matches the number predicted in step two.

Verify it yourself

  • Maximum exposure equals min(longest outstanding not_after, staleness bound) and is computed from credential_grant rather than estimated.
  • No code path can reach another region's entitlement data, asserted by a test on the region-scoped client rather than by policy.
  • Every refusal during the partition carries a distinct cause and is counted apart from client-side failures, so the blast radius is readable after the fact.

Follow-up

  • An engagement is terminated for cause during the partition. What is the real access window, and what would you change afterwards to shorten it?
  • Half the region's workers can reach the entitlement service and half cannot. What breaks in that case that a total partition would not have broken?
  • Your staleness bound is 10 seconds and the partition lasts 90 minutes, so delivery stops entirely. Is that bound right, and what evidence would change it?

Connector workers are OOM-killed every day and a half

Medium · memory leak · caching · heap analysis

Connector runtime workers are OOM-killed after 18 to 40 hours and a restart clears it. RSS grows monotonically at about 1.9 MB per completed run, flat regardless of how many records that run moved, and workers serving more distinct bindings do not grow faster than workers serving one. Live heap measured immediately after a forced full collection rises with the same slope as RSS. Traffic is unchanged week over week. Give the ordered diagnosis, the specific object you expect at the root, and the fix.

Approach

  1. Prove retention before hunting for it. RSS and the post-collection live set rising with the same slope means the objects are reachable, which is a leak in the ordinary sense. RSS rising while the live set stays flat would be allocator fragmentation or native buffers, and that is a different investigation with different tools.
  2. Find the unit the growth is keyed on by division rather than by guessing. 1.9 MB per completed run, flat across record counts and flat across distinct bindings, points at something retained exactly once per run. Per-record growth would implicate an accumulating buffer; per-binding growth would implicate a legitimate cache that is merely unbounded.
  3. Diff two heap snapshots an hour apart by retained size and walk the dominator tree to a single root. Expect a module-level dictionary whose entry count equals the number of runs completed since process start, which is the observation that confirms step two.
  4. Read the key and the value separately, because they are two different bugs. The key contains something run-scoped, an idempotency key or a correlation id, which is why the hit rate is zero and the map only ever grows. The value is megabytes rather than kilobytes because the compiled schema validator closes over the first parsed page, so every entry pins a client payload as well.
  5. Fix both halves. Re-key on (connector_id, source_schema_version), whose cardinality is in the hundreds and stable. Cap the map with a fixed-size LRU so an unforeseen key can never make it unbounded again. Stop the closure capturing the page, so an entry costs a compiled schema rather than a schema plus a payload.
  6. Make the class of bug visible next time: export entry count and hit rate for every process-level cache. A zero hit rate is the unmistakable signature of a run-scoped key and appears within minutes, whereas this one took 18 hours to present itself as an OOM kill.

Follow-up

  • Those retained pages are client payloads sitting in a long-lived shared process. What does that mean beyond memory?
  • A worker killed mid-run leaves connector_run rows in 'running'. Who reconciles them, and how do you avoid applying that range twice?
  • Same symptom, but the post-collection live set is flat and only RSS climbs. What are you looking at instead, and which tool do you reach for?

Suggested practice checklist

For a candidate senior enough that the loop turns on design and judgement rather than on whether the coding round gets finished. Five days build one system properly and then stress it; coding gets a single maintenance day, on the assumption that the risk at this level is an unexamined tradeoff rather than a missed algorithm.

Day 1 — Numbers before diagrams

  • Build your own reference card of the figures you will re-derive all week: bytes for a realistic record, requests per second implied by a given daily active count, and the storage that a year at a given write rate produces. Derive each one rather than copying it, because the derivation is what survives a follow-up.
  • Turn one product statement into capacity requirements. From ten million daily users at four writes and forty reads each, state the peak-to-average factor you are assuming and why, then produce peak write QPS, peak read QPS and a year of storage.
  • Write the two numbers whose order of magnitude changes the design, the read-to-write ratio and the working-set size against memory per node, and state the threshold at which each one flips your answer.

Deliverable: A one-page numbers card and one worked capacity estimate with every assumption written down.

Day 2 — One system, from requirements to schema

  • Spend the first ten minutes producing only functional requirements, non-functional targets with numbers attached, a p99 latency, a durability expectation, a consistency requirement, and an explicit out-of-scope list.
  • Define the interface before the boxes: the three or four endpoints, their parameters, what each returns, and which of them are idempotent.
  • Write the data model, then write the single access pattern that justifies it, and state what the schema would have to become if the dominant access pattern were the other one.

Deliverable: One design carried to endpoint-and-schema depth, with non-functional targets expressed as numbers and a written out-of-scope list.

Day 3 — The consistency you are actually buying

  • Write out what a client sees under asynchronous replication when its write commits on the leader and its next read is served by a lagging follower, then write the two fixes, pinning that session's reads to the leader for a bounded window or carrying a version token the replica must reach, and the cost of each.
  • Work the quorum arithmetic on paper for N of three with W and R of two, and separate what R + W > N does guarantee, that any read set intersects any write set, from what it does not: on its own it is not linearizability, and a sloppy quorum that accepts writes on nodes outside the preference list breaks even the intersection.
  • Take two storage choices with different defaults, a single-leader relational store committing synchronously and a quorum-replicated store that converges eventually, and write the specific product behaviour that would be wrong under each, rather than a general statement about which is stronger.

Deliverable: A page separating what quorum overlap guarantees from what it does not, with one concrete product misbehaviour attached to each gap.

Day 4 — Failure is the design

  • For one write path, work through the case where the client times out after the server has already committed, then design the idempotency key: who generates it, how long it is retained, and what the duplicate request returns.
  • Express the retry policy as parameters rather than as a word: maximum attempts, base delay, backoff factor, jitter, and which error classes are retried at all. Then state why retrying a non-idempotent write without a key is a correctness bug and not merely waste.
  • Compute the fan-out effect on tail latency. If a request waits on ten backends and each independently exceeds its p99 one percent of the time, the chance at least one is slow is 1 - 0.99^10, about ten percent. Then write why independence is the optimistic assumption and what correlates them in practice.
  • Name the backpressure mechanism for one queue or one dependency in the design, a bounded queue with shedding or a concurrency limit, and write what the caller is told when it engages.

Deliverable: One write path with an idempotency design, a parameterised retry policy, and a written tail-latency calculation with its assumption named.

Day 5 — Scaling the hot path

  • Choose cache-aside or write-through for one read path and write the staleness window each produces, then name the invalidation event and what the system does when that event is lost.
  • Design against the stampede: either coalesce requests so only one recomputes a missing key, or refresh early with jittered expiry, and write why identical TTLs on keys populated in the same moment produce a synchronised expiry and a thundering herd.
  • Shard one table by a key you choose, then answer the two questions that break the choice: which queries now require a scatter-gather, and what happens to the distribution when one tenant is a hundred times larger than the median.
  • Write the cost of adding a node under plain modulo placement, where nearly every key moves, against consistent hashing, where roughly one key in n+1 moves, and state what virtual nodes are for.

Deliverable: A caching and sharding decision for one path, each with its failure mode and its rebalancing cost written beside it.

Day 6 — Keep the coding hand in, at the bar that applies to you

  • Solve one medium problem in thirty minutes, then spend twenty more making it production-shaped: named invariants, validation at the boundary, and errors that distinguish a caller mistake from an internal fault.
  • Write the tests you would require of a colleague's version of that function: one for empty input, one for the boundary, and one for the case the implementation is most likely to get wrong.
  • Read a piece of your own code from six months ago and write the change you would ask for, phrased as you would actually phrase it in review.

Deliverable: One problem hardened to review standard, with its test list and one written review comment.

Day 7 — Defend it while being interrupted

  • Run a forty-five-minute design mock with an interviewer briefed to change a requirement halfway, a tenfold traffic increase or a new strict consistency requirement, and to push on one number you estimated.
  • Rehearse the two sentences a senior loop is listening for: naming the tradeoff you are choosing against and why, and saying what you would measure to learn that the choice was wrong.
  • Prepare the design you regret: a real decision, the constraint that produced it, what it cost, and what you changed afterwards.

Deliverable: Mock notes recording how the design changed under the new requirement, plus a written account of one regretted decision.

The estimate and the day you knew

A slipped date is only a bad story if you sat on it. What matters is what you believed when you gave the number, the signal that told you it was wrong, how many days passed before you said so, and what you cut rather than asking for more time. Scope you defended counts as much as scope you dropped.

Why are you looking to leave your current organization?

Medium · behavioural and engineering judgement

Approach

  1. Give the blast radius: what could have broken, and what you measured.
  2. Name the disagreement and how you resolved it with evidence.
  3. Pick a story where you made the decision, not one where you watched it.

Follow-up

  • What would you do differently if you ran that again?
  • What did you decide not to do, and why?

What interests you specifically about the mobility and manufacturing technology space?

Medium · behavioural and engineering judgement

Approach

  1. Give the blast radius: what could have broken, and what you measured.
  2. Name the disagreement and how you resolved it with evidence.
  3. Close with what you would do differently, concretely.

Follow-up

  • How did you know your change caused the improvement?
  • What would you do differently if you ran that again?

Describe a time you worked on a cross-functional team to solve a technical bottleneck.

Medium · behavioural and engineering judgement

Approach

  1. Name the disagreement and how you resolved it with evidence.
  2. Pick a story where you made the decision, not one where you watched it.
  3. State the situation in two sentences and spend the rest on the reasoning.

Follow-up

  • What did you decide not to do, and why?
  • What would you do differently if you ran that again?