DataAnnotation Software Engineer Interview Guide

The 60-second brief

DataAnnotation hires Software Engineers; this guide collects what candidates report about the process.

This guide is scoped to a Software Engineer candidate at DataAnnotation.

Focus areas

• Python
• SQL
• Evaluation of AI model outputs (response ranking)

The DataAnnotation interview process

DataAnnotation candidates report 4 rounds over 3-5 weeks. The stages below are what candidates describe, not a published process.

Profile Creation

Submit background details, programming languages, and portfolio links.

What to demonstrate

• Submit background details, programming languages, and portfolio links
• Depth in Python

How to prepare

• Work Python until you can explain it without notes
• Work SQL until you can explain it without notes

Starter Coding Assessment

Complete an untimed coding assessment in your local environment.

What to demonstrate

• Complete an untimed coding assessment in your local environment
• Depth in Python

How to prepare

• Work Python until you can explain it without notes
• Work SQL until you can explain it without notes

Specialized Onboarding Modules

Access paid onboarding modules including coding qualifications and training.

What to demonstrate

• Access paid onboarding modules including coding qualifications and training
• Depth in Python

How to prepare

• Work Python until you can explain it without notes
• Work SQL until you can explain it without notes

Advanced Qualifications

Pass advanced qualifications to unlock access to high-paying projects.

What to demonstrate

• Pass advanced qualifications to unlock access to high-paying projects
• Depth in Python

How to prepare

• Work Python until you can explain it without notes
• Work SQL until you can explain it without notes

Where candidates lose points

Prioritize Written Explanations

Never rush your written justifications. A correct piece of code with a vague, single-sentence explanation will often be rejected. Explain why you chose a specific data structure and how your logic processes the input step-by-step.

Double-Check Your Output

Treat the output specifications like a strict contract. If a problem asks you to output a parsed secret message, verify that there are no trailing whitespaces, incorrect line breaks, or missing characters.

Read Instructions Completely Before Starting

Qualification tests often contain detailed instructions outlining prohibited approaches, required libraries, or formatting constraints. Read the document twice before writing a single line of code.

DataAnnotation enforces a strict zero-tolerance policy regarding AI-assisted plagiarism

Submitting AI-generated responses for your evaluations or code assessments will result in immediate termination from the platform.

Questions & practice

Diff a projection against the primary without per-row point reads

Hard · reconciliation · range hashing · throttling

The listing projection has drifted and some rows show a stale version. The primary holds 40,000,000 resource rows across 12,000 tenants while serving 1,200 writes and 14,000 reads per second. The obvious repair, reading each resource row and comparing its version against the projection, is correct and would eventually finish. Explain precisely why it is unacceptable here, then give a diff that finds the differing rows, state its complexity, and make it safe to run against a live primary. Replication lag is usually under 100 ms and is not bounded.

Approach

1. Quantify the naive cost rather than calling it slow: 40,000,000 point reads at even 0.5 ms each is over five hours serialised, and the only lever is concurrency, which is exactly what you cannot spend. The primary's pool is sized for the write path, and 40,000,000 random reads evict the buffer cache that sustains the 85 percent cache hit rate, so the audit degrades the system it is auditing.
2. Replace random access with one ordered pass per side. Both sides can be read in (tenant_id, resource_id) order, which is a sequential scan on each and a merge join in O(n) time and O(1) memory. For a dense diff that is the whole answer, and it reads the primary once instead of 40,000,000 times.
3. For the expected sparse case, compare range hashes instead of rows: partition the key space, compute per range an order-independent aggregate over hash(resource_id, version), compare aggregates, and descend only into ranges that differ. With d differing rows and branching factor B, at most d ranges mismatch per level, so the drill-down examines O(d log_B(n/d)) ranges and reads full rows only in mismatching leaves.
4. Aggregate with a sum modulo 2^64 or a multiset hash, never XOR. XOR is order-independent but self-cancelling, so two rows wrong in the same way, or a row duplicated on one side, leave the range aggregate matching and the range is declared clean.

Follow-up

• The diff reports 900 stale rows. How do you decide between patching those rows and rebuilding the projection from resource_revision?
• Same job, but the projection lives in a search index that cannot be scanned in key order. What changes?

Merge partitioned event streams into one ordered feed with bounded lateness

Hard · k-way merge · watermarks · out-of-order streams

The read-model service consumes 64 log partitions carrying about 4,000 events per second in total. Each partition is ordered within itself, but partitions drift by up to 30 seconds, and the activity feed must present a tenant's events in occurred_at order. Produce the merge. State its complexity, the buffer it requires in events and in bytes, what happens when one partition is idle, and what you do with an event that arrives after you have already emitted its position. Payloads average 1 KB.

Approach

1. Merge with a min-heap over the 64 partition heads keyed on (occurred_at, event_id): O(log P) per event and O(n log P) overall. The tie-break on event_id is what makes the output deterministic when two partitions carry the same millisecond, which matters because the feed is paginated and a non-deterministic order reorders pages under the reader.
2. Emitting the heap head is only correct once every partition has produced everything up to that timestamp, so the emit condition is a watermark: the minimum across partitions of the highest occurred_at seen, less the allowed lateness. Events are held until the watermark passes them, which is what turns individually ordered streams into a jointly ordered one.
3. Size the buffer from the lateness rather than guessing: 4,000 events per second times 30 seconds is 120,000 buffered events, and at 1 KB each about 120 MB of heap. That number is the real price of the ordering guarantee and belongs in front of whoever asked for it.
4. Handle the idle partition explicitly, because it fails the feed rather than corrupting it: a partition with no traffic never advances its own maximum, so the watermark freezes and output stops entirely. Either every partition emits a periodic idle marker carrying the broker's current time, or the watermark falls back to wall clock for a partition silent beyond a threshold.

Follow-up

• The lateness budget is raised to five minutes. What is the new buffer, and what besides memory changes?
• The consumer restarts. Where does it resume from, and what does the feed look like for the first 30 seconds?

Archive a resource graph without breaking live references or recursing

Medium · graph traversal · topological order · tenant isolation

Resources reference other resources within a tenant; for the largest tenant the reference table holds up to 2,000,000 nodes and 8,000,000 edges. Archiving a resource must archive everything reachable from it that nothing outside the set still references, refuse when a live external referrer exists, and terminate when references form cycles, which they legitimately do. Produce the archive order and the refusal list, targeting O(V+E). Say what stops the traversal crossing a tenant boundary, and why recursion is the wrong control structure at this size.

Approach

1. Load the subgraph with the tenant predicate on both endpoints of the edge, not only on the side you started from. Scoping the left table alone is the classic cross-tenant leak: one mis-entered edge then pulls another tenant's resources into the traversal and, worse, into the archive.
2. Traverse iteratively with an explicit stack. A 2,000,000-node graph can hold a chain deep enough to exhaust a native stack in the low tens of thousands of frames, and that failure is a process crash rather than an error you can return.
3. Treat cycles as data rather than corruption: compute strongly connected components with Tarjan in O(V+E) using its own explicit stack, then condense. The condensation is a DAG, so a topological order over it gives the archive order, and every member of a component archives in one transaction because no order within a cycle is valid.
4. Decide refusals with reverse edges. A candidate is archivable only if every in-edge originates inside the candidate set, so build the transpose or count in-degrees restricted to the visited set, and emit each blocked resource with the id of the external referrer, which is the only part of the answer an operator can act on.

Follow-up

• The graph is read in one query and the archive writes a minute later. What can change in between, and how do you make the write safe?
• The candidate set is 400,000 resources. Is that one transaction, and if not, what does a half-finished archive look like to a reader?

Keep soft-deleted accounts from blocking re-registration

Easy · soft delete · unique constraints · engine differences

app_user holds user_id, tenant_id, email CITEXT, password_hash (NULL for SSO principals), email_verified_at, auth_version, status ('invited','active','suspended','deactivated'), created_at, updated_at, deleted_at. Two live accounts for one address inside a tenant must be impossible, but an address freed by a soft delete must be reusable, and the same tenant may delete and re-register it repeatedly. Write the uniqueness DDL for PostgreSQL 16, then the equivalent for MySQL 8 where partial indexes do not exist, and say what each permits once three deleted rows already hold that address.

Approach

1. Start from what is actually unique: not (tenant_id, email), but (tenant_id, email) among live rows. PostgreSQL says that directly — CREATE UNIQUE INDEX app_user_live_email ON app_user (tenant_id, email) WHERE deleted_at IS NULL. A full constraint over the same two columns burns the address permanently the first time someone deletes an account.
2. Keep case-insensitivity in the type or the index, never in the application: CITEXT as given, or UNIQUE (tenant_id, lower(email)) as an expression index where the extension is unavailable. A case-sensitive unique column is exactly how two accounts for one human appear.
3. For MySQL 8 the predicate has to move inside the key: add a discriminator column that is a constant 0 while the row is live and is set to user_id on delete, with UNIQUE (tenant_id, email, deleted_marker). Live rows share the constant and still collide; deleted rows differ from each other and stop colliding.
4. State the NULL variant and its dependency: leaving the marker NULL for deleted rows also works, because a unique index treats NULLs as distinct — true in MySQL, and true in PostgreSQL only under the default NULLS DISTINCT, which PostgreSQL 15 lets you reverse. Check the polarity against the three existing deleted rows: constant-on-live is what preserves the collision you want, and reversing it silently admits duplicate live accounts.

Follow-up

• A deleted account re-registers with the same address the next day. Do the old resource rows follow the new user_id, and how does the API keep the two principals apart?
• How do you honour an erasure request while resource_revision.actor_user_id still references this table?

Find version gaps and relay lag with window functions

Medium · window functions · event tables · frame semantics

outbox_event holds event_id, aggregate_type, aggregate_id, aggregate_version, event_type, payload, status ('pending','published','dead'), attempts, created_at, published_at. A projection is missing rows and you must decide whether the relay skipped events or the consumer dropped them. Write three queries over the last seven days: one listing every aggregate_id whose published aggregate_version sequence has a hole, one giving per-day counts with a running total, and one returning the newest published event per aggregate. For each, say where the window function is evaluated relative to WHERE and LIMIT. PostgreSQL 16.

Approach

1. Gaps: compute lead(aggregate_version) OVER (PARTITION BY aggregate_id ORDER BY aggregate_version) in a subquery, then filter next_version <> aggregate_version + 1 in the outer query. Window functions are evaluated after WHERE, GROUP BY and HAVING and before the outer ORDER BY and LIMIT, so the predicate cannot sit in the same WHERE clause and PostgreSQL 16 has no QUALIFY.
2. Say what the seven-day filter does to the answer: it truncates every partition, so the first row per aggregate has no predecessor inside the window and a hole spanning the boundary is invisible. Widen the window, or join to resource.version as the authority for the true maximum.
3. Running total: SELECT date_trunc('day', created_at) AS d, count() AS n, sum(count()) OVER (ORDER BY date_trunc('day', created_at) ROWS UNBOUNDED PRECEDING). An aggregate inside a window call is legal because grouping runs before windowing. The grouping key is unique per row here so ROWS and RANGE agree, but write the frame anyway — over ungrouped rows with tied timestamps the default RANGE frame pulls in every peer row and the total jumps.
4. Newest per aggregate: DISTINCT ON (aggregate_id) ... ORDER BY aggregate_id, aggregate_version DESC is the cheap PostgreSQL-only form when an index matches that order; row_number() OVER (PARTITION BY aggregate_id ORDER BY aggregate_version DESC) = 1 is the portable form and needs a subquery for the same evaluation-order reason as the gap query.

Follow-up

• Relay failover redelivers events. Does a duplicate break the gap query, and how would you detect one from this table alone?
• Turn the gap check into a continuous monitor rather than a query someone runs after an incident. What does it watch?

Convert the listing endpoint from offset pages to stable cursors

Medium · pagination · cursors · index access path

GET /v1/resources returns a tenant's resources newest-updated first, today with page and per_page, backed by index (tenant_id, status, updated_at DESC, resource_id DESC). Callers are a browser feed and a nightly sync job that walks every page. Users report items appearing twice or vanishing between pages, and page 400 is slow. Design the cursor contract: what the cursor contains and how it is encoded, the exact WHERE and ORDER BY, what happens when a row's updated_at changes mid-walk, how a client detects the end, and what the sync job does when a cursor is rejected.

Approach

1. Separate the two defects, because they have different fixes. Cost: OFFSET n makes the engine produce and discard n rows, so price grows with page depth rather than page size and page 400 pays for 400 pages of work. Correctness: while the set shifts, rows cross the offset boundary and are skipped or repeated, and nothing in the response lets the client detect it.
2. Write the seek: WHERE tenant_id = $1 AND status = $2 AND (updated_at, resource_id) < ($k, $id) ORDER BY updated_at DESC, resource_id DESC LIMIT n. The tie-break is not decoration - updated_at is not unique, and two rows sharing a timestamp across a page boundary reintroduce exactly the skip this was adopted to remove. PostgreSQL seeks the composite index on the row-value comparison directly; on an engine that does not optimise a row constructor, expand it into the equivalent OR form or the plan quietly degrades to a scan.
3. Encode the cursor as an opaque token carrying the sort key, the id, and a fingerprint of the filter and sort order, signed or at minimum validated. A cursor replayed against a different sort or filter must be a 400 with its own code, not a silently wrong page - the sync job cannot notice the difference otherwise.
4. State the guarantee precisely rather than generously. Keyset is stable against inserts and deletes elsewhere in the set, because the position is a value and not a count. It is not a snapshot: updated_at is mutable, so a row that is edited during the walk re-sorts and may be seen twice or not at all. If the sync job needs exactly-once coverage, order on an immutable key such as (created_at, resource_id), or walk resource_revision by revision_id and treat updated_at as data.

Follow-up

• The client wants a total count and the ability to jump to page 400. What can you honestly offer instead, and what does each option cost?
• How do you paginate backwards, and what does that require of the index?

Choose what to break when replication lag reaches forty seconds

Hard · replication lag · read-after-write · consistency vs availability

Reads are served from two replicas: 14k requests/second, about 85% absorbed by cache, so roughly 2.1k reads/second reach the database. Writes go to the primary at 1.2k/second. A tenant's backfill drives replication lag from under 100 ms to 40 seconds and it is still climbing. Sessions that have just written are pinned to the primary. Decide, endpoint class by endpoint class, whether to serve stale, fail, or route to the primary, and justify each choice with the load it adds to the primary. Then state what you would have built beforehand.

Approach

1. Establish blast radius before cause, because mitigation and diagnosis have different deadlines. The decisive arithmetic is what happens if the database reads move to the primary: 2.1k reads/second on top of 1.2k writes/second roughly triples its operation count, on the node already absorbing the backfill that caused this. Reads and writes are not equal in cost, so treat that as an argument against a blanket move rather than as a capacity model - but it is enough to rule out routing everything to the primary.
2. Classify endpoints by what staleness costs, not by how important they feel. Reads whose staleness is invisible - listings, search, counters - stay on the replica and return the watermark so the client can tell. Reads that immediately follow that same session's write keep their primary pin, which is a small bounded slice of traffic rather than the whole 2.1k/second. Reads that feed a decision with a side effect - authorisation, quota, the read half of a read-modify-write - must not be stale at all, because a 40-second-old permission row is the stale-permission failure wearing a different costume; those go to the primary or fail.
3. Shed instead of queueing. If the must-be-fresh class alone exceeds the primary's headroom, refuse its lowest-value slice with 503 and a retry-after. A request queued behind a saturated primary holds a connection for a client that has already given up, and the retry storm that follows is what turns degradation into an outage. Bound the connection pool per role so the read fallback cannot consume the write path's connections - that bulkhead is the single decision that determines whether writes survive the next ten minutes.
4. Attack the cause in parallel, since it is the one thing that can be stopped. The backfill is the load generator. A backfill that reads replication lag as its throttle signal and pauses above a threshold would have made this a non-event, with batch sizes small enough that each batch's write volume is a fraction of what a replica can apply per second. That is most of the answer to what should have existed beforehand.

Follow-up

• Lag returns to normal in nine minutes. Which mitigation do you remove first, and which one stays permanently?
• A user reports their change did not save, and the write committed. Trace the path that produces that report and name the signal that would have shown it before the report arrived.

One customer endpoint stalls deliveries to every other destination

Medium · bulkhead · timeouts · saturation

The egress service delivers about 1.5k webhooks/second across 40,000 destinations, with a per-destination concurrency cap of 4 and a 10-second connect-plus-read timeout. Throughput falls to 300/second, queue depth climbs, and p99 delivery latency for unaffected destinations goes from 200 ms to minutes, while the error rate barely moves. One tenant holds 900 destination rows whose URLs share a hostname that now answers in 9.5 seconds. Explain the mechanism with the arithmetic, then give the containment in the order you would apply it.

Approach

1. Look at saturation before errors. A flat error rate with collapsing throughput says nothing is failing, things are waiting, so the first signal to pull is in-flight request count or pool wait time rather than the error counter. This is the distinction that decides the whole investigation.
2. Group in-flight work by resolved host, not by destination id. The cap is keyed per destination row, so 900 rows sharing one hostname buy 3,600 concurrent slots against a single host, each held for 9.5 seconds. The bulkhead was never a bulkhead for that host, and grouping by the wrong dimension is why the dashboard looked healthy.
3. Do the arithmetic in both directions. Required concurrency is arrival rate times latency, so 1.5k/second at 200 ms needs about 300 in flight, which is entirely consumed by 3,600 slow slots; conversely whatever concurrency is left sustains rate equals concurrency divided by 9.5 seconds, which is the 300/second you are seeing. Matching both numbers is what promotes this from a plausible story to the mechanism.
4. Explain why the circuit breaker never helped. It opens on consecutive failures, and a 9.5-second response inside a 10-second timeout is a success. Slow is not failing, so an error-rate breaker cannot see this; you need a slow-call ratio, a deadline propagated from the caller's remaining budget, or a concurrency limiter.

Follow-up

• The host recovers to 80 ms. How long does the queue take to drain, and what does the drain do to the recovered host?
• Where should the 10-second timeout number actually come from?

Suggested practice checklist

Built from the rounds and topics DataAnnotation candidates report.

Day 1 — Map the DataAnnotation loop

• Write out the reported sequence: Profile Creation, Starter Coding Assessment, Specialized Onboarding Modules, Advanced Qualifications.
• For each round, write one sentence on what it is judging, from the description above, and mark the one you are least ready for.

Deliverable: A one-page map of the 4 reported rounds, with the weakest marked.

Day 2 — Work Python

• Spend the session on Python, which DataAnnotation candidates report being tested on.
• Write one worked example in Python and time yourself on it.

Deliverable: One timed worked example in Python.

Day 3 — Work SQL

• Spend the session on SQL, which DataAnnotation candidates report being tested on.
• Write one worked example in SQL and time yourself on it.

Deliverable: One timed worked example in SQL.

Day 4 — Work Evaluation of AI model outputs (response ranking)

• Spend the session on Evaluation of AI model outputs (response ranking), which DataAnnotation candidates report being tested on.
• Write one worked example in Evaluation of AI model outputs (response ranking) and time yourself on it.

Deliverable: One timed worked example in Evaluation of AI model outputs (response ranking).

Day 5 — Consolidate

• Re-work the problem you got wrong earliest in the week, from scratch, without looking at your previous attempt.

Deliverable: A second, cleaner solution to the problem you got wrong first.

Day 6 — Rehearse your own examples

• Prepare three examples from your own work where you made the decision, each with the outcome you can quantify.

Deliverable: Three examples written out, each with a number attached.

Day 7 — Dry run for DataAnnotation

• Run one full mock under time, then write down the two questions you most want to ask your interviewers.

Deliverable: A completed timed mock and two questions to ask.

Behavioural preparation

Behavioural rounds judge the decision you made and what it cost.

Turn a code review disagreement into a decision

Easy · code review · optimistic concurrency · communication

A colleague's change updates a row with UPDATE resource SET version = version + 1 WHERE resource_id = $1 AND version = $2 and treats an affected-row count of zero as a successful no-op. You read that as a silently lost update; they think returning 200 is friendlier to clients than returning a conflict. Describe how you have handled a review disagreement of this shape: what goes in the comment, when you leave the thread, and who decides. Then write the comment you would leave here, in under 80 words.

Approach

1. Sort the disagreement before writing anything. A silently discarded write is a correctness claim about data; the choice between 409 and 412 is taste. Only the first justifies blocking a merge, and saying which one you are doing is most of the value of the comment.
2. Make the claim reproducible in the comment itself with an interleaving rather than a principle: A reads version 7, B reads version 7, B commits version 8, A's predicate matches zero rows, A is told it succeeded and A's edit is gone.
3. Offer the alternative with its cost attached: return 409 carrying the current version and the revision that won, so the client can re-read and re-apply. Note that automatic retry is not the fix, because a retry re-reads the winner's state and reapplies an intent formed against data that no longer exists.
4. Apply an escalation rule you can state: two round trips on the thread, then a call, and the service's owner decides rather than the reviewer. A reviewer who cannot be overruled is a bottleneck with extra steps.

Follow-up

• Where would you put the test that fails if someone reintroduces the swallowed zero rowcount?
• The author says clients cannot handle a 409. How do you check whether that is true?

Unblock an engineer without taking the keyboard

Easy · mentoring · leases · at-least-once

A teammate has spent two days on a job handler that occasionally writes duplicate rows. They are certain the queue is delivering twice by mistake. You suspect a lease expiring under a slow handler, so the job is running concurrently with itself. Describe how you have unblocked someone in this position: what you asked before offering a hypothesis, what you showed them rather than told them, and what you left them owning. Then say what you would do if their theory turned out to be the right one.

Approach

1. Ask before diagnosing, and ask for things answerable from data they already have: the attempt count on the job rows that produced duplicates, the handler's observed duration against its lease expiry, and whether the duplicate rows share a natural key that a unique constraint could have caught.
2. Teach the shape rather than the answer. A lease cannot distinguish a dead worker from a slow one, so a handler that outruns its lease is running twice by design, and deploys deliver the other half by killing handlers mid-run on every rollout. Both of their candidate theories produce identical duplicate rows, which is why the evidence has to come from timings rather than from argument.
3. Hand over a checklist they execute: a natural key on every write the handler performs so the second copy collides rather than appends, the record of intent written before any external effect, a lease heartbeat while running, and the metric that shows it working.
4. Keep ownership with them deliberately. Pair on the first write, then step back; if you finish it yourself you have closed one ticket and left the same person stuck on the next redelivery.

Follow-up

• How would you distinguish a genuine double-delivery from a lease expiry using only the data already stored?
• Their handler calls an external endpoint before recording that it did. What do you tell them to change first?

Estimate work you have never done and defend the range

Hard · estimation · backfills · expand-contract

You are asked to estimate a change you have never attempted: add a column to a 100-million-row table, populate it, move reads across, and drop the old shape. Give a range with the assumptions that generate it, including batch size, the signal your backfill throttles on, and wall-clock hours, and name the three unknowns that would move the number most. Then describe a real estimate you gave under comparable ignorance: how you expressed its uncertainty, what you committed to, and how wrong you turned out to be.

Approach

1. Decompose into independently deployable steps before estimating anything: add the column nullable, write both shapes, backfill in batches, verify, move reads, stop writing the old shape, drop it. That is four deploys spread over days, and the calendar estimate is dominated by them rather than by the loop's runtime.
2. Do the arithmetic aloud for the part that has arithmetic in it: batch size times number of batches times per-batch duration, at a write rate the primary can absorb alongside roughly 1.2k writes per second of production traffic. The loop is throttled by replication lag and lock waits, not by how fast it can issue statements.
3. Price the schema step by its lock rather than its statement duration. In PostgreSQL an ALTER TABLE taking ACCESS EXCLUSIVE waits for every open transaction on that table while later queries queue behind it, so a millisecond change issued during a thirty-second analytics query stalls that table for thirty seconds. Adding a nullable column with a non-volatile default avoids a rewrite from version 11; a new index wants CREATE INDEX CONCURRENTLY, which cannot run inside a transaction block and leaves an invalid index behind if it fails.
4. Express the answer as a range whose endpoints each trace to a stated assumption, then name the cheapest experiment that collapses it, which is almost always running one real batch against the real table and multiplying.

Follow-up

• How do you verify the backfill genuinely finished, given rows written by production traffic while it ran?
• Where does the backfill resume from after a worker is killed mid-batch, and what makes that resume point trustworthy?