Aptima Software Engineer Interview Guide

The 60-second brief

Candidates describe this opening as a Senior Software Engineer post in Aptima's Productization & AI Systems group. Researchers build AI prototypes, and the engineer turns them into software that runs in operational settings for national security customers, where the aim is to help teams train, develop and perform. The reported scope covers four things. The engineer owns the technical roadmap for these AI-enabled systems, including the criteria that separate an MVP, a beta and a production release. They ship containerized deployments that follow DevSecOps practices for security and observability. They also work between researchers, UI/UX designers and end users to turn requirements into usable software. The listed must-haves are 6+ years of professional software engineering, strong Python, a record of taking prototypes to production, hands-on Docker, and eligibility for a U.S. government security clearance. Advanced MLOps frameworks, national security experience and CI/CD optimisation are listed as nice to have.

This guide is for experienced engineers interviewing for Aptima's Software Engineer (Productization & AI Systems) role, which candidates describe as a senior position. It covers the three reported rounds and the fourteen reported questions, which fall into technical and AI engineering, system design and architecture, and behavioral and leadership. It also covers the skills the posting lists: production Python, prototype-to-production work, architecture for AI-enabled systems, MLOps and Docker. Algorithm puzzles get little space here because candidates report that the technical assessment is about deployment and architecture problems, not abstract coding exercises.

Candidates report three rounds over roughly 3-5 weeks. A Technical Screening checks Python and AI systems, Architectural Discussions follow, and the last round is a Comprehensive Evaluation that may include a system design exercise or a technical presentation.

Focus areas

• Python (production software development)
• Productization (prototype to production)
• System architecture for AI-enabled systems

Overview

Candidates describe the job as taking AI prototypes that researchers built and turning them into systems that hold up in operational use for national security customers. That theme runs through the reported questions. Ten of the fourteen ask how you move, package, version, deploy and monitor models. The other four ask how you plan and how you handle researchers, designers and non-technical stakeholders. None of them is a classic algorithm problem.

Most reported questions start with "how do you" or "describe a time", so you answer them from your own work. Prepare two layers for each. The first is the mechanism. For reproducibility, that means recording the git SHA, dataset hash, lockfile, container image digest and random seed against every model version. The second is a real project where you used that mechanism, with a number attached. Listing tools like MLflow, Docker or Kubernetes without saying what you did with them shows no production experience.

The requirements candidates describe add constraints a typical web-service loop lacks. The system must deploy to both cloud and on-premise environments, follow DevSecOps practices, and secure AI pipelines in government settings. You must also be able to obtain a U.S. government security clearance. Be ready to design for restricted networks: mirrored registries, offline image bundles, signed artefacts and self-hosted monitoring. Be ready to explain drift detection when ground-truth labels arrive late or never. Candidates also report that their assumptions get challenged during the process, so attach a failure mode and a mitigation to each design choice.

Python is the language candidates report the screening checks. Make sure you can write and explain production-quality Python: typed interfaces, pytest suites, packaging with pinned dependencies, structured logging, and timeouts and error handling around a model call.

The Aptima interview process

Candidates report three rounds in this order. Only the round names and one-line descriptions are reported, so prepare each round's question types rather than expecting a fixed script.

Technical Screening

Candidates report the Technical Screening as a first check of Python and AI-systems proficiency. Whether it involves live coding or discussion is not reported. Candidates do say the technical assessment centres on real-world engineering and deployment rather than puzzles. Prepare for both. Be ready to write clean Python that wraps a model. Also be ready to explain the model lifecycle in a few sentences: experiment, packaged artefact, registry, serving, monitoring, retraining. The reported technical and AI engineering questions are good practice material for this stage. They cover the research-to-production shift, CI/CD for ML, reproducibility, field evaluation and refactoring research code. Answer each with a concrete sequence and a project you worked on.

What to demonstrate

• Proficiency in Python for production code, as reported for this stage
• Working knowledge of AI systems and the path from experiment to deployed model
• Ability to explain the reasoning behind a technical choice, not just name the tool

How to prepare

• Write a small Python inference module. It loads a versioned model artefact, validates input with a typed schema (pydantic or dataclasses), returns predictions with the model version attached, and has pytest tests for bad input and a missing artefact.
• Rehearse spoken answers to the research-to-production question and the CI/CD for ML question. Each should name the order of steps and tools you have actually used.
• Review production Python basics you may need to defend: virtual environments and lockfiles, type hints with mypy, logging configuration, exception boundaries, and profiling a slow function with cProfile.

Architectural Discussions

Candidates report the Architectural Discussions as in-depth conversations that assess both technical knowledge and fit in a cross-functional team. The reported design questions are the material to prepare: APIs and pipelines for real-time inference, the criteria for flexibility versus performance, containerizing multi-service AI applications, deploying to both cloud and on-premise, and building in observability. For each, be able to sketch one architecture and explain the trade-off behind every component. The role works with researchers, UI/UX designers and customers. So when you justify a design choice, say who it serves: a researcher who swaps models often, an operator who needs stable releases, or an end user who needs fast responses.

What to demonstrate

• Depth of architectural knowledge for AI-enabled, multi-component systems
• Fit within a cross-functional team of researchers, engineers and designers, as reported
• Reasoning about the tension between research flexibility and production stability

How to prepare

• Sketch one real-time inference architecture end to end (gateway, inference service, model server, feature lookup, async queue, prediction log) and list one failure mode and its mitigation for each component.
• Write a one-page comparison of the same system deployed to a public cloud and to an on-premise site. Cover what changes for storage, the container registry, secrets, identity, monitoring and updates.
• Prepare one example where a researcher's or designer's requirement changed your architecture, and what you gave up to meet it.

Comprehensive Evaluation

Candidates report the Comprehensive Evaluation as the final assessment, and say it may include a system design exercise or a technical presentation. Prepare both. For a presentation, pick one research-to-production project you can defend in depth. Cover where the code started, the architecture you chose, the trade-offs you rejected, how you showed the production version matched the research results, how you monitored it, and what you would change. For a design exercise, practise turning a vague prompt into requirements and an architecture, using the reported design questions as prompts. Re-architecting a legacy research script into a service-oriented API is a useful one to rehearse.

What to demonstrate

• End-to-end system design for a production AI system, when the exercise is given
• Clear presentation of a past project's architecture, trade-offs and results, when a presentation is given
• Spotting operational risks and building mitigations into the design, which candidates report being challenged on

How to prepare

• Build a short slide outline for your strongest research-to-production project: starting state, constraints, architecture diagram, two rejected alternatives, parity and monitoring evidence, results, lessons.
• Run a mock design exercise: re-architect a single research script into a containerized, service-oriented API that runs on cloud and on-premise. Say the requirements aloud before drawing anything.
• For your design, list the top three operational risks (model drift, a dependency outage, a bad model release) and the specific control for each (drift alert, fallback path, versioned rollback).

Where candidates lose points

Answering the research-to-production and CI/CD questions with a list of tools (MLflow, Docker, Kubernetes) and no sequence or project behind them.

For each technical question, prepare the order of steps (freeze a baseline, split training from inference, package, gate, deploy, monitor) and attach one project where you did it, with a number: latency, release frequency, defects caught.

Telling the refactoring story with no proof that the model behaved the same after the refactor.

Build golden-output parity into the story: characterization tests on fixed inputs, outputs compared within a stated tolerance, and what you did when a difference appeared. It is the first thing a skeptical listener asks.

Designing the inference API or the hybrid deployment as if it only ever runs in one connected public cloud.

For every managed service you name, have an on-premise substitute ready (MinIO for S3, self-hosted Prometheus for SaaS monitoring, a mirrored registry). Explain how updates reach a site with no internet connection.

Treating model monitoring as uptime monitoring when asked how to evaluate a deployed model or build in observability.

Keep service health (latency, errors, saturation) separate from model health (input and prediction drift, delayed-label accuracy, slice metrics). Name the statistic you would use, such as PSI or a KS test, and what each alert triggers.

Mentioning security only at the end of a design, or being vague about the security clearance requirement.

Put DevSecOps controls into the design as you go: image scanning, SBOM, signed artefacts, runtime secrets, least-privilege access and audit logs. Prepare a short, factual answer on your clearance eligibility and willingness to go through vetting.

Questions & practice

Describe a situation where you had to lead technical planning for an ambiguous, high-stakes project.

Medium · Behavioral & Leadership

Approach

1. This is a behavioral question despite the coding label. Make the ambiguity and the stakes concrete: an unclear requirement, uncertainty about whether the research would work, and a customer or deadline depending on it.
2. Show how you turned the ambiguity into decisions. You listed the unknowns, ran short spikes to retire the biggest risk first, and wrote a design doc. You set milestones with exit criteria (MVP, beta, production-ready), got stakeholders to sign off, and scheduled checkpoints to re-plan.
3. Include one mid-project change, such as a new requirement or a failed experiment, and how the plan absorbed it. That is the proof you were leading the planning, not executing someone else's plan.
4. Result and lesson: delivery outcome, quality and adoption numbers, plus what you would plan differently. The story fails if the plan came from your manager or if nothing was actually uncertain.

Follow-up

• Which risk did you retire first, and why that one?
• What did you cut from scope, and how did you get agreement?
• What would you plan differently now?

Find overlapping job attempts and peak concurrency from lease records

Medium · sweep line · intervals · leases

A day of job_run history yields about 50,000,000 attempt records: (job_run_id, job_type, attempt, started_at, finished_at which is NULL when the worker died, lease_expires_at). Leases expire on a clock, so a job that outran its lease ran twice. Produce (a) every job_run_id whose attempts overlapped in wall-clock time and (b) the peak number of simultaneously running attempts per job_type with the minute it occurred. Target O(n log n). State how you treat a NULL finished_at and what clock skew does to your answer.

Approach

1. Define the interval before sorting anything: an attempt occupies [started_at, COALESCE(finished_at, lease_expires_at)). finished_at is observed and lease_expires_at is only a promise, so every attempt without a finish contributes an estimate and the whole result is a lower bound on overlap rather than an exact count.
2. For peak concurrency, sweep: emit 2n endpoints, sort by (timestamp, kind) with ends ordered before starts at equal timestamps, then walk the sequence maintaining a counter per job_type and record each type's maximum with its timestamp. O(n log n) dominated by the sort, O(n) space, or O(1) extra if the sort is external and the walk streams.
3. For overlap detection, do not compare attempts pairwise. A single global sort by (job_run_id, started_at) gives both the grouping and the order; within a group, keep the maximum end seen so far and report an overlap exactly when the next start is less than that running maximum, which is one linear pass after the sort.
4. Half-open intervals matter and are easy to get wrong: with closed intervals an attempt ending at the same millisecond another begins reads as concurrency two, and across 50,000,000 records that artefact swamps the real signal.

Follow-up

• A handler is not idempotent and you have found 400 overlapping jobs. Which of them actually caused damage, and what would you query to find out?
• Peak concurrency for one job_type is 4 against a configured cap of 4. Is the cap working, or is the data hiding attempts that never started?

Archive a resource graph without breaking live references or recursing

Medium · graph traversal · topological order · tenant isolation

Resources reference other resources within a tenant; for the largest tenant the reference table holds up to 2,000,000 nodes and 8,000,000 edges. Archiving a resource must archive everything reachable from it that nothing outside the set still references, refuse when a live external referrer exists, and terminate when references form cycles, which they legitimately do. Produce the archive order and the refusal list, targeting O(V+E). Say what stops the traversal crossing a tenant boundary, and why recursion is the wrong control structure at this size.

Approach

1. Load the subgraph with the tenant predicate on both endpoints of the edge, not only on the side you started from. Scoping the left table alone is the classic cross-tenant leak: one mis-entered edge then pulls another tenant's resources into the traversal and, worse, into the archive.
2. Traverse iteratively with an explicit stack. A 2,000,000-node graph can hold a chain deep enough to exhaust a native stack in the low tens of thousands of frames, and that failure is a process crash rather than an error you can return.
3. Treat cycles as data rather than corruption: compute strongly connected components with Tarjan in O(V+E) using its own explicit stack, then condense. The condensation is a DAG, so a topological order over it gives the archive order, and every member of a component archives in one transaction because no order within a cycle is valid.
4. Decide refusals with reverse edges. A candidate is archivable only if every in-edge originates inside the candidate set, so build the transpose or count in-degrees restricted to the visited set, and emit each blocked resource with the id of the external referrer, which is the only part of the answer an operator can act on.

Follow-up

• The graph is read in one query and the archive writes a minute later. What can change in between, and how do you make the write safe?
• The candidate set is 400,000 resources. Is that one transaction, and if not, what does a half-finished archive look like to a reader?

Find version gaps and relay lag with window functions

Medium · window functions · event tables · frame semantics

outbox_event holds event_id, aggregate_type, aggregate_id, aggregate_version, event_type, payload, status ('pending','published','dead'), attempts, created_at, published_at. A projection is missing rows and you must decide whether the relay skipped events or the consumer dropped them. Write three queries over the last seven days: one listing every aggregate_id whose published aggregate_version sequence has a hole, one giving per-day counts with a running total, and one returning the newest published event per aggregate. For each, say where the window function is evaluated relative to WHERE and LIMIT. PostgreSQL 16.

Approach

1. Gaps: compute lead(aggregate_version) OVER (PARTITION BY aggregate_id ORDER BY aggregate_version) in a subquery, then filter next_version <> aggregate_version + 1 in the outer query. Window functions are evaluated after WHERE, GROUP BY and HAVING and before the outer ORDER BY and LIMIT, so the predicate cannot sit in the same WHERE clause and PostgreSQL 16 has no QUALIFY.
2. Say what the seven-day filter does to the answer: it truncates every partition, so the first row per aggregate has no predecessor inside the window and a hole spanning the boundary is invisible. Widen the window, or join to resource.version as the authority for the true maximum.
3. Running total: SELECT date_trunc('day', created_at) AS d, count() AS n, sum(count()) OVER (ORDER BY date_trunc('day', created_at) ROWS UNBOUNDED PRECEDING). An aggregate inside a window call is legal because grouping runs before windowing. The grouping key is unique per row here so ROWS and RANGE agree, but write the frame anyway — over ungrouped rows with tied timestamps the default RANGE frame pulls in every peer row and the total jumps.
4. Newest per aggregate: DISTINCT ON (aggregate_id) ... ORDER BY aggregate_id, aggregate_version DESC is the cheap PostgreSQL-only form when an index matches that order; row_number() OVER (PARTITION BY aggregate_id ORDER BY aggregate_version DESC) = 1 is the portable form and needs a subquery for the same evaluation-order reason as the gap query.

Follow-up

• Relay failover redelivers events. Does a duplicate break the gap query, and how would you detect one from this table alone?
• Turn the gap check into a continuous monitor rather than a query someone runs after an incident. What does it watch?

Hold a per-tenant active cap against concurrent creates

Hard · write skew · isolation levels · constraints

A tenant on the standard plan may hold at most 50 resources with status='active'. The create handler runs SELECT count(*) FROM resource WHERE tenant_id = $1 AND status = 'active', compares to 50, then inserts. Two creates arrive 3 ms apart on different instances and the tenant lands at 51. Name the anomaly, say whether PostgreSQL 16 READ COMMITTED or REPEATABLE READ prevents it and why, then give an implementation that holds the cap at READ COMMITTED with the exact statements. Finally, say what changes when the cap is 'at most one running export per tenant' on job_run.

Approach

1. Name it: write skew. The two transactions read an overlapping set and write disjoint rows, so there is no row-level conflict for the engine to detect and each commit is individually legal.
2. Rule out the levels precisely. READ COMMITTED takes a fresh snapshot per statement and takes no lock on the counted rows, so both see 49. PostgreSQL's REPEATABLE READ is snapshot isolation: it removes non-repeatable reads and phantoms within the snapshot but still admits write skew, because the anomaly is not a re-read of a changed row, it is a read of a set that a concurrent transaction invalidates. Only SERIALIZABLE closes it, by tracking the read dependency and aborting one transaction with SQLSTATE 40001 — a guarantee that exists only if the application re-runs the whole transaction from the read.
3. Convert the set predicate into a single-row conflict: keep tenant.active_resource_count and run UPDATE tenant SET active_resource_count = active_resource_count + 1 WHERE tenant_id = $1 AND active_resource_count < 50 in the same transaction as the INSERT. Zero affected rows is the cap, returned as 409. The row lock serialises the decision at any isolation level, and contention is bounded to one tenant's row — which is also the fair-scheduling unit, unlike a global counter that would convoy every tenant behind one row.
4. State the cost you just took on: a counter is a second source of truth that can drift, so every path that changes status must adjust it inside the same transaction, and a periodic reconciliation has to exist, with resource_revision as the authority for what the count should have been.

Follow-up

• A resource moves from archived back to active. Which statements change, and what breaks if the counter update and the status change land in different transactions?
• The cap becomes plan-dependent and a plan can change mid-month. Where does the number 50 live, and who reads it?

Explain your approach to implementing CI/CD pipelines specifically for AI/ML workflows.

Medium · Technical & AI Engineering

Approach

1. Say that ML pipelines version three things, code, data and model, and that each needs its own checks. On every commit, run lint, type checks, unit tests for feature code, schema validation of input data (pandera or Great Expectations), and a smoke training run on a small fixture dataset.
2. Add an evaluation gate. A candidate model is trained and scored against a fixed held-out set and against the current production model. It is promoted only if it clears metric thresholds and does not regress on important slices.
3. For delivery, build an immutable image tagged with the git SHA and model version. Scan it, and generate an SBOM and signature as DevSecOps steps. Deploy through staging, then shadow or canary, then promote. Rollback means redeploying the previous image or model version.
4. Name the trade-offs. Full training is too slow and costly for every commit, so separate fast per-commit checks from scheduled or triggered training jobs on GPU runners. Assert metrics within a tolerance, because nondeterministic training makes exact asserts flaky.

Follow-up

• How do you stop a model that improves the aggregate metric but regresses on an important subgroup?
• How would you roll back a model without rolling back the code?
• Where do the expensive training jobs run, and what triggers them?

What strategies do you use to ensure reproducibility and versioning in complex AI systems?

Medium · Technical & AI Engineering

Approach

1. Version every input to a model. Record the code by git SHA. Record data with DVC, lakeFS or immutable snapshots identified by a content hash. Record the environment with a lockfile and the container image digest. Also record config and hyperparameters, and the random seeds.
2. Track experiments with MLflow, Weights & Biases or similar, logging parameters, metrics and artefacts. Keep a model registry where each model version links back to its data version, code SHA and evaluation report.
3. Treat determinism honestly. Seed Python, NumPy and the framework and turn on deterministic algorithm flags where they exist. Then define 'reproduced' as matching within a stated tolerance, because GPU kernels are not always bit-for-bit repeatable.
4. Version the serving contract separately from the weights. Store the feature schema and preprocessing version with the model, so a new model cannot be deployed against inputs it was not trained on.
5. Cover the edge cases: upstream data that changes in place, library upgrades that change numerics, and datasets too large to copy. For those, store manifests and hashes rather than full copies.

Follow-up

• A model in the field misbehaves: how do you reconstruct exactly what produced it?
• How do you version a dataset too large to copy for every experiment?
• What do you accept as 'reproduced' when training is not bit-for-bit deterministic?

How do you evaluate the performance of an AI model once it is deployed in the field?

Medium · Technical & AI Engineering

Approach

1. Separate what you can measure right away from what needs labels. Right away you have input feature distributions, the prediction distribution, confidence scores, latency and error rates. Accuracy, precision and recall have to wait for ground truth.
2. Detect drift against a reference window from training. Use PSI or a Kolmogorov-Smirnov test per numeric feature and chi-square for categorical features, plus a drift check on the prediction distribution. Alert on a sustained shift, not a single spike.
3. Plan how labels arrive: sample predictions for human review, capture user corrections, and join delayed outcomes back to logged predictions by request id. Evaluate by slice, because aggregate metrics hide subgroup regressions.
4. Before a new model replaces the old one, compare them online with shadow mode or a canary, with guardrail metrics. Link the model metric to the task outcome the end user cares about.
5. Note the field-specific problems. Disconnected deployments need metrics buffered locally and synced later. Thresholds need tuning to avoid alert fatigue. Decide in advance which signals trigger a retrain and which trigger a rollback.

Follow-up

• What do you do when labels arrive weeks later, or never?
• How do you tell a broken upstream data pipeline from real drift?
• What triggers a retrain versus a rollback?

How do you design APIs and data pipelines to support real-time AI inference?

Medium · System Design & Architecture

Approach

1. Pin down requirements first: latency target, request rate, payload size, whether callers can wait or need an async job, whether requests can be batched, and model size and hardware.
2. Synchronous path: authenticated gateway, then a stateless inference service (FastAPI or gRPC) with schema validation, then a model server with dynamic batching (Triton, TorchServe or in-process for small models), then the response. Look up features from an online store or cache. Log each request and prediction to a stream for monitoring and retraining.
3. Async path for slow inference: the client submits a job and gets back an id. Workers consume from a queue (Kafka, RabbitMQ or Redis streams), write results to a store, and the client polls or receives a callback. Make submission idempotent so retries do not create duplicate jobs.
4. Contract and rollout: version the endpoint and return the model version in every response. Set timeouts and a fallback (cached or rule-based answer). Blue-green or canary releases let you swap models without downtime.
5. Scaling and skew: autoscale on queue depth or GPU utilisation and keep models warm to avoid cold starts. Larger batches raise throughput but add latency. Share feature code between training and serving and use point-in-time-correct features.

Follow-up

• How do you roll out a new model version behind the same API without downtime?
• What happens when the feature store is slow or unavailable?
• How does the design change if one inference takes several seconds?

When making tradeoffs between system flexibility and performance, what criteria do you prioritize?

Medium · System Design & Architecture

Approach

1. Give your criteria explicitly. Who changes the system and how often (researchers swapping models weekly versus a stable operational release)? What are the hard limits (latency, target hardware, security accreditation)? How expensive is reversing the choice later? Can the team operate what you build?
2. Put flexibility at the boundaries and performance on the hot path. A model adapter interface or config-driven pipeline lets researchers plug in new models, while the serving path is optimized with a compiled format, batching and caching.
3. Measure before trading anything away. Profile, set a latency or cost budget, and only then decide which abstraction to remove. Treat easily reversible choices lightly and spend scrutiny on the ones that are hard to undo.
4. Anchor it with one real example and its numbers: what you chose, what it cost, and what the other option would have cost. An answer with no example sounds like a textbook.

Follow-up

• Give an example where you chose flexibility and later regretted it.
• How do you decide a research-friendly abstraction must go for performance?
• Who gets a say in that decision, and how do you settle disagreement?

How do you approach containerization for complex, multi-service AI applications?

Medium · System Design & Architecture

Approach

1. Decompose by responsibility: API service, model server, background worker, queue, database and monitoring stack. Run one main process per container, connected through explicit network interfaces.
2. Image hygiene: multi-stage builds, a slim pinned base image, pinned Python dependencies, a non-root user, image scanning (Trivy or Grype) and an SBOM. For GPU images, match the CUDA base image to the host driver version.
3. Model weights: either bake them into the image (self-contained, good for offline sites, but images grow to gigabytes) or pull them from an object store or volume at startup (smaller images, but a runtime dependency and a slower cold start). Choose per deployment target and say why.
4. Orchestration: Docker Compose for development and single-node sites, Kubernetes with Helm for scale. Add readiness probes that pass only once the model is loaded, resource and GPU limits, and secrets injected at runtime rather than baked into images. Tag images with the git SHA and model version.

Follow-up

• Do you bake model weights into the image or load them at startup, and why?
• How do you handle GPU and CUDA compatibility across different hosts?
• A container works locally but fails in the target environment: how do you debug it?

Describe how you would design a system to be deployed in both cloud and on-premise environments.

Medium · System Design & Architecture

Approach

1. Hide environment-specific services behind interfaces so the core code runs anywhere. Use the S3 API for object storage (MinIO on-premise), a portable queue, Vault or a similar secrets store, and standard identity (OIDC). Keep cloud-only managed services out of the core path.
2. Ship one artefact set: container images plus Helm charts or Compose files, with per-environment config. Run CI tests against both targets so they do not drift apart.
3. For disconnected or air-gapped sites, plan an offline bundle of signed images, a mirrored registry, vendored dependencies and model artefacts, plus a documented update and rollback procedure for sites with no internet access.
4. Make observability work without SaaS: Prometheus and Grafana, local log storage, and export or sync when a connection exists.
5. State the cost. Designing to the lowest common denominator gives up managed features, the test matrix grows, and on-premise capacity is fixed, so autoscaling assumptions change.

Follow-up

• How do you ship an update to a site with no internet connection?
• Which managed cloud service would you miss most on-premise, and what replaces it?
• How do you keep the cloud and on-premise deployments from drifting apart?

How do you incorporate observability and monitoring into your system architecture?

Medium · System Design & Architecture

Approach

1. Cover the three signals. Metrics: rate, errors and duration per endpoint, plus saturation of CPU, memory and GPU. Logs: structured, with a correlation id on every request. Traces: distributed tracing, for example OpenTelemetry, across the API, model server and data stores.
2. Add the ML-specific layer: the model version on every request, distributions of inputs and predictions, data quality checks in pipelines, and drift alerts. A service can be healthy while the model is wrong.
3. Define SLOs and alert on symptoms users feel, such as p99 latency and error rate, rather than every internal cause. Give each alert a runbook and decide which alerts page someone and which go to a ticket.
4. Build it in from the start with a shared service template that includes instrumentation, health and readiness endpoints, and audit logs. Keep sensitive inputs out of logs by hashing or redacting fields.
5. Edge cases: high-cardinality metric labels that blow up storage, log volume cost, and sites that cannot send telemetry out, which need local collection and later sync.

Follow-up

• p99 latency doubled while p50 stayed flat: where do you look first?
• How do you log predictions without leaking sensitive inputs?
• Which alerts should wake someone up at night?

p99 jumped on one listing filter while p50 stayed flat

Medium · query planning · composite index · percentiles

After a release that added an owner_user_id filter to the resource listing, p99 rose from 90 ms to 1.9 s while p50 stayed at 40 ms. Traffic and row counts are unchanged. resource carries the index (tenant_id, status, updated_at DESC, resource_id DESC). The new query filters tenant_id and owner_user_id, orders by updated_at DESC, resource_id DESC, and takes 20 rows. On PostgreSQL, explain the shape of the regression, prove it from a query plan, and give the index you would add.

Approach

1. Start from the shape. A flat p50 with a moved p99 means a subset of requests changed cost, not all of them, so the first job is naming the subset. Bucket the endpoint's latency by the tenant's row count; the natural hypothesis is that large tenants are a small share of requests and all of the tail.
2. Get the plan for the new query on a large tenant with EXPLAIN (ANALYZE, BUFFERS). Expect an index scan over the tenant's range, a filter discarding most of it, then a Sort feeding the Limit, possibly reporting Sort Method: external merge Disk. Read actual rows on the scan node, not estimated.
3. Explain why the existing index cannot serve it. A composite B-tree is seekable only as a left prefix, and with no equality predicate on status the scan cannot treat updated_at as an ordering, because rows in the tenant's range are ordered by status first. Everything matching must be read and sorted before LIMIT 20 can apply, so a tenant with 400,000 rows pays 400,000 rows to return 20.
4. Add (tenant_id, owner_user_id, updated_at DESC, resource_id DESC). Equality on the first two columns leaves the index ordered by updated_at within that pair, so the plan becomes an index scan that stops after 20 rows with no Sort node. PostgreSQL can scan a B-tree backwards, so the DESC markers matter only if the two sort columns ever disagree in direction; keeping them explicit documents the order the keyset cursor depends on.

Follow-up

• The endpoint paginates with OFFSET. What does page 500 cost with your index, and what does the keyset version cost?
• How would you have caught this before release, given that a 10,000-row seed database produces the same plan shape at an unnoticeable cost?

Suggested practice checklist

Each day covers one group of the reported questions and ends in something you can rehearse from: working code, an architecture sketch or written stories. Days 1-5 build the technical and design material for the screening and architectural rounds. Days 6-7 cover the behavioral questions and the design exercise or presentation that candidates report for the final round.

Day 1 — Production Python and a containerized inference service

• Write a Python inference module with a typed input schema, a versioned model artefact, structured logging and pytest tests for bad input and a missing model file.
• Containerize it with a multi-stage Dockerfile, a pinned base image and a non-root user, and add a readiness check that passes only after the model loads.
• Solve the lease-records peak-concurrency problem in Python with a sweep over sorted start and end events, and state its complexity.

Deliverable: A tested inference module, a working Dockerfile with readiness check, and a commented solution to the peak-concurrency problem.

Day 2 — Research to production and reproducibility

• Write out the steps you would follow to productize a research model, from freezing a baseline to the parity test, and map each step to a project you have done.
• Write a reproducibility checklist: code SHA, data hash, lockfile, image digest, seeds, config, plus the registry metadata that links them.
• Outline the refactoring story in STAR form, with the parity evidence and the researchers' role spelled out.

Deliverable: A step-by-step productization answer, a reproducibility checklist, and a STAR outline for the refactoring story.

Day 3 — CI/CD for ML and evaluating models in the field

• Sketch a pipeline with per-commit checks, a triggered training job, an evaluation gate against the production model, an image build with scan and signature, and a canary rollout with rollback.
• Compute PSI between a reference and a shifted synthetic feature in Python and choose an alert threshold you can justify.
• Write how you would collect delayed labels and which signals trigger a retrain versus a rollback.

Deliverable: A pipeline stage diagram, a PSI script with a chosen threshold, and a one-page field-evaluation plan.

Day 4 — Real-time inference APIs and observability

• Draw the synchronous and async inference paths with the gateway, model server, batching, feature lookup, queue and prediction log, and list one failure and its fallback per component.
• Define the metrics, log fields, traces and drift alerts for that design, and mark which alerts page and which open a ticket.
• Work through the debugging case where p99 jumped while p50 stayed flat, and write the order in which you would check things.

Deliverable: An annotated inference architecture, an observability spec with alert tiers, and a written debugging sequence for the p99 case.

Day 5 — Hybrid deployment and trade-off criteria

• Write the cloud-versus-on-premise comparison for your day 4 design: storage, registry, secrets, identity, monitoring, and offline updates.
• Write your criteria for trading flexibility against performance and attach one real example with numbers.
• Rehearse both answers aloud, then list the weakest assumption in each and add a mitigation.

Deliverable: A hybrid-deployment comparison sheet, a trade-off answer with a worked example, and a list of assumptions with mitigations.

Day 6 — Behavioral and leadership stories

• Write STAR stories for planning an ambiguous high-stakes project and for a research-versus-engineering disagreement, each with a decision you made and a measured result.
• Write the stakeholder trade-off story using the structure decision, options, costs, recommendation, outcome.
• Pick one Aptima value, write a definition and a story for it, and rehearse all four aloud.

Deliverable: Four written behavioral stories, each with a result and a lesson, rehearsed aloud once.

Day 7 — Comprehensive Evaluation rehearsal

• Build the slide outline for your strongest research-to-production project and present it aloud, ending with what you would change.
• Run a mock design exercise: turn a single research script into a containerized service-oriented API that deploys to cloud and on-premise, stating requirements before drawing.
• List the three biggest operational risks in your mock design and the control for each, then prepare your clearance answer and questions to ask about model lifecycle and customer integration.

Deliverable: A presentation outline, a mock design with a risk table, and a short list of answers and questions for the final round.

Behavioural preparation

Candidates report four behavioral and leadership questions for this role: planning under ambiguity, conflict between research and engineering, explaining trade-offs to non-technical people, and Aptima's listed values. The reported job involves setting roadmaps and working between researchers, designers and end users. So each story should show you making a technical decision together with other people and backing it with evidence.

1. Describe a situation where you had to lead technical planning for an ambiguous, high-stakes project.
2. Describe a time you had to design a system with strict security or compliance requirements.

How do you handle the architectural shift when moving a research-based model into a production environment?

Medium · Technical & AI Engineering

Approach

1. Name what actually changes. The research code is usually a notebook or script with global state, hard-coded paths and an implicit dataset. The production version needs a defined interface, external config, versioned artefacts, tests, and someone who owns it when it breaks.
2. Freeze a reference first. Pin the research commit, data snapshot and environment, and record baseline metrics on a held-out set, so you can later show the production port gives the same results.
3. Split training code from inference code. Move preprocessing into one tested library used by both, because duplicated feature logic is the usual cause of training/serving skew.
4. Package the model in an explicit format (TorchScript, ONNX or a pickled sklearn pipeline with pinned versions) and register it with metadata. Wrap it in a validated API and containerize it with pinned dependencies.
5. Settle the non-functional requirements researchers rarely state: latency budget, batch or online serving, CPU or GPU, fallback when the model fails, and the logging needed for drift monitoring. Close with a parity test that compares the research and production paths on golden inputs within a tolerance.

Follow-up

• How do you prove the production model produces the same outputs as the research version?
• What do you do when the researchers keep changing the model after productization has started?
• When would you rewrite the research code instead of wrapping it?

Describe a time you had to refactor a research codebase to make it modular and testable.

Medium · Technical & AI Engineering

Approach

1. Situation and task: describe the concrete starting state (one long notebook, global variables, hard-coded file paths, no tests) and what production needed from it (an API, repeatable runs, deployment in a container).
2. Actions, in order. First, write characterization tests that capture current outputs on fixed inputs. Then extract modules for data loading, features, model and evaluation. Move config out of the code, inject I/O so units can be tested, and add CI. Change things step by step rather than in one big rewrite.
3. Show how you kept the researchers productive: a shared package they could still import, a review process they agreed to, and a stable interface that let them swap models.
4. Result: give measurable outcomes such as test coverage, outputs matching the original within tolerance, time for a new person to run it, deploy frequency, or bugs caught. The story fails if you rewrote everything alone with no parity check, or if the researchers never appear in it.

Follow-up

• How did you make sure the refactor did not change the model's behaviour?
• How did the researchers react, and how did you keep them productive during it?
• What did you deliberately leave untouched, and why?

How do you handle disagreements between research objectives and engineering constraints?

Medium · Behavioral & Leadership

Approach

1. Show you understand both sides. Researchers want to iterate freely, use new libraries and push accuracy. Engineering needs stability, latency, maintainability and security. Neither side is wrong.
2. Make the disagreement concrete with data. Benchmark the research approach's latency, memory and dependency risk, and measure how much accuracy it actually adds. A vague argument often settles once both numbers are on the table.
3. Look for options beyond yes or no: separate research and production tracks, a stable interface contract, feature flags, staged adoption, or a time-boxed experiment with agreed success criteria.
4. Use a STAR example where the decision was made against criteria both sides accepted and you committed to the outcome. The story fails if you describe researchers as obstacles or the conclusion is that you won the argument.

Follow-up

• Tell me about a time you were the one who conceded.
• How do researchers keep experimenting once the system is in production?
• What if the accuracy gain is real but the model cannot meet the latency budget?

Tell me about a time you had to explain a complex technical tradeoff to a non-technical stakeholder.

Medium · Behavioral & Leadership

Approach

1. Choose a real decision with real options, such as shipping a smaller model now versus a more accurate one later, or cloud versus on-premise hosting. Avoid stories about explaining a concept with no decision attached.
2. Structure what you told the stakeholder: the decision needed, two or three options, what each costs in schedule, money, risk and capability for the end user, your recommendation, and what you needed from them.
3. Translate into their terms. Drop the jargon, and use a concrete scenario ('if the network drops, the operator sees X') rather than technical metrics.
4. Close with evidence they understood (they restated it or made the call) and the outcome. Any condescending note in the telling undermines the story.

Follow-up

• How did you know they understood?
• What did you leave out, and why?
• What happened when they chose the option you did not recommend?

Which of Aptima’s core values (Integrity, Ingenuity, Excellence, Respect, Engagement, Teamwork) resonates most

Medium · Behavioral & Leadership

Which of Aptima’s core values (Integrity, Ingenuity, Excellence, Respect, Engagement, Teamwork) resonates most with your professional philosophy?

Approach

1. Pick one value from the list (Integrity, Ingenuity, Excellence, Respect, Engagement, Teamwork). Do not list all six. Say in one sentence what it means in your day-to-day engineering.
2. Back it with a short, specific story. For Integrity, that could be reporting a model's limitation or a failed evaluation honestly instead of overstating results. For Ingenuity, a creative fix for a deployment constraint. For Teamwork, unblocking a researcher at a cost to your own work.
3. Tie it to the role's reported work: moving research into production and working with researchers, designers and customers.
4. The answer fails if it is generic praise with no story, or if it picks a value and then describes behaviour that contradicts it.

Follow-up

• Tell me about a time that value cost you something.
• Which of these values do you find hardest to live up to?
• How would your teammates say you show it?