Interview guides · ClearEdge

ClearEdge Software Engineer Interview Guide

ClearEdge Software Engineer interview guide: three reported rounds, Docker, Kubernetes, GitLab CI and Python topics, behavioral answers and a 7-day plan.

The 60-second brief

Candidates describe ClearEdge as a company whose engineers build software for Department of Defense programs, working alongside government and industry partners. The reported work includes designing and deploying containerized applications and building data-driven analytics systems. The languages named for the role are Python services, Rust capabilities and React front ends, and engineers are expected to fit their code in with COTS, GOTS and FOSS products. Nice-to-have items candidates list include AWS, signal processing, audio and video streaming formats, Java/J2EE and Kafka. For you, that means a role where writing code and shipping it (containers, orchestration, pipelines) are one job.

This guide is for engineers preparing for the ClearEdge Software Engineer loop. The requirements candidates describe ask for 7+ years of experience, Linux command line, Bash and Python skills, and hands-on Docker, Kubernetes and GitLab CI, so the guide assumes a mid-to-senior profile. It covers the three reported rounds, the reported technical, design and behavioral questions (with worked approaches for the behavioral ones), the mistakes that cost candidates, and a seven-day plan. It does not cover clearance paperwork or compensation.

Candidates report three rounds over roughly 3-5 weeks: Technical Screening, Technical Deep-Dives and Behavioral Rounds. The first two test technical depth across the stack the role lists, and the last tests fit with the values candidates describe.

Focus areas

Python (3.x) Development
Containerization (Docker)
Bash Shell Scripting

Overview

The reported work sits between development and operations. Candidates describe modular software that processes complex data flows, shipped as containers, run on Kubernetes and released through GitLab CI pipelines, with Python and Bash for glue and automation. Infrastructure-as-code with Ansible or Terraform and database tuning in MongoDB or PostgreSQL come up as deeper areas. Preparing only algorithm puzzles leaves the largest part of that list untouched, so give at least half of your time to deployment, pipelines and data stores.

The reported questions fall into three families. The technical family covers Helm charts, MongoDB query optimization, container orchestration trade-offs, Bash automation of system and CI/CD tasks, and integrating custom software with GOTS or COTS products. The design family covers a scalable containerized service for high-volume data flows, software quality in a CI/CD pipeline for a secure environment, turning an architecture diagram into code, and hardware versus software trade-offs early in a project. The behavioral family covers leading a late project, ambiguous customer requirements, a technical disagreement, and balancing rapid deployment against rigorous testing. Reports do not say which round asks which question, so cover every family.

Two reported points shape how you answer. First, candidates say engineers from the team take part in the process, and the tip is to say "I" and explain why you chose what you chose, because your own contribution is what matters. Second, the requirements list TS/SCI with Polygraph as non-negotiable, and candidates advise showing discretion about past work. Prepare to describe projects precisely without revealing anything restricted, and prepare to explain technical choices to someone outside your discipline, since reports stress documentation and briefings.

The ClearEdge interview process

Candidates report three rounds over roughly 3-5 weeks. Reports do not tie particular questions to particular rounds, so treat each question family as fair game at every stage.

Technical Screening

Candidates describe the first stage as an initial check of core competencies and technical knowledge, and say screenings may be remote. The requirements candidates list for the role (Linux command line, Bash, Python, Docker, Kubernetes, GitLab CI) are the likeliest ground, so be ready to discuss each with specifics from your own work. One reported example scenario is troubleshooting a failed deployment in a Kubernetes cluster, which rewards a fixed diagnostic order: pod status, then events, then logs, then the rollout or release history.

What to demonstrate

Breadth across the stack the role lists: Python 3, Linux/Unix, Bash, containers
Hands-on fluency, not definitions: what you ran, what broke, what you changed
Clear spoken explanation of tools you have actually used

How to prepare

Rehearse the Kubernetes failed-deployment sequence: kubectl get pods, describe pod, read events, kubectl logs (and --previous), rollout status, then roll back. Tie each symptom (ImagePullBackOff, CrashLoopBackOff, Pending) to its usual cause.
Refresh Python 3 you can write unaided: comprehensions, generators, context managers, subprocess, exception handling, a short log-parsing script.
Review Bash essentials: pipelines, grep/awk/sed, find, exit codes, quoting, set -euo pipefail, trap for cleanup.
Prepare a short account of one Docker or Kubernetes deployment you personally owned, including one thing that went wrong.

Technical Deep-Dives

Candidates describe in-depth technical interviews that focus on specific skills and problem-solving. The reported evaluation areas are containerization (Docker and Kubernetes orchestration), CI/CD with GitLab CI, database work in MongoDB or PostgreSQL including query optimization, and infrastructure-as-code with Ansible or Terraform. A reported example asks you to contrast service-oriented architecture with a monolith from your past projects. Candidates also advise being ready to draw systems on a whiteboard or digital equivalent. Pick two or three areas where you can go several levels deep rather than skimming all of them.

What to demonstrate

Depth in the specific tools you claim, down to configuration and failure modes
Debugging logic and architecture reasoning, including performance against maintainability
The reasons behind technical choices, not only the choices

How to prepare

Build and read a small Helm chart (helm create, values.yaml, helm template, helm lint, helm rollback) so you can explain templating, per-environment values and release upgrades.
Write a trade-off table for orchestration choices and for SOA versus monolith, with the decision criteria from your own projects.
Practise drawing a containerized data-processing pipeline and narrating components, queues, failure modes and deployment path in order.

Behavioral Rounds

Candidates describe the closing stage as behavioral interviews that assess fit and alignment with company values. Reports name honesty, integrity and accountability, and the reported questions in this family cover leading a project that fell behind, handling unclear customer requirements, disagreeing with a team decision, and balancing fast deployment with testing. Candidates also describe leadership and collaboration expectations for mid-to-senior engineers: mentoring, leading technical discussions, and explaining decisions in writing and in briefings. Build a story bank in advance and keep each story in the first person.

What to demonstrate

Fit with the values candidates report: honesty, integrity, accountability
Leadership and collaboration: mentoring, leading technical discussions
Clarity when explaining technical choices to non-specialists

How to prepare

Write one story per reported value, each with a real dilemma, the decision you made, what it cost you and what happened.
Prepare the four reported behavioral questions as stories with a measured outcome, plus one story where you mentored someone or led a technical discussion.
Practise describing past projects precisely while leaving out anything sensitive, since candidates advise showing discretion.
Say "I" in every story and keep a single sentence ready for what you personally did.

Where candidates lose points

Describing projects as "we" so your own contribution is invisible

Candidates are advised to say "I" and explain why they chose what they chose. For every story, write the sentence beginning "I decided" or "I built" before you rehearse it, and be able to defend each choice.

Strong coding answers but thin infrastructure depth in CI/CD and containerization

Candidates report that this gap sinks otherwise strong engineers. Build a multi-stage Dockerfile, a GitLab CI pipeline and a small Helm chart yourself, and be able to explain every line and what happens when each fails.

Naming orchestration or architecture options without trade-offs

For orchestration strategies and for service-oriented versus monolithic designs, state the decision criteria (team size, deploy frequency, failure isolation, operational load) and give a case from your own work where you chose one.

Giving a vague answer on integrating with COTS, GOTS or FOSS products

Prepare one case with the interface or contract, the adapter you wrote, how you tested against the product, a limitation you could not change, and how you handled version or licence constraints.

Telling values stories that are abstract or embellished, or that reveal sensitive details

Use a real dilemma with a decision and a cost for each of honesty, integrity and accountability. Describe past work precisely but without restricted specifics, and say plainly when you do not know something.

Questions & practice

Archive a resource graph without breaking live references or recursing

Medium · graph traversal · topological order · tenant isolation

Resources reference other resources within a tenant; for the largest tenant the reference table holds up to 2,000,000 nodes and 8,000,000 edges. Archiving a resource must archive everything reachable from it that nothing outside the set still references, refuse when a live external referrer exists, and terminate when references form cycles, which they legitimately do. Produce the archive order and the refusal list, targeting O(V+E). Say what stops the traversal crossing a tenant boundary, and why recursion is the wrong control structure at this size.

Approach

Load the subgraph with the tenant predicate on both endpoints of the edge, not only on the side you started from. Scoping the left table alone is the classic cross-tenant leak: one mis-entered edge then pulls another tenant's resources into the traversal and, worse, into the archive.
Traverse iteratively with an explicit stack. A 2,000,000-node graph can hold a chain deep enough to exhaust a native stack in the low tens of thousands of frames, and that failure is a process crash rather than an error you can return.
Treat cycles as data rather than corruption: compute strongly connected components with Tarjan in O(V+E) using its own explicit stack, then condense. The condensation is a DAG, so a topological order over it gives the archive order, and every member of a component archives in one transaction because no order within a cycle is valid.
Decide refusals with reverse edges. A candidate is archivable only if every in-edge originates inside the candidate set, so build the transpose or count in-degrees restricted to the visited set, and emit each blocked resource with the id of the external referrer, which is the only part of the answer an operator can act on.

Follow-up

The graph is read in one query and the archive writes a minute later. What can change in between, and how do you make the write safe?
The candidate set is 400,000 resources. Is that one transaction, and if not, what does a half-finished archive look like to a reader?

Collapse a redelivered event batch into per-aggregate high-water marks

Easy · hashing · at-least-once · aggregation

You drain a batch of up to 5,000,000 events, each (aggregate_id BIGINT, aggregate_version INT, event_type, payload). The log guarantees order within one aggregate only; the batch merges 64 partitions, and a relay failover has redelivered a range, so an older version for an aggregate can appear after a newer one. Given a map of last_applied_version per aggregate, produce the events worth applying, at most one per (aggregate_id, version), plus the count discarded. Target O(n) time. State the memory for 2,000,000 distinct aggregates and what you do when it does not fit.

Approach

One pass, one hash map from aggregate_id to the highest version kept, and a discard counter. An event whose version is at or below last_applied_version for its aggregate is dropped without further work, which is the whole reason the event carries its version rather than a delta. O(n) expected time, O(d) space in distinct aggregates.
Keep the maximum, never the last occurrence. The redelivered range means the final appearance of an aggregate in the batch can be an older version than one seen earlier in the same batch, so last-wins applies stale state over newer state and the projection regresses with no error anywhere.
Cost the memory instead of calling it large: an 8-byte key plus a 4-byte version is 12 bytes of payload, and an open-addressed table held at a 0.7 load factor costs roughly 17 bytes per entry before per-slot metadata, so 2,000,000 aggregates is tens of megabytes in a native layout and several times that in a runtime that boxes both key and value.
If the distinct set exceeds memory, partition on hash(aggregate_id) mod P and reduce each partition independently. Every event for one aggregate hashes to the same partition, so the per-partition result is exact and the merge is concatenation rather than a second reduction.

Follow-up

The payload is a patch rather than a snapshot, so applying only the highest version loses the intermediate changes. What changes in your reduction?
How do you detect that version 7 arrived while version 6 was never delivered, and what should the consumer do about the gap?

Identify the heaviest tenants in a five-minute window under memory pressure

Medium · top-k · heavy hitters · streaming

The edge service handles about 3,000 requests per second across roughly 50,000 tenants, peaking near 9,000. Expose the 50 heaviest tenants by request count over the trailing five minutes so limits can be tightened before one tenant's backfill starves the fleet. You may not retain five minutes of raw records. Give the exact solution and its memory, then the bounded-memory approximation with its error stated as a formula, and say which you would ship and at what tenant cardinality that choice changes.

Approach

Do the exact version first, because it is affordable at this cardinality: a ring of 300 one-second counters per tenant, advanced lazily, is 1,200 bytes of counters per tenant and roughly 60 to 90 MB for 50,000 tenants with overhead. Carry a running total and subtract the bucket you overwrite so a window read is O(1) rather than 300 adds.
Extract the top 50 with a size-k min-heap over the tenant sums: O(d log k) for d tenants, against O(d log d) to sort them all. Maintaining the heap continuously instead of on query requires a tenant-to-heap-index map, because incrementing a count already inside the heap means sifting from a known position, and without that map you rebuild the heap on every request.
State the approximation precisely rather than gesturing at sketches. Misra-Gries with m counters retains every item whose true count exceeds N/(m+1), and each retained count underestimates the truth by at most N/(m+1). With m = 1,000 and N = 900,000 requests in the window the error is roughly 900 requests, which is fine for spotting a tenant sending 50,000 and useless for ranking two tenants 200 apart.
Say what breaks when the window slides: Misra-Gries and Space-Saving are insert-only and cannot be decremented as records age out. The workable construction is one summary per sub-window, say ten seconds, with 30 summaries merged at query time, and the merged error is the sum of the per-summary errors, so the bound degrades linearly in the number of sub-windows.

Follow-up

The heaviest tenant is heavy because of one export job rather than user traffic. Should the limiter treat those as the same tenant?
Two tenants sit tied at the boundary of the top 50. Does your answer flap, and does the flapping matter?

Explain why the owner filter ignores the listing index

Medium · composite index · query plans · sort elimination

The only index on resource is (tenant_id, status, updated_at DESC, resource_id DESC). A new endpoint returns one user's resources across all statuses, newest created first: WHERE tenant_id = $1 AND owner_user_id = $2 ORDER BY created_at DESC LIMIT 20. On a tenant with 2M rows it takes 900 ms and EXPLAIN shows a sort above a large scan. Explain precisely why the existing index cannot serve it, give the index that can, and state which of these the new index still will not help: owner_user_id alone across tenants; the same query ordered by updated_at. PostgreSQL 16.

Approach

Separate the two jobs an index does. For filtering, a composite btree is seekable only on a left prefix, so with no predicate on status the scan can at best range over tenant_id and test owner_user_id per row; PostgreSQL 16 has no btree skip scan to jump the unconstrained column.
For ordering, the index is sorted by (status, updated_at) within a tenant and not by created_at, so the LIMIT cannot stop early: every matching row is read and then sorted. That is the 'Sort Method: top-N heapsort' line, and it is why the plan reads 2M rows to answer with 20.
Derive the replacement from the access path — equality, equality, then the ordering column: CREATE INDEX CONCURRENTLY ON resource (tenant_id, owner_user_id, created_at DESC). The scan seeks to the (tenant, owner) range and walks 20 entries in order, so the Sort node disappears along with the row-read.
Treat INCLUDE (title, status) as conditional, not free. An index-only scan still visits the heap for any row whose page is not marked all-visible, so on a table taking 1.2k writes/second the win depends on autovacuum keeping the visibility map current, and the wider index costs more on every insert.

Follow-up

90% of rows are status='active'. Would a partial index WHERE status = 'active' change your answer, and for which of the three queries?
A dashboard runs this for 40 owners in one page load. What changes about the design?

Stop tag and share joins from fanning out a page

Medium · join fan-out · semi-joins · many-to-many

resource_tag is (resource_id, tag_id) with PK (resource_id, tag_id); resource_share is (resource_id, shared_with_user_id, permission). The tagged-and-shared listing inner-joins resource to both, filters tenant_id, tag_id = ANY($2) and shared_with_user_id = $3, orders by updated_at DESC and takes 50. Pages come back with fewer than 50 distinct resources and the total in the header is far too high. Explain the row multiplication, rewrite both the page query and the count query so each is correct, and name the index each one needs. PostgreSQL 16.

Approach

Do the arithmetic against the predicates that are actually there. An inner join emits one row per matching child row, and both joins are filtered: tag_id = ANY($2) admits only the requested tags, shared_with_user_id = $3 admits one user's share rows. So a resource holding three of the requested tags and shared with $3 once yields three rows, not one — the multiplier is its count of matching tags times its share rows for that single user, and that second factor is 1 unless the table admits duplicate (resource_id, shared_with_user_id) pairs. LIMIT 50 then limits rows rather than resources, and COUNT(*) counts pairs — the header is the product, not the population.
Reject DISTINCT as the fix. It deduplicates after the product has been built, so the planner must materialise and sort the fanned-out set before the LIMIT can apply, and it leaves any SUM or AVG in the same select list wrong.
Rewrite both filters as semi-joins, keeping resource as the only row source: AND EXISTS (SELECT 1 FROM resource_tag rt WHERE rt.resource_id = r.resource_id AND rt.tag_id = ANY($2)) and the same shape against resource_share. A semi-join stops at the first match per resource and preserves the driving index order, so ORDER BY updated_at DESC, resource_id DESC LIMIT 50 still stops after 50 rows.
Count with the same predicates and no join at all: SELECT count(*) FROM resource r WHERE r.tenant_id = $1 AND r.status = 'active' AND EXISTS (...) AND EXISTS (...). Nothing multiplies a resource, so the number is the population.

Follow-up

The filter changes from 'any of these tags' to 'all of these tags'. Rewrite it and state what it costs relative to the ANY form.
A resource can be shared with the same user twice under different permissions. Does your count change, and should it?

How do you approach creating and maintaining Helm charts for Kubernetes deployments?

Medium · Technical & Domain Expertise

Approach

Lay out the chart: Chart.yaml with a chart version (SemVer, bumped on every chart change) separate from appVersion, values.yaml holding safe defaults, templates/ for manifests, and _helpers.tpl for named templates that generate names and labels consistently.
Keep environments apart with override files (values-dev.yaml, values-prod.yaml) passed with -f; later files win. Add values.schema.json so a mistyped or missing value fails at install time instead of producing a broken manifest.
Validate before anything reaches a cluster: helm lint, then helm template piped into a schema validator such as kubeconform against the target Kubernetes version, and run both in CI on every merge request. Pin images by tag or digest, never latest.
Release with helm upgrade --install --atomic (which implies --wait and rolls back on failure), use the helm-diff plugin to review changes, and keep helm rollback <revision> as the recovery path. Keep secrets out of plain values files (SOPS, Sealed Secrets or an external secrets operator).
Name the edge cases: CRDs placed in crds/ are installed but never upgraded or deleted by Helm, and Deployment spec.selector is immutable, so changing selector labels breaks upgrades. In a disconnected cluster, charts and images must be mirrored into an internal registry.

Follow-up

An upgrade failed halfway and the release is stuck in a pending state; how do you recover it?
How would you structure charts for ten related services: one umbrella chart with subcharts, a library chart, or separate charts?
How do you deliver a chart and its images to a cluster with no internet access?

What are the trade-offs you consider when choosing between different container orchestration strategies?

Medium · Technical & Domain Expertise

Approach

Name the real options first: Docker Compose on a single host, Docker Swarm, self-managed Kubernetes (kubeadm, k3s, RKE2), managed Kubernetes or a managed container service such as ECS, and HashiCorp Nomad. Then say which constraints decide between them.
Weigh operating cost against capability. Kubernetes brings self-healing, autoscaling, rolling updates and a large ecosystem (Helm, operators), but someone must upgrade the control plane, back up etcd and manage networking. Compose is simple but has no multi-node scheduling or failover.
Treat managed versus self-managed as its own trade-off. A managed control plane offloads upgrades and availability, but it may not exist on a disconnected or accredited network, where you run your own distribution, mirror every image and own every patch.
Match the strategy to the workload: stateless services suit Deployments; databases need StatefulSets and persistent volumes or should stay outside the cluster; batch work maps to Jobs and CronJobs. Also compare rollout strategies: rolling update (no extra capacity, slower rollback) versus blue/green (double capacity, instant switch) versus canary (needs traffic splitting and metrics).
Close with a decision rule tied to an example from your own work: the number of services and nodes, the team's operational skill, and the environment's constraints, and what you would revisit as it grows.

Follow-up

When would you argue against adopting Kubernetes at all?
Would you run the database inside the cluster or outside it, and why?
How does rollback time and resource cost differ between rolling, blue/green and canary deployments?

How have you utilized Bash scripting to automate routine system operations or CI/CD tasks?

Medium · Technical & Domain Expertise

Approach

Pick one concrete script and describe it end to end: what triggered it (cron, systemd timer, CI job), its inputs, what it changed, and how you knew it worked. Examples: rotating and pruning logs, backing up a database to object storage, or building, tagging and pushing an image in CI.
Show defensive habits: set -euo pipefail, quote every expansion ("$var", "$@"), use mktemp for temp files and trap cleanup on EXIT, parse flags with getopts, and run shellcheck in CI. Know the caveat: set -e does not fire for commands in if/while conditions, in && or || lists except the last command, or after !.
Make it safe to rerun: check state before acting (does the file, user or tag already exist), use mkdir -p and atomic mv into place, support a --dry-run flag, return meaningful exit codes and log to stderr with timestamps.
For CI/CD, explain that a non-zero exit from a script line fails a GitLab CI job, so keep logic in versioned scripts in the repo that you can also run locally, rather than long inline YAML, and test them with bats where it pays off.
Say where Bash stops being the right tool: once you need structured data beyond what jq handles, real error handling, retries or unit tests, you move the logic into Python and keep Bash as glue.

Follow-up

What failures does set -e not catch, and how do you handle them?
How do you process filenames that contain spaces or newlines safely?
At what point did you, or would you, rewrite that script in Python?

Explain a scenario where you had to integrate custom software with existing GOTS or COTS products.

Medium · Technical & Domain Expertise

Approach

Set up the scenario precisely: which product it was (government off-the-shelf or commercial), what interface it exposed (REST or SOAP API, SDK, file drop, shared database, message bus), what you could not change (vendor code, licence terms, its accreditation), and what your component had to do.
Describe the integration pattern: an adapter or anti-corruption layer that translates the product's data model and formats into your own, so product quirks stay in one module, plus contract tests against a sandbox instance or recorded responses.
Cover failure handling: timeouts on every call, retries with exponential backoff and jitter only for idempotent operations, a deduplication key so a retry cannot apply twice, and a dead-letter queue for records the product rejects, with alerting.
Address versioning and upgrades: pin the product version you integrated against, run the integration suite against each new release before rollout, and isolate version differences behind the adapter. Mention licence compatibility when FOSS is involved (copyleft versus permissive).
End with your own contribution and a measurable result: records processed, error rate before and after, or manual steps removed, using I for what you personally built and decided.

Follow-up

A new release of the product changed its behaviour; how did you find out, and what broke?
How did you test the integration without access to a production instance of the product?
What did you do when the product had a defect you could not fix yourself?

Walk me through how you would design a scalable, containerized service for processing high-volume data flows.

Medium · System Design & Architecture

Approach

Clarify requirements before drawing: peak and average record rate, record size, latency target versus throughput, whether ordering matters (and per what key), delivery semantics (at-least-once or effectively-once), retention, and which sources and sinks are involved.
Outline the pipeline: ingest endpoint or collectors, then a durable log or queue such as Kafka, then stateless worker containers in a consumer group, then sinks (PostgreSQL, MongoDB or object storage). The log decouples producers from consumers and lets you replay from an offset.
Explain scaling and its limits: run workers as a Kubernetes Deployment with a HorizontalPodAutoscaler on CPU or on consumer lag (for example through KEDA). Within one consumer group at most one consumer reads each partition, so partition count caps parallelism, and the downstream database is often the next bottleneck.
State correctness choices: with at-least-once delivery, duplicates happen, so make writes idempotent (upsert on a record key) and commit offsets only after a successful write. Use a partition key to keep per-key ordering, and route poison messages to a dead-letter topic instead of blocking the partition.
Cover operations: readiness and liveness probes, CPU and memory requests and limits, graceful shutdown on SIGTERM that finishes in-flight work and commits offsets, and metrics for lag, throughput and error rate with alerts on sustained lag.

Follow-up

Traffic grows ten times and lag keeps rising even after you add pods; what is limiting you?
How would you reprocess a full day of data after fixing a bug in the transform?
How would you guarantee a record is not written twice to the sink?

How do you ensure software quality in a CI/CD pipeline when deploying to a secure environment?

Medium · System Design & Architecture

Approach

Walk the pipeline stage by stage: lint and static analysis, unit tests, build the container image once, integration tests against that image, then security checks: SAST, dependency scanning, an image scan (for example Trivy) and an SBOM for the artifact.
Promote the same artifact, never a rebuild: tag and reference the image by digest, sign it (for example with cosign), and verify the signature and checksum before deploying, so the thing that was tested is provably the thing that runs.
Handle the secure-environment constraints: mirror base images and package dependencies into internal registries, assume deployment may cross a network boundary by transfer rather than direct push, and keep secrets out of the repo using protected and masked CI variables or a secrets manager.
Add gates and recovery: protected branches, required reviews and a passing pipeline before merge, a documented exception path for findings you cannot fix yet, post-deploy smoke tests in a staging environment that mirrors production, and a tested rollback (helm rollback or redeploying the previous digest).
Name the trade-off: every added check lengthens feedback, so run fast checks on every commit and heavier scans on merge or on a schedule, and track pipeline duration alongside defect escapes.

Follow-up

A critical CVE appears in your base image with no patched version available; do you block the release?
How do you run integration tests when CI cannot reach the target environment?
How would you show an auditor exactly what code and dependencies are in a deployed image?

Describe your process for translating a high-level architectural diagram into functional code.

Medium · System Design & Architecture

Approach

Read the diagram as a list of contracts: each box becomes a deployable or a module with one owner, and each arrow becomes an interface with a protocol, payload, direction, sync or async behaviour, and auth. Write down every arrow that lacks an error case, a volume estimate or a security boundary, and resolve those questions with the architect.
Write the interfaces first: OpenAPI specs, protobuf or JSON schemas and the core data models, checked into the repo, plus short architecture decision records for choices the diagram leaves open.
Build a walking skeleton: one thin request path through every component, deployed through the real CI/CD pipeline with stubs behind it, so integration and deployment problems show up in the first week instead of the last.
Fill in vertical slices ordered by risk: the component with the most uncertainty (an unfamiliar external system, the highest data rate, a security boundary) gets built and load-tested first.
Keep the diagram honest: maintain it as code (C4 with PlantUML or Mermaid) next to the services and update it in the same merge request that changes a contract.

Follow-up

Halfway through, you find the diagram is wrong; what do you do?
How would you split this work across three engineers without them blocking each other?
Which component would you build first, and why that one?

How do you approach hardware and software trade-offs during the early stages of a project?

Medium · System Design & Architecture

Approach

Start from measurable requirements: data rate, latency budget, data volume and retention, availability, and physical or environmental limits (size, weight, power, cooling, disconnected operation), plus budget and procurement lead time, since hardware choices are slow to reverse.
Do back-of-envelope sizing aloud: rate times retention gives storage (100 MB/s for a day is 100 x 86,400 = 8.64 TB), records per second times CPU cost per record gives cores, and peak traffic gives network bandwidth. Add headroom and say how much.
Compare the options on throughput per watt and per dollar against development cost: general-purpose CPUs are flexible and fast to write for; GPUs suit parallel numeric work; FPGAs or ASICs give the best throughput per watt for fixed signal-processing pipelines but take longer to develop and change. Also compare cloud against on-premises when the network permits.
Reduce risk with evidence: benchmark a prototype on representative data, profile to see whether the bottleneck is compute, memory, disk I/O or network, and decide on that, not on vendor specifications.
Keep software portable so the hardware decision can change: containers, an abstraction over hardware-specific code, and a written record of which decisions are expensive to reverse (procurement) and which are cheap (software configuration).

Follow-up

When would you move a processing step from software on CPUs to a GPU or an FPGA?
The hardware is already purchased and the software cannot keep up with the data rate; what are your options?
How do you size a system before you have any hardware to benchmark on?

Read latency spikes on a sixty-second sawtooth

Medium · cache stampede · single-flight · ttl

The cached listing read path serves about 14k reads/second at an 85% hit rate. p99 sits at 35 ms for 57 seconds, jumps to 900 ms for 3, and repeats. During each spike the primary shows several hundred identical listing queries starting within the same millisecond, all carrying one large tenant's id. Cache entries use a 60-second TTL. Give the mechanism, the ordered checks, the fix, and the correctness hazard your fix must not introduce.

Approach

Match the period to a configured number before theorising about load. A spike every 60 seconds against a 60-second TTL is an entry expiring, and you confirm it by correlating spike timestamps with the entry's write time rather than with the traffic curve. If the period had matched a cron or a GC interval instead, this is a different investigation.
Establish the concurrency of the miss. Several hundred identical queries in one millisecond means the miss path has no coalescing: every request that arrives between expiry and repopulation recomputes. The herd size is that key's arrival rate times its recompute time, so at 1.2k reads/second for the hot key and a 250 ms recompute you expect about 300 concurrent misses, which matches what is observed.
Add single-flight on the miss path so one caller per key recomputes under a short-lived lock while the rest wait for its result. Prefer stale-while-revalidate where the read tolerates it: return the expired value immediately and refresh asynchronously, which removes the latency spike rather than serialising it into a queue of waiters.
De-synchronise the keys. Write TTLs with jitter, for example 60 seconds plus or minus 10%, so a deploy or a mass invalidation does not align every key on the same second and turn a per-key herd into a fleet-wide one.

Follow-up

The same sawtooth appears on a key that is invalidated on write rather than expired. Is that the same bug?
How does your answer change if the recompute takes 4 seconds instead of 250 ms?

Suggested practice checklist

The plan starts with the infrastructure topics candidates report as the weak spot (Docker, Kubernetes, GitLab CI, Bash), moves to databases and design, then builds the behavioral story bank, and ends with a full spoken run-through.

Day 1 — Skills inventory and Python/Bash baseline

Write the three reported rounds on one page, then list each skill the role names (Python, Linux CLI, Bash, Docker, Kubernetes, GitLab CI, MongoDB or PostgreSQL) and note one project of yours as evidence for each; mark the gaps.
Write a short Python 3 script that parses a log file and counts events per key, then redo it as a Bash pipeline.

Deliverable: A skills-and-evidence page with gaps marked, plus a Python script, a Bash pipeline and one solved practice drill.

Day 2 — Containers, Kubernetes and Helm

Write a multi-stage Dockerfile with a non-root user and a small final image, and note what each layer does.
Create a Helm chart with helm create, move environment settings into values.yaml, and run helm template and helm lint.
Break a deployment on purpose three ways (bad image tag, failing start command, impossible resource request) and write the diagnosis steps for each.
Answer aloud, comparing orchestration strategies and their trade-offs.

Deliverable: A Dockerfile, a linted Helm chart, a three-case troubleshooting checklist and a spoken answer on the Helm and orchestration questions.

Day 3 — Bash automation and GitLab CI

Write a .gitlab-ci.yml with build, test, scan and deploy stages, including caching and a manual approval gate for deploy.
Write an idempotent Bash deployment script with set -euo pipefail, a trap for cleanup and meaningful exit codes.
Prepare your answer on quality gates in a pipeline for a secure environment, covering what blocks a release and what evidence the pipeline stores.

Deliverable: A pipeline file, a deployment script and two spoken answers: Bash automation and CI/CD quality.

Day 4 — Databases and query optimization

On a sample collection, run explain("executionStats") before and after adding a compound index in equality, sort, range order.
Run EXPLAIN ANALYZE on a PostgreSQL query with a filter that ignores its index and fix it.
Write your MongoDB optimization story with your own before and after numbers.

Deliverable: Before and after plans for one MongoDB and one PostgreSQL query, two solved SQL drills and a written MongoDB story.

Day 5 — System design and trade-offs

Draw a containerized service for high-volume data flows: ingest, queue, workers, storage and deployment, then list what fails first as volume grows.
Write the steps you take to turn an architecture diagram into code: interfaces, repository layout, stubs, tests, first slice.
Write a hardware versus software trade-off list for an early project, including what you would measure first.

Deliverable: Two annotated diagrams, a diagram-to-code checklist, a trade-off list and two solved practice drills.

Day 6 — Integration story and behavioral story bank

Write your COTS, GOTS or FOSS integration story: the interface, your adapter, testing, a limitation and the outcome.
Write one story each for the late project, the ambiguous requirements, the technical disagreement and the speed-versus-testing balance.
Map your stories to honesty, integrity and accountability, and make sure each value has a real dilemma behind it.

Deliverable: An integration story and four behavioral stories, each with a decision, a cost and an outcome, mapped to the three values.

Day 7 — Full spoken run-through

Answer aloud, without notes: the Helm question, the container design question and the CI/CD quality question.
Answer aloud: the disagreement story and the speed-versus-testing question, using the first person throughout.
Redraw one diagram on a whiteboard or digital equivalent while narrating it, then fix the weakest answer from the week.

Deliverable: Recorded spoken answers to five questions, one redrawn diagram and a list of the single weakest answer to rework.

Values and leadership stories

Candidates report that the final rounds assess behavioral fit and alignment with honesty, integrity and accountability, and that mid-to-senior candidates are expected to show leadership, mentoring and clear communication. The reported prompts ask for stories about a late project, unclear requirements, a technical disagreement and the speed-versus-testing trade-off, plus a question about MongoDB query optimization that is answered as an experience story. In every story, make your own decision, its cost and its result explicit.

Tell me about a time you had to explain a technical choice to a non-technical stakeholder.

Can you describe your experience optimizing complex queries in a NoSQL database like MongoDB?

Medium · Technical & Domain Expertise

Approach

Choose one slow MongoDB query you personally fixed and open with the baseline: the query shape, the collection's rough size and the latency you measured from the profiler or slow-query log. Use your own real numbers and never invent them.
Describe the diagnosis. Run explain("executionStats") and compare totalDocsExamined and totalKeysExamined against nReturned, then look for a COLLSCAN or an in-memory SORT stage. A large gap between examined and returned documents is the signal that the index does not fit the query.
Describe the fix: a compound index ordered equality fields, then sort fields, then range fields; a projection that lets the index cover the query; moving $match and $sort to the front of an aggregation; replacing skip-based paging with a range on an indexed key; avoiding unanchored regex and $where.
State the cost and the proof. Every index slows writes and takes memory, so say how you checked for unused indexes ($indexStats), how you built the index with production in mind, and the before and after explain output that showed the change worked.
Close with what you would do differently, and say which result you owned versus what a teammate or DBA did.

Follow-up

How did you decide the field order in the compound index?
What did the new index cost in write throughput and memory, and how did you check?
How would the same diagnosis look in PostgreSQL with EXPLAIN ANALYZE?

Describe a time you had to provide technical leadership on a project that was falling behind schedule.

Medium · Behavioral & Values-Based

Approach

Open with the slip in concrete terms: what was due, what the gap was and what caused it (scope growth, an external dependency, an estimate that was wrong). State causes without blaming a person.
Show the leadership moves in order: list remaining work, find the critical path, and agree with the customer or stakeholder which scope to cut, defer or simplify. Then remove the top blocker, and pair or reassign work to match skills.
Describe the communication. Say how you told the customer or stakeholder early, what you reported (new date, what was dropped, risks) and how often you updated them. This is the place to show honesty and accountability.
Give the outcome with its cost: what shipped and when, what was deferred, and what quality checks you refused to skip. Finish with the process change that would have caught the slip earlier.
Avoid two failure modes: a story about working extra hours with no change in plan, and a story where you describe the team's actions but not your own decisions.

Follow-up

How did you decide what to cut, and who had to agree?
How and when did you tell the customer the date had moved?
What signal would have told you earlier that the project was slipping?

How do you handle ambiguity when requirements are not clearly defined by the customer?

Medium · Behavioral & Values-Based

Approach

Pick a real case where the customer's requirement was vague or conflicting and you had to move anyway. Name what was unknown, not just that it was unclear.
Show the sequence: separate facts from assumptions, write the assumptions down, and ask the smallest set of questions that would change the design. Say who you asked and what you did when no one could answer.
Reduce risk before certainty arrives. Describe a thin vertical slice, a prototype, an interface contract or a mock that let the customer react to something concrete, and which choices you kept reversible.
Show the paper trail: the assumptions and decisions you recorded and the confirmation you sought, so a later change in requirements could be traced to a decision rather than argued from memory.
Say what happened when an assumption was wrong and how cheap the correction was. Avoid answers that are only 'I ask lots of questions' or that end with building silently to a guess.

Follow-up

What did you do when the customer could not answer your question?
How did you record what you assumed, and who saw it?
Tell me about an assumption that turned out wrong and what it cost.

Tell me about a time you disagreed with a technical decision made by your team; how did you resolve it?

Medium · Behavioral & Values-Based

Approach

Choose a disagreement about a real technical decision, such as a data store, a deployment approach or a framework, with consequences you can describe. Skip personality clashes.
State the other side's position fairly, in terms they would accept, then state yours. Name the criteria that mattered (performance, maintainability, risk, delivery date) so the dispute reads as a trade-off.
Describe the evidence you used to resolve it: a benchmark, a spike, a failure scenario, a documented constraint. Say who made the final call and how.
Show accountability afterwards. If the decision went against you, describe how you committed and helped it succeed. If it went your way, describe how you checked that it was right. If you were wrong, say so plainly.
Avoid stories where you won by persistence, where the colleague is cast as the problem, or where the resolution was never tested.

Follow-up

What would you have done if the evidence had been inconclusive?
Looking back, was the team's decision or your position the better one?
How did the working relationship look afterwards?

How do you balance the need for rapid deployment with the requirement for rigorous testing?

Medium · Behavioral & Values-Based

Approach

Reject the either-or framing and give a principle: put checks in proportion to risk. Automated unit, integration and security checks on every merge block a release; slower or broader checks can run on a schedule or after release.
Describe how you make the fast path safe: small changes, a GitLab CI pipeline with build, test, scan and deploy stages, feature flags, canary or rolling updates on Kubernetes, health checks and a rehearsed rollback (helm rollback or kubectl rollout undo).
For a secure environment, add that the pipeline should produce evidence, such as test reports, scan results and image digests, because releases there often need review.
Give one real example where you held a release for testing and one where you shipped faster, each with the reasoning and the result. Say what you would not skip under any deadline.
Avoid claiming you test everything every time, or saying you ship first and fix later; both signal poor judgment.

Follow-up

Tell me about a time you shipped with too little testing and what followed.
Which checks would you refuse to skip, and why those?
How would you shorten a pipeline that has become too slow to run on every merge?