2nd Watch Software Engineer Interview Guide

The 60-second brief

2nd Watch builds and maintains cloud-based solutions for client businesses in several industries, and candidates describe projects that range from infrastructure automation to application development. The responsibilities candidates describe for a Software Engineer include cloud application and service work, code review and troubleshooting, in collaboration with teams such as product management, operations and customer support. The requirements candidates describe name Python, Java or C#, at least one of AWS, Azure or GCP, Docker and Kubernetes, and familiarity with Agile and DevOps practice, with Terraform or CloudFormation, data structures and algorithms, and software testing listed as extras.

This guide is for engineers interviewing for the Software Engineer role at 2nd Watch, particularly those whose background is application development and who need to show working knowledge of cloud infrastructure, or infrastructure engineers who need to show software discipline. It covers the two reported stages, every reported question with a question-specific approach and follow-ups, the topics candidates report (Terraform, cloud computing, Infrastructure as Code, AWS and EC2 instance management), and a week of preparation you can run on a personal AWS account.

Candidates report two stages over roughly 2-4 weeks: an Initial Screening Call, then Technical Interviews, which can be one interview or several. Candidates describe the technical interviews as fair and centred on real-world applications rather than abstract algorithms.

Focus areas

• Terraform
• Cloud Computing
• Infrastructure as Code (IaC)

Overview

The reported questions fall into three groups. Technical and domain questions ask how you would launch an EC2 instance, how IaaS, PaaS and SaaS differ, what your Infrastructure as Code and Terraform experience is, how you secure cloud applications, and what hard technical problem you have solved. Problem-solving cases ask how you would optimise a cloud application's performance, how you would spend a limited budget, and how you traded technical debt against a deadline. Behavioral questions cover a difficult teammate, prioritising across projects, leading a project and handling criticism.

Because candidates describe the technical interviews as practical rather than algorithmic, the useful depth to build is operational: knowing what a security group, an IAM instance profile, a Terraform state file or an Auto Scaling group actually does, and being able to say what you would check, in what order, when something is slow or broken. Hands-on practice beats reading here. Launching, breaking and tearing down a small stack on your own AWS account with both the CLI and Terraform gives you the concrete details (flags, resource names, failure modes) that a spoken answer needs.

The requirements list containers (Docker, Kubernetes) and DevOps practice, and other questions candidates mention ask about your preferred languages, debugging in a cloud environment and CI/CD. The plan below includes a container build and deployment, a pipeline outline and a cloud-debugging sequence for these. Be ready to discuss every tool on your resume, so audit it before the screening call and remove or downgrade anything you cannot discuss at the level of a specific incident or design decision.

Data structures and algorithms appear only as a nice-to-have, so keep coding preparation to a refresh of core patterns and spend most of the week on cloud fundamentals, Terraform, containers, security, performance diagnosis and a set of behavioral stories told in the first person with measurable outcomes.

The 2nd Watch interview process

Candidates report two stages, an Initial Screening Call followed by Technical Interviews, across roughly 2-4 weeks. The technical stage may be a single interview or several.

Initial Screening Call

Candidates describe the Initial Screening Call as first contact, used to assess fit and discuss the role. Be ready to connect your resume to the cloud and language requirements candidates describe: which cloud platforms you have shipped on, which of Python, Java or C# you write day to day, and where you have used Docker, Kubernetes or Terraform, each with one concrete example. Use the call to ask what the technical stage looks like for your team, including how many technical interviews to expect.

What to demonstrate

• Fit with the role, as candidates describe the purpose of the call
• Discussion of the role itself
• How you describe your background and interest in the role

How to prepare

• Write a short background pitch that names the cloud platform, language and IaC tool you know best and one system you built with them.
• Audit your resume: for every listed technology (AWS, Terraform, Docker, Kubernetes, CI/CD), note one specific thing you built, fixed or decided with it, and cut anything you cannot back up.
• Prepare questions about the technical stage: how many interviews, whether any involve live coding, and which cloud platform the team works in.

Technical Interviews

Candidates report one or more interviews on technical skills and problem-solving capabilities, and describe them as fair and focused on real-world applications rather than abstract algorithms. Candidates do not say which question comes up in which interview, so prepare the whole reported set below, give each answer a clear structure, and back it with something you have actually built or run.

What to demonstrate

• Technical skills applied to real-world scenarios, as candidates describe
• Problem-solving capabilities, which candidates name as a focus of the stage

How to prepare

• Practise the cloud, Terraform, security, performance, budget and technical-debt questions below aloud, each with a structure: prerequisites-then-launch for EC2, layers for security, measure-then-change for performance, and symptom, hypotheses, root cause and measured result for the hard-problem story.
• Launch an EC2 instance three ways (console, aws ec2 run-instances, a Terraform aws_instance resource) and be able to explain every input: AMI, instance type, subnet, security group, instance profile, user data.
• Build a small Terraform project with a remote S3 backend using native state locking (use_lockfile = true on Terraform 1.10 and later; a DynamoDB lock table is the older, deprecated pattern), then practise plan, apply, import and destroy so you can describe state handling from experience.
• Containerise a small service with a multi-stage Dockerfile and a non-root user, push it to ECR, run it on ECS or EKS, and be ready to explain a Kubernetes Deployment, Service, liveness and readiness probes, and a rolling update.

Where candidates lose points

Describing EC2 creation as clicking through the console wizard, with no mention of subnets, security groups, IAM instance profiles or a repeatable method.

Practise the launch with the CLI and with Terraform until you can name every input and why it is set, and add how you would make it production-ready with a launch template, Auto Scaling group and multiple Availability Zones.

Listing Terraform, Docker, Kubernetes or a cloud certification on the resume and then having no answer about state locking, import, drift, probes, rolling updates or a real incident.

Audit the resume before the screening call. For each tool, prepare one concrete thing you built or fixed and the details behind it; remove anything you cannot discuss in detail.

Answering the cloud security question with a list of terms (encryption, IAM, firewalls) and no structure or example.

Answer by layer (identity, network, data, workload, detection), name the specific AWS control for each, and finish with an example such as replacing static keys with roles or your response to a leaked credential.

Jumping straight to "add more instances" or "add a cache" on the performance and budget cases.

State the metric and baseline first, find the bottleneck layer, then choose the fix and its trade-off. On the budget case, clarify what the budget covers and protect must-haves before naming cost levers.

Behavioral and trade-off stories told from the team's point of view, with no decision of your own and no measured outcome.

Write each story with the decision you made, what it cost, and the number that shows the result. For the technical-debt story, include how the debt was tracked and paid down.

Questions & practice

Identify the heaviest tenants in a five-minute window under memory pressure

Medium · top-k · heavy hitters · streaming

The edge service handles about 3,000 requests per second across roughly 50,000 tenants, peaking near 9,000. Expose the 50 heaviest tenants by request count over the trailing five minutes so limits can be tightened before one tenant's backfill starves the fleet. You may not retain five minutes of raw records. Give the exact solution and its memory, then the bounded-memory approximation with its error stated as a formula, and say which you would ship and at what tenant cardinality that choice changes.

Approach

1. Do the exact version first, because it is affordable at this cardinality: a ring of 300 one-second counters per tenant, advanced lazily, is 1,200 bytes of counters per tenant and roughly 60 to 90 MB for 50,000 tenants with overhead. Carry a running total and subtract the bucket you overwrite so a window read is O(1) rather than 300 adds.
2. Extract the top 50 with a size-k min-heap over the tenant sums: O(d log k) for d tenants, against O(d log d) to sort them all. Maintaining the heap continuously instead of on query requires a tenant-to-heap-index map, because incrementing a count already inside the heap means sifting from a known position, and without that map you rebuild the heap on every request.
3. State the approximation precisely rather than gesturing at sketches. Misra-Gries with m counters retains every item whose true count exceeds N/(m+1), and each retained count underestimates the truth by at most N/(m+1). With m = 1,000 and N = 900,000 requests in the window the error is roughly 900 requests, which is fine for spotting a tenant sending 50,000 and useless for ranking two tenants 200 apart.
4. Say what breaks when the window slides: Misra-Gries and Space-Saving are insert-only and cannot be decremented as records age out. The workable construction is one summary per sub-window, say ten seconds, with 30 summaries merged at query time, and the merged error is the sum of the per-summary errors, so the bound degrades linearly in the number of sub-windows.

Follow-up

• The heaviest tenant is heavy because of one export job rather than user traffic. Should the limiter treat those as the same tenant?
• Two tenants sit tied at the boundary of the top 50. Does your answer flap, and does the flapping matter?

Archive a resource graph without breaking live references or recursing

Medium · graph traversal · topological order · tenant isolation

Resources reference other resources within a tenant; for the largest tenant the reference table holds up to 2,000,000 nodes and 8,000,000 edges. Archiving a resource must archive everything reachable from it that nothing outside the set still references, refuse when a live external referrer exists, and terminate when references form cycles, which they legitimately do. Produce the archive order and the refusal list, targeting O(V+E). Say what stops the traversal crossing a tenant boundary, and why recursion is the wrong control structure at this size.

Approach

1. Load the subgraph with the tenant predicate on both endpoints of the edge, not only on the side you started from. Scoping the left table alone is the classic cross-tenant leak: one mis-entered edge then pulls another tenant's resources into the traversal and, worse, into the archive.
2. Traverse iteratively with an explicit stack. A 2,000,000-node graph can hold a chain deep enough to exhaust a native stack in the low tens of thousands of frames, and that failure is a process crash rather than an error you can return.
3. Treat cycles as data rather than corruption: compute strongly connected components with Tarjan in O(V+E) using its own explicit stack, then condense. The condensation is a DAG, so a topological order over it gives the archive order, and every member of a component archives in one transaction because no order within a cycle is valid.
4. Decide refusals with reverse edges. A candidate is archivable only if every in-edge originates inside the candidate set, so build the transpose or count in-degrees restricted to the visited set, and emit each blocked resource with the id of the external referrer, which is the only part of the answer an operator can act on.

Follow-up

• The graph is read in one query and the archive writes a minute later. What can change in between, and how do you make the write safe?
• The candidate set is 400,000 resources. Is that one transaction, and if not, what does a half-finished archive look like to a reader?

Merge partitioned event streams into one ordered feed with bounded lateness

Hard · k-way merge · watermarks · out-of-order streams

The read-model service consumes 64 log partitions carrying about 4,000 events per second in total. Each partition is ordered within itself, but partitions drift by up to 30 seconds, and the activity feed must present a tenant's events in occurred_at order. Produce the merge. State its complexity, the buffer it requires in events and in bytes, what happens when one partition is idle, and what you do with an event that arrives after you have already emitted its position. Payloads average 1 KB.

Approach

1. Merge with a min-heap over the 64 partition heads keyed on (occurred_at, event_id): O(log P) per event and O(n log P) overall. The tie-break on event_id is what makes the output deterministic when two partitions carry the same millisecond, which matters because the feed is paginated and a non-deterministic order reorders pages under the reader.
2. Emitting the heap head is only correct once every partition has produced everything up to that timestamp, so the emit condition is a watermark: the minimum across partitions of the highest occurred_at seen, less the allowed lateness. Events are held until the watermark passes them, which is what turns individually ordered streams into a jointly ordered one.
3. Size the buffer from the lateness rather than guessing: 4,000 events per second times 30 seconds is 120,000 buffered events, and at 1 KB each about 120 MB of heap. That number is the real price of the ordering guarantee and belongs in front of whoever asked for it.
4. Handle the idle partition explicitly, because it fails the feed rather than corrupting it: a partition with no traffic never advances its own maximum, so the watermark freezes and output stops entirely. Either every partition emits a periodic idle marker carrying the broker's current time, or the watermark falls back to wall clock for a partition silent beyond a threshold.

Follow-up

• The lateness budget is raised to five minutes. What is the new buffer, and what besides memory changes?
• The consumer restarts. Where does it resume from, and what does the feed look like for the first 30 seconds?

Write the update path that detects a concurrent edit

Medium · optimistic concurrency · transactional outbox · audit history

resource carries version INT NOT NULL DEFAULT 1. resource_revision holds revision_id, resource_id, version, actor_user_id, change_kind, patch JSONB, request_id, created_at with UNIQUE (resource_id, version). outbox_event holds aggregate_type, aggregate_id, aggregate_version, event_type, payload, status. A PUT carries the version the client read. Write the exact statements for the single transaction that applies the edit, records the revision and enqueues 'resource.updated', and give the handler's branch on zero affected rows. Then say what PostgreSQL 16 does under READ COMMITTED when two of these updates hit one row at once.

Approach

1. One transaction, three writes, no network call inside it: UPDATE resource SET title = $3, version = version + 1, updated_at = now() WHERE resource_id = $1 AND tenant_id = $4 AND version = $2; then INSERT the resource_revision row at version $2 + 1; then INSERT the outbox_event row at the same aggregate_version. The event goes to a table rather than a broker because no transaction spans both.
2. Branch on the affected-row count before doing anything else. Zero has three causes — stale version, wrong tenant, row gone — so re-read once and map to 409 carrying the current version, or 404 for an id outside the caller's tenant, which also stops the endpoint confirming that another tenant's id exists.
3. State the engine behaviour instead of assuming it. Under READ COMMITTED the second UPDATE blocks on the row lock, and when the first commits PostgreSQL re-evaluates the WHERE clause against the newly committed row, so the version predicate now fails and the statement reports zero rows. Under REPEATABLE READ the identical collision raises SQLSTATE 40001 instead, so the handler must fold both shapes into one conflict response.
4. Keep UNIQUE (resource_id, version) even though the predicate already serialises writers. It is what makes a lost update unwritable if any other path ever reaches the revision table, and it converts a logic bug into 23505 rather than into a silently missing history row.

Follow-up

• A client sends the version it read ten minutes ago and the resource has moved three versions. What is in your 409 so it can resolve the conflict without a full re-fetch?
• Two editors, two disjoint fields, no overlap. Does your answer still refuse the second write, and should it?

Replace offset paging on the resource feed with keyset

Easy · keyset pagination · composite index · sort order

resource holds resource_id, tenant_id, owner_user_id, title, body_ref, version, status ('draft','active','archived','deleted'), created_at, updated_at, deleted_at, with an index on (tenant_id, status, updated_at DESC, resource_id DESC). The listing endpoint returns active resources for one tenant, newest update first, 50 per page, today with LIMIT 50 OFFSET n. Tenants reach page 400 and rows are created while they read. Write the keyset query, define what the cursor carries and how it is encoded, and say which part of the index each predicate uses. Assume PostgreSQL 16.

Approach

1. Name the two failures separately. OFFSET 20000 makes the server produce and discard 20,000 rows, so page cost grows with depth rather than with page size. Independently, any write that changes how many rows sort above the offset moves the window between two fetches, and the direction decides which anomaly you get: an insert lands at the head of updated_at DESC and pushes already-returned rows down past the boundary, so they are returned a second time; a delete above the offset, or a row whose updated_at is bumped above the cursor, pulls rows up and one is never returned at all. Nothing in the response reveals either.
2. Write the seek: WHERE tenant_id = $1 AND status = 'active' AND (updated_at, resource_id) < ($2, $3) ORDER BY updated_at DESC, resource_id DESC LIMIT 50. The row-value comparison is one index range rather than a disjunction, and both columns are NOT NULL, which is what makes that comparison well defined.
3. Map each predicate onto the index: tenant_id and status are equality on the leading columns, (updated_at, resource_id) is the range, and the ORDER BY matches the index order so no Sort node appears and the scan stops after 50 rows. The DESC in the definition only matters for mixed directions — a plain ascending btree on the same columns is read backwards for this query.
4. Put both sort columns in the cursor and nothing the client can tamper with into another tenant: base64 of (updated_at, resource_id), validated server-side, with tenant_id taken from the principal.

Follow-up

• The client asks for 'jump to page 400'. What do you offer instead, and what does the honest version cost?
• Sort order becomes user-selectable across four columns. How many indexes is that, and which would you refuse to add?

Explain how you would create an EC2 instance in AWS.

Medium · Technical / Domain Questions

Approach

1. Start with prerequisites before the launch call: region, a VPC and subnet (public for a bastion or web host, private for an app server behind a load balancer), and a security group that only opens the ports the workload needs, for example 443 from the load balancer's security group rather than 0.0.0.0/0, and no open SSH.
2. Name the launch inputs and why each matters: AMI (look it up by filter rather than hardcoding an ID), instance type family and size, EBS root volume type, size and encryption, an IAM instance profile so the instance never holds access keys, user data for bootstrapping, and tags for cost and ownership.
3. Show the repeatable way: aws ec2 run-instances --image-id --instance-type --subnet-id --security-group-ids --iam-instance-profile --user-data, and then the Terraform version, an aws_instance resource fed by a data "aws_ami" lookup, kept in version control and applied through plan review.
4. Cover access and hardening: SSM Session Manager instead of SSH key pairs, IMDSv2 required (metadata_options http_tokens = "required"), and patching through a new AMI rather than hand edits.
5. For production, move from a single instance to a launch template plus an Auto Scaling group across at least two Availability Zones behind a load balancer, and verify the launch with status checks, SSM connectivity and CloudWatch metrics.

Follow-up

• The instance in a private subnet cannot download packages. What do you check? (Route table, NAT gateway, security group egress, network ACLs.)
• How would you roll out a new AMI to an Auto Scaling group without downtime?
• Why use an instance profile instead of putting access keys in the instance's environment?

What are the key differences between IaaS, PaaS, and SaaS?

Medium · Technical / Domain Questions

Approach

1. Frame the three models by where the line of responsibility sits. With IaaS you manage the OS, runtime, patching and scaling on rented compute, storage and networking (EC2, Azure Virtual Machines, Compute Engine).
2. With PaaS the provider runs the OS and runtime and you deploy code and configuration (Elastic Beanstalk, Azure App Service, App Engine; managed databases such as RDS are often grouped here). With SaaS you consume a finished application and configure it (email, CRM or ticketing products).
3. Give the trade-off: moving from IaaS to SaaS you give up control and portability in exchange for less operational work, faster delivery and a different cost model (instance-hours versus per-request or per-seat pricing).
4. Place the in-between options, since they come up in real projects: managed Kubernetes (EKS, AKS, GKE) sits between IaaS and PaaS, and functions such as Lambda are often called FaaS or serverless.
5. Close with a selection example: a legacy application that needs a specific OS configuration goes to IaaS first, a stateless web service fits PaaS or containers, and commodity functions such as email are bought as SaaS.

Follow-up

• How does the security shared-responsibility split change between IaaS and PaaS, for example for OS patching?
• When would you move a workload from EC2 to a managed platform, and what would make you stay?
• Where do containers on Kubernetes and Lambda functions fit in this model?

How do you ensure the security of applications deployed in the cloud?

Medium · Technical / Domain Questions

Approach

1. Organise the answer by layer and name the control at each: identity (least-privilege IAM roles, no long-lived access keys, SSO and MFA for people), network (private subnets, tight security groups, WAF in front of public endpoints, TLS everywhere).
2. Data: encryption at rest with KMS-managed keys, encryption in transit, secrets in Secrets Manager or Parameter Store fetched at runtime rather than in code, images or environment files, and S3 Block Public Access on by default.
3. Workload and pipeline: patched base images, container image scanning, dependency and static analysis in CI, and policy checks on Terraform (Trivy, Checkov or OPA) so insecure infrastructure fails before apply.
4. Detection and response: CloudTrail on all accounts, GuardDuty or equivalent, AWS Config rules for drift from policy, centralised logs with alerts, and a rehearsed response for leaked credentials.
5. Anchor it with the shared-responsibility model and one example from your own work, such as replacing static keys with an instance role or locking down a public bucket.

Follow-up

• An access key has been pushed to a public repository. Walk through your response. (Deactivate and rotate it, review CloudTrail for its use, clean the history, add secret scanning.)
• How would you let an application on EC2 or Lambda read one S3 bucket without any stored credentials?
• How would you harden a Docker image and its Kubernetes Deployment (minimal base image, non-root user, image scanning, resource limits)?

Describe a challenging technical problem you solved in a past project.

Medium · Technical / Domain Questions

Approach

1. Pick a problem with real technical content, ideally in a cloud system: an intermittent failure, a performance cliff, a data-consistency bug or a migration. Say why it was hard (intermittent, cross-service, no obvious logs).
2. Walk the diagnosis as a sequence: the symptom and how it was measured, the hypotheses you formed, what you checked to rule each out (logs, metrics, traces, a reproduction), and the evidence that pinned the root cause.
3. Describe the fix and its trade-off, then how you verified it (metric before and after, a load test, an error rate) and what you added to prevent a repeat (an alarm, a test, a runbook).
4. Tell it in the first person: what you did and decided, separate from what the team did. A story where you cannot explain the root cause, or where someone else found it, undermines the answer.

Follow-up

• What did you rule out first, and why was that the cheapest check?
• What monitoring would have caught this earlier?
• If you met the same symptom tomorrow, what would you do differently?

Walk me through how you would approach optimizing a cloud application for performance.

Medium · Problem-Solving / Case Studies

Approach

1. Define the target first: which metric (p95 or p99 latency, throughput, error rate) for which user path, and the current baseline from APM, tracing (X-Ray, OpenTelemetry) or CloudWatch.
2. Locate the bottleneck layer before changing anything. Database: slow-query logs, missing indexes, N+1 queries, connection pool exhaustion, offset pagination on large tables. Compute: CPU or memory saturation, garbage-collection pauses. Network: chatty cross-service or cross-region calls, large payloads.
3. Match the fix to the bottleneck: indexes and query rewrites, caching (CloudFront for static or cacheable responses, ElastiCache or Redis for hot reads, with an invalidation strategy), read replicas, moving heavy work to a queue such as SQS, right-sizing or horizontal scaling with autoscaling policies.
4. Change one thing at a time, load test, compare with the baseline, and state the trade-off of each fix (cache staleness, replica lag, extra cost).
5. Mention guarding the gain afterwards: alarms on the target metric and a performance test in CI.

Follow-up

• p50 latency is fine but p99 is bad. Where do you look?
• The database is the bottleneck and the workload is read-heavy. Read replicas or a cache, and why?
• How would you confirm a scaling change did not just move the bottleneck elsewhere?

If given a limited budget for a project, how would you allocate resources?

Medium · Problem-Solving / Case Studies

Approach

1. Clarify the constraints first: what the budget covers (people, cloud spend, licences), what is fixed (deadline, scope, compliance), and which requirements are must-haves.
2. Spend first on the must-haves and the highest risks (security, data durability, the core user path), and keep a contingency rather than allocating everything up front.
3. Cut cloud cost with known levers: estimate with the AWS Pricing Calculator, right-size instances, autoscale instead of provisioning for peak, Savings Plans or Reserved Instances for steady baseline load, Spot for interruptible batch work, non-production environments shut down outside working hours, S3 lifecycle rules.
4. Weigh managed services against self-hosting: a managed database costs more per hour but saves engineering time, which is usually the larger budget line.
5. Make spend visible: cost-allocation tags, AWS Budgets alerts and a regular Cost Explorer review so overruns surface early.

Follow-up

• Halfway through, spend is well above plan. What do you do first?
• What would you cut first, and what would you refuse to cut?
• How would you justify spending on reliability or security when the budget is tight?

Describe a scenario where you had to make a trade-off between technical debt and delivering a feature on time.

Medium · Problem-Solving / Case Studies

Approach

1. Set up the situation: the deadline, the clean solution and the shortcut, and why the deadline mattered (a client commitment, a dependent launch).
2. Show the shortcut was a deliberate decision: you assessed its blast radius, contained it (feature flag, tests around the shortcut, an isolated module) and logged the debt with an owner and a date.
3. Give the outcome on both sides: the feature shipped, and when and how the debt was paid down, or the cost you paid because it was not.
4. Add the line you will not cross, such as security or data-integrity shortcuts, to show judgement. Avoid answers that treat debt as never acceptable or that end with the debt still unaddressed.

Follow-up

• How did you get the debt paid down instead of forgotten?
• How do you decide which debt is unacceptable to take on?
• How did you explain the trade-off to product or the client?

Edge instances grow 400 MB per hour until the nightly restart

Hard · memory leak · heap analysis · metric cardinality

Edge API instances start at 700 MB resident and grow about 400 MB/hour; a nightly rolling restart has hidden it for weeks. Growth continues unchanged when request rate halves overnight, p99 degrades in the last hours before an instance is recycled, and heap used immediately after a forced full GC rises monotonically. The service holds no product state. Name the discriminating measurement that separates the plausible causes, give the most likely cause, and give the fix and how you would verify it.

Approach

1. Separate resident memory from live heap first, because they fail differently. Resident size can grow from fragmentation, native buffers or thread stacks while the heap is flat; heap used after a full GC rising monotonically is the measurement that says objects are reachable and not being released. You already have it, so this is retention, not fragmentation, and that closes off half the candidate list.
2. Use the rate's independence from traffic as the discriminator. Growth that continues at half the request rate rules out per-request objects that are merely slow to collect and points at a structure that grows with distinct values observed rather than with call volume. Write the candidates that have that property: a metrics registry keyed on a high-cardinality label, an unevicted cache, an interner, a per-key lock map.
3. Take two heap snapshots an hour apart and diff by retained size, reading the dominator tree, not by allocation count or instance count. Expect one root holding a map with millions of entries, then follow the reference chain to the code that inserts and never removes. Allocation profilers point at churn, which is the wrong signal here.
4. The candidate that fits this service is an observability label carrying an identifier, such as a request path recorded before templating so that /v1/resources/48213 becomes its own metric series. That grows with distinct ids seen, is independent of rate, and explains the late p99 degradation, since GC cost rises with the size of the live set.

Follow-up

• Post-GC heap is now flat but resident size still creeps. What are you looking at, and does it matter?
• How would you have detected this before an OOM, given the nightly restart masked the trend?

Suggested practice checklist

The plan follows the two reported stages and the topics candidates report: resume and screening preparation first, then AWS compute, Terraform and CI/CD, security and containers, performance, debugging and cost cases, behavioral stories, and a final mock of both stages.

Day 1 — Resume audit and screening call

• List every technology on your resume and write one concrete example (built, fixed or decided) for each; remove anything you cannot defend.
• Write your background pitch naming your main cloud platform, language (Python, Java or C#) and IaC tool, plus one system you built with them.
• Write the questions you will ask on the screening call about the number and format of technical interviews and the team's cloud platform.

Deliverable: An audited resume with an example per technology, a written background pitch, and a list of screening-call questions.

Day 2 — AWS compute and service models

• On a personal AWS account, launch an instance in the console, then with aws ec2 run-instances, using an instance profile, a restrictive security group, IMDSv2 and SSM access; terminate it afterwards.
• Put the same instance behind a launch template and Auto Scaling group across two Availability Zones and note each component's purpose.
• Build a table comparing IaaS, PaaS and SaaS by who manages what, with two AWS or Azure examples per row and where Kubernetes and Lambda fit.

Deliverable: A CLI command and notes for a hardened EC2 launch, an Auto Scaling setup description, and a service-model comparison table.

Day 3 — Terraform, Infrastructure as Code and CI/CD

• Write a Terraform project for a VPC, security group and EC2 instance with an S3 remote backend using native state locking (use_lockfile = true, Terraform 1.10 and later); note that a DynamoDB lock table is the older, deprecated pattern you may still meet in existing code.
• Practise plan, apply, terraform import of a hand-made resource, and a for_each refactor; observe what a count index shift would recreate.
• Outline a CI/CD pipeline you have run: build, tests, dependency and image scan, Terraform plan on the pull request, deploy, and rollback to the previous image or release.
• Write your answer to the IaC experience question: scope, state handling, workflow, and one real incident.

Deliverable: A working Terraform repository, notes on import and for_each behaviour, a written CI/CD pipeline outline with its rollback step, and a written IaC experience answer.

Day 4 — Cloud security and containers

• Write a layered security checklist (identity, network, data, workload, detection) with the specific AWS service for each control, plus a step-by-step response for a leaked access key, and say both aloud.
• Write a Dockerfile for a small service with a multi-stage build and a non-root user, push the image to ECR, and run it on ECS or EKS.
• Write out how a Kubernetes Deployment, Service, liveness and readiness probes and a rolling update work, using your service as the example.
• Run Trivy (or Checkov) against your Terraform repository and Trivy against the container image, and fix what they flag.

Deliverable: A security checklist with a leaked-key runbook, a container image built from your Dockerfile and running on ECS or EKS from ECR, written Kubernetes notes, and a scan-clean Terraform repository and image.

Day 5 — Performance, debugging and cost cases

• Write a measure-first sequence for a slow cloud application and apply it to the keyset-paging practice question as a database example.
• Write a cloud-debugging sequence (CloudWatch Logs and metrics, X-Ray traces, SSM Session Manager into the instance, then recent deploys and config changes) and run it against the memory-growth debugging practice question.
• Outline a budget allocation for a small web project: must-haves, AWS cost levers, and how spend is tracked with tags and budget alerts.

Deliverable: A performance diagnosis sequence, a written keyset-paging solution, a cloud-debugging sequence applied to the memory-growth exercise, and a one-page budget allocation.

Day 6 — Behavioral and trade-off stories

• Write first-person STAR stories for the hard technical problem, the difficult teammate, leading a project, and the technical-debt trade-off, each with a measured outcome.
• Write your prioritisation method with one real example, and one example of feedback you acted on.
• Say each story aloud and rewrite any line where the team stands in for what you personally did.

Deliverable: Six written stories, each with your decision and a measurable result, rehearsed aloud.

Day 7 — Mock loop and coding refresh

• Run a mock screening call using your pitch and resume examples.
• Run a mock technical interview on the EC2 launch, security and performance questions plus your container, CI/CD and cloud-debugging answers, with a friend asking follow-ups.
• Refresh core data structures with one practice coding question, since the requirements list algorithms as a nice-to-have.

Deliverable: Notes from both mocks with weak answers marked and fixed, and one solved coding practice question.

Behavioural preparation

The reported behavioral questions for this role cover working with a difficult teammate, prioritising across several projects, taking the lead, and handling criticism, and the case questions add a technical-debt trade-off. Candidates describe client projects and work alongside teams such as operations and customer support, so prepare stories that show your own decisions, how you communicated trade-offs, and a measured result.

1. Describe a scenario where you had to make a trade-off between technical debt and delivering a feature on time.
2. Describe a time when you had to troubleshoot a significant issue in production.

Can you describe your experience with Infrastructure as Code (IAC) and tools like Terraform?

Medium · Technical / Domain Questions

Approach

1. Lead with a specific scope: what you managed with Terraform (VPCs, EC2 or ECS, IAM, RDS), how many environments, and whether you wrote modules or consumed them. Only claim what you have done, and be ready to give the details behind each tool.
2. Explain how state was handled: a remote backend such as S3 with native state locking (use_lockfile = true, Terraform 1.10 and later) or HCP Terraform, state encryption because it can hold secrets, and separate state per environment through directories or workspaces. If your project used a DynamoDB lock table, say so and note it is the older pattern, now deprecated in recent releases.
3. Describe the workflow: terraform fmt and validate in CI, terraform plan posted on the pull request for review, apply only from the pipeline after approval, provider and module versions pinned.
4. Show you know the sharp edges: count versus for_each (removing an item from a count list shifts indexes and recreates resources), lifecycle prevent_destroy and create_before_destroy, terraform import for resources created by hand, and drift from console changes.
5. Add one real incident, such as a plan that would have replaced a database because an immutable attribute changed, and how you caught and handled it. If you have used CloudFormation, contrast its managed state and change sets with Terraform's state file.

Follow-up

• A teammate's apply crashed and the state lock is stuck. What do you do?
• How would you bring a set of manually created resources under Terraform without recreating them?
• How do you structure modules so that dev, staging and production share code but not state?

Describe a time when you had to work with a difficult team member. How did you handle it?

Medium · Behavioral / Leadership

Approach

1. Describe the behaviour, not the person: a concrete pattern with a cost, such as reviews that stalled for days or infrastructure changes made in the console outside the pipeline.
2. Show what you did first: a private conversation to understand their constraints or reasons, then an agreed working rule (review turnaround, all changes through pull requests) rather than a complaint upward.
3. State the outcome with evidence (review time, fewer drift incidents, a delivered milestone) and where the working relationship ended up.
4. Avoid stories that blame the other person, end with a manager fixing it, or show no change in your own behaviour.

Follow-up

• What would you have done if the conversation had not changed anything?
• At what point would you escalate, and to whom?
• What did you learn about how you work with others?

How do you prioritize your tasks when working on multiple projects?

Medium · Behavioral / Leadership

Approach

1. Name your method concretely: list the commitments, then rank by impact, deadline and whether the work blocks someone else; unblocking others and production issues usually go first.
2. Show that you make the trade-off visible: when two items collide, you tell the stakeholders what moves and get agreement instead of silently slipping one.
3. Give a real instance with several workstreams in flight, what you deprioritised, how you communicated it, and how things turned out.
4. Avoid a tool-only answer ("I keep a task list") with no example of a hard call.

Follow-up

• Two stakeholders each say their work is the top priority. What do you do?
• A production incident lands in the middle of a deadline week. How do you re-plan?
• Tell me about a time your prioritisation was wrong. How did you notice?

Give an example of a situation where you took the lead on a project. What was the outcome?

Medium · Behavioral / Leadership

Approach

1. Explain why you took the lead: a gap in ownership, a deadline at risk or a technical area you knew best.
2. Cover the decisions you made: scope, design choices, how work was split, and the risk you managed first. Keep the pronoun "I" for your actions so your role is unambiguous.
3. Include one setback and how you handled it, then a measurable outcome (shipped date, cost or latency change, incidents avoided).
4. Close with what you would change; an answer with no reflection or no measurable result reads as thin.

Follow-up

• Which decision would you reverse if you ran the project again?
• How did you handle someone who disagreed with your plan?
• How did you hand the project off or keep it maintainable afterwards?

How do you approach feedback and criticism from peers or supervisors?

Medium · Behavioral / Leadership

Approach

1. Use a specific piece of feedback you received, such as a code review saying your Terraform module was too tightly coupled or your pull requests were too large to review.
2. Be honest about your first reaction, then show the process: clarifying what was meant, deciding what was valid, and changing your practice.
3. Give evidence the change stuck (smaller pull requests, a reworked module reused by other teams) and mention how you now give feedback in reviews.
4. Avoid feedback that is really a disguised strength, and avoid answers with no example at all.

Follow-up

• Tell me about feedback you disagreed with. What did you do?
• How do you give critical feedback on a colleague's code?
• How do you ask for feedback before a review forces it?