API Gateway vs Load Balancer: What Each Does and When You Need Both

You are two minutes into a system design interview, you have drawn a box labeled "LB / gateway," and the interviewer asks: which one is it? Here is the short answer. A load balancer distributes connections across identical copies of a server and cares only that each copy is healthy. An API gateway reads each request and makes decisions about it: who is calling, whether they are allowed, which of several different services should handle it, and what to do when that service misbehaves. Large multi-service stacks typically run both, in that order. The rest of this page is the long answer, taught through questions companies actually asked.

Key Takeaways

• A load balancer answers "which copy of the same thing takes this connection." An API gateway answers "what is this request, and what policy applies to it." If your backends are interchangeable, you want the first; if they are distinct services behind one front door, you need the second.
• The real dividing line is the network layer. An L4 balancer forwards TCP segments and never sees a URL. Routing on paths, verifying a JWT, or returning a 429 all require parsing HTTP, which means an L7 component.
• The canonical edge order is DNS → L4 load balancer → gateway fleet → services. The load balancer sits in front because the gateway itself needs to be balanced: it is just another horizontally scaled service.
• Where you reject excess traffic changes its cost. A request refused at the gateway consumed one cheap hop. The same request refused inside the service already crossed two proxies, took a connection from a pool, and occupied a worker thread.
• "Do I need both?" is a live interview trap. One stateless service with auth in the app needs only a load balancer. Adding a gateway there is pure cost: extra latency and a second system to misconfigure, in exchange for nothing.

A load balancer spreads traffic; a gateway makes decisions about it

Asked at Anthropic — Design a scalable, reliable system An open-ended canvas: pick a domain such as a content feed, ride matching, or file storage, and carry it from requirements through a complete architecture, with an edge tier fronting several backend services and storage and messaging behind them. The follow-ups probe whether you can defend every box you drew, and the gateway box gets challenged more than any other.

This question is where the vocabulary gets tested. Candidates draw the standard diagram, and the follow-up is almost always some version of "what exactly does that gateway box do that the load balancer doesn't?"

Split the responsibilities cleanly:

The load balancer's job is distribution and failover across a pool of identical backends. It health-checks each member (an HTTP probe to /healthz every few seconds, marking a backend down after 2–3 consecutive failures), picks a member per connection or per request using an algorithm like round-robin or least-connections, and drains connections when you take an instance out for deploys. It has no opinion about what the traffic means. That indifference is a feature: it is why an L4 balancer can push millions of concurrent connections on modest hardware.

The gateway's job is per-request policy across a set of different backends. It terminates TLS, parses the HTTP request, authenticates the caller, checks a rate limit, matches the path and method against a routing table (/v2/orders/* → orders service, /v2/users/* → users service), rewrites the request, and forwards it. It can also version APIs, canary a percentage of traffic to a new service build, aggregate two backend calls into one client response, and translate protocols. REST outside, gRPC inside is a common arrangement.

The common mistake in this interview is treating the two as interchangeable synonyms for "the thing in front." When the interviewer pushes ("your gateway is overloaded, what do you do?"), the right instinct is to say the gateway is itself a stateless fleet behind a load balancer, and you scale it horizontally like any other service. Candidates who conflated the two boxes get stuck here, because in their mental model there is nothing in front of the gateway to spread the new instances' load.

	Load balancer	API gateway
Core question	Which healthy copy takes this?	What is this request allowed to do, and where does it go?
Backends	Interchangeable replicas	Distinct services
Typical layer	L4 (TCP) or L7 (HTTP)	Always L7
Understands URLs, headers, bodies	Only in L7 mode	Yes, by definition
Owns auth, quotas, API keys	Not its job (bolt-ons exist: ALB OIDC, nginx auth_request)	Yes, as its core job
Request transformation, versioning	No	Yes
State it keeps	Health of pool members, connection table	Routing table, rate-limit counters, credential/key config
Failure blast radius	All traffic to that pool	Every API behind it, plus policy bugs (bad auth config)

One table is enough; the rest of the distinctions fall out of the layer question, which is the next section.

L4 vs L7 is the actual dividing line

Asked at BitGo — Compare core system design tradeoffs with scenarios A rapid-fire tradeoffs round for a web-scale service, covering monolith vs microservices, database families, sync RPC vs events, REST vs gRPC, caching write policies, consistency models, and partitioning schemes. The balancing item asks you to choose among round-robin, least-connections, and consistent hashing, and to produce a scenario where each one is the right call.

Nearly every gateway-vs-balancer confusion dissolves once you anchor on the OSI layer, because the layer determines what information the component can even act on.

Layer 4. An L4 balancer operates on TCP/UDP. It sees source and destination IP and port, and nothing else — no paths, no headers, no cookies, because it never parses (and usually never decrypts) the byte stream. It forwards segments by rewriting addresses (NAT mode) or, in direct-server-return setups, only touches the inbound half of the connection. The payoff is throughput: no TLS termination, no HTTP parsing, and in kernel-space implementations like IPVS the forwarding never leaves the kernel. AWS NLB with plain TCP listeners and nginx in stream mode are L4. (NLB can also terminate TLS for you; the moment it does, it is doing more than pure L4 forwarding.)

Layer 7. An L7 component terminates the TCP connection and the TLS session, parses the HTTP request, and only then decides what to do. That decision can use everything in the request: path, method, Host, arbitrary headers, even the body. AWS ALB, nginx with location blocks, Envoy, and every API gateway product are L7.

Here is the same tool doing each job, which makes the difference concrete:

# L4: nginx as a TCP balancer (stream module).
# It forwards bytes. It cannot route /orders differently
# from /users because it never sees a URL.
stream {
  upstream gateway_pool {
    least_conn;
    server 10.0.1.11:8443 max_fails=3 fail_timeout=10s;
    server 10.0.1.12:8443 max_fails=3 fail_timeout=10s;
  }
  server {
    listen 443;
    proxy_pass gateway_pool;
  }
}

The balancing algorithms from the BitGo question live at either layer, and the scenario logic is worth having ready:

• Round-robin when requests are uniform in cost and backends are identical. It degrades when one slow request pins a backend, because round-robin keeps feeding it.
• Least-connections when request cost varies (some queries take 10ms, some take 2s). Connection count is a cheap proxy for load, so slow backends automatically get fewer new requests.
• Consistent hashing when you want the same key to land on the same backend: cache affinity, per-user session state, shard ownership. This one is doing something categorically different, and it gets its own section below.

Now the trap this section exists for: "L7 load balancer" and "API gateway" are not the same thing, even though both parse HTTP. Be careful how you state the difference, because the boundary is about ownership, not raw capability. An ALB can authenticate callers with its authenticate-oidc listener action, and nginx can gate requests with auth_request and throttle clients with limit_req, so claiming an L7 balancer "cannot do auth" is factually wrong and an interviewer who has configured either will catch it. What the balancer does not do is own that policy. It picks a target group per request, with auth features bolted onto the listener; it is not where API-key lifecycles, per-tenant quota configuration, request rewriting, or versioned routing tables live. A gateway is an L7 proxy built around exactly that policy engine. The dividing question to offer in an interview: "is this component the system of record for who may call what, and how much?" If yes, it is doing gateway work, whatever the box is labeled.

Video: System Design Interview: When to use API Gateway, Load Balancer, or Reverse Proxy?

A real request path runs both, and you should know what each hop costs

Asked at Adobe — Analyze end-to-end request latency You are handed a single web request whose journey crosses a CDN, an edge proxy tier, several microservices, caching layers, databases, and outside vendor APIs. The job is to build a latency model: work out where the milliseconds actually go, identify which hops sit on the critical path, and argue for the fixes with the best payoff.

Adobe's prompt treats "load balancer / API gateway" as a single tier, which reflects how the edge actually looks. Here is the shape to draw:

The order is not arbitrary. The L4 balancer fronts the gateway fleet because the gateway is stateless and horizontally scaled, and something has to spread connections across its instances and route around a dead one. In cloud terms this pair is often NLB → Envoy fleet, or the cloud provider fuses the two into one managed product (an ALB doing L7 balancing with an API gateway service behind or beside it). Fused or not, both roles exist.

For the latency model, be able to put numbers on each hop:

• Client to CDN edge: dominated by the client's last mile. A cold TLS 1.3 handshake adds one round trip; session resumption saves the certificate validation and key-exchange work but still costs that round trip, unless 0-RTT early data is enabled, which many deployments avoid because of replay concerns.
• CDN to origin balancer: cross-region RTT if your origin is far from the edge; this is frequently the largest single term for global users.
• L4 balancer: effectively free on the wire, tens of microseconds; it is forwarding packets, not proxying requests.
• Gateway: a real hop. TLS termination, HTTP parse, auth check, route match — typically 0.5–2ms in-VPC if connections to upstreams are pooled. That conditional is the point.
• Service, cache, DB: where the rest of the budget goes, and where the HTTP-level debugging guide picks up.

The named failure mode to bring up: connection churn between tiers. If the gateway opens a fresh TCP+TLS connection to the upstream per request instead of reusing a keep-alive pool, every request pays 1–2 extra round trips plus a TLS handshake, and under load the service drowns in handshakes rather than work. The same applies gateway-side: an L4 balancer with very short idle timeouts silently kills pooled connections, and the first request on each dead connection eats a retry. When an interviewer asks "where did the p99 go," stale connection pools and idle-timeout mismatches between tiers are a strong specific answer — far better than "add a cache."

The gateway owns cross-cutting policy: auth, quotas, versioning

Asked at Anthropic — Design a low-latency ML inference API Design a real-time inference API with explicit p50/p95 latency SLOs and availability targets. The prompt names authentication, rate limiting, and multitenancy as first-class requirements, alongside load balancing, a stateless API tier, model serving, batching, caching, and canary rollouts.

Notice what this prompt did: it listed auth, rate limiting, and multitenancy next to load balancing, as separate line items. That is the interviewer telling you these are different concerns that live in different places. The gateway is where the first three consolidate, because implementing per-client policy inside each of a dozen services means a dozen inconsistent implementations and a dozen places to get a security bug.

A single gateway route, sketched as config, shows the full per-request pipeline:

# One gateway route. Everything here requires reading the
# request — none of it can live in an L4 balancer.
routes:
  - name: infer-v2
    match:
      path_prefix: /v2/infer
      methods: [POST]
    plugins:
      - jwt_auth:
          required: true
          forward_claims_header: X-Tenant-Id   # signed, not client-settable
      - rate_limit:
          key: tenant                          # per-tenant, not per-IP
          limit: 600/min
          on_exceed: 429
    upstream:
      service: model-serving
      timeout_ms: 800          # inside the p95 SLO, not above it
      retries:
        count: 1
        on: [connect_failure]  # only if the request never reached
                               # the service; POST /infer is not
                               # idempotent

And the per-request flow the config implies:

Four details in that config are interview gold because each encodes a real production incident class:

The identity header is gateway-injected, not client-supplied. If services trust an X-Tenant-Id header and the gateway does not strip the client's own attempt to set it, any caller can impersonate any tenant. Either the gateway signs what it forwards, or services re-verify the JWT themselves. Deciding which component is authoritative for auth, and saying it out loud, separates candidates who have run multi-tenant systems from those who have read about them.

The rate-limit key is the tenant, not the IP. For an ML API, one tenant behind a NAT can be a thousand IPs, and a thousand tenants can share one egress IP. Per-IP limiting fails in both directions. This is also why the limit must live at the gateway or above: only a component that has already authenticated the request knows the tenant.

The upstream timeout is inside the SLO. An 800ms upstream timeout under a 1s p95 target leaves room to fail fast and return a clean error. Candidates routinely set timeouts above their SLO, which means by the time the timeout fires, the SLO is already blown and the client has probably given up.

Retries fire only if the request never reached the service. POST /v2/infer is not idempotent: if the upstream accepted the request and the connection then broke, a blind retry runs the same inference twice and bills the tenant twice. Restricting the retry to connection failures, where the request never left the gateway, is the safe default. The stronger design, worth saying out loud, is an idempotency key: the client sends a unique key per logical request, the service deduplicates on it, and retrying on 502/503 becomes safe.

Rejecting work: which layer drops the request changes what it costs

Asked at TikTok — Design overload protection with load shedding Keep a high-traffic service inside its p99 SLO while traffic spikes. The prompt wants admission control and token buckets, priority queues, deadlines and timeouts, a comparison of shedding strategies (drop-new, drop-tail, random, deadline-aware), and an explicit position on which rejections belong at the load balancer and which belong in the application.

That last clause is the whole question for our purposes: TikTok is directly asking where in the stack rejection should happen. The answer is a gradient, and the gradient is cost-versus-information:

• At the L4 balancer, shedding is nearly free but nearly blind. You can cap connections per backend or per source, and the kernel's SYN backlog protects against connection floods. But the balancer cannot distinguish a health probe from a checkout request — it drops by connection, not by meaning.
• At the gateway, you know the route, the client, and the priority. This is where token buckets per tenant live, where you return 429 with a Retry-After header, and where you can shed /recommendations while protecting /checkout. A rejected request here cost one proxy hop and a counter increment: no service thread, no DB connection.
• In the application, you have the most information (estimated request cost, current queue depth, remaining deadline) and the highest price for using it. The cost-aware half of deadline shedding lives here, because only the app knows its own queue depth and how long its work typically takes; with deadline propagation (a grpc-timeout header, for instance) a gateway can drop already-expired requests too, but it cannot judge whether the remaining budget covers the work. Every request shed at this layer already crossed two proxies and consumed a worker slot.

So the design that answers the question: cheap, coarse limits at the edge; per-client fairness at the gateway; deadline-aware and cost-aware shedding in the app as the last line. Each layer sheds what only it can see.

The failure mode to name is retry amplification. If the client retries twice, the gateway retries twice, and the service's internal call retries twice, one failing request becomes up to 27 attempts — a self-inflicted spike precisely when the system is least able to absorb it. The fix is a retry budget (retries as a capped percentage of traffic, the Envoy approach) and retrying at one layer only, ideally with the gateway honoring the request's overall deadline.

Whether the spike itself arrives as a flood of pulls or a fan-out of pushes is its own design axis; the push vs pull architecture guide covers that half.

Key-based routing is a third thing, and it is neither of the first two

Asked at SoFi — Scale a key-value store with consistent hashing A single-server key-value store has to scale across many machines. Design the partitioning with consistent hashing: how clients or a gateway route requests, virtual nodes, minimal data movement on node joins and leaves, replication and quorums, plus hot keys and load skew.

Here the backends are not interchangeable — server 7 owns keys that server 12 does not have. Put a round-robin balancer in front of this pool and 1-in-N requests succeed. Candidates make exactly this mistake on the whiteboard: they draw the load-balancer box in front of stateful shards out of habit, and the interviewer asks "so a GET for key user:42 lands on a random node... then what?"

What this tier actually needs is a router: a component that computes hash(key) and maps it to the owning shard on the consistent-hash ring. The SoFi prompt names both places it can live:

• A smart client embeds the ring in a client library, saving a network hop. Cost: every client language needs the library, and ring updates must propagate to every client. A slow-updating client sends traffic to a node that gave up the range.
• A gateway/router tier centralizes the ring. Clients stay dumb, ring changes deploy in one place, and you pay one extra in-VPC hop (sub-millisecond) for the privilege. This is a gateway in the routing sense — it makes a per-request decision based on request content — but it does no auth or quota work.

Virtual nodes (each physical server appears at 100–200 points on the ring) smooth the load distribution, and a node join moves only ~1/N of the keys instead of rehashing everything. If a single key gets hot enough to melt its owner, hashing cannot save you: the fixes are key-splitting (hotkey#1..k sub-keys) or a read cache in front, a tradeoff family the DynamoDB data-modeling guide works through in the context of partition design.

The takeaway for the comparison in this article's title: load balancers assume any backend can serve the request; key-based routers know exactly one group can. An API gateway can host either behavior, because it reads the request, which is the recurring theme.

When a plain load balancer is enough

Asked at Salesforce — Architect serving and storage for recommender Sketch the serving side of a recommendations product: how clients fetch data, what the API looks like, and how a fleet of identical stateless app servers runs behind a single fronting proxy with caching and explicit latency targets. Then design storage schemas for users, items, and interaction logs so that nightly batch training and live online serving can share the same data without interfering.

Salesforce's own prompt asks for a reverse proxy or load balancer — no gateway — and that is the correct architecture for what it describes: one service, stateless replicas, auth handled in the application. Proposing an API gateway here is a negative signal, because you would be adding:

• A hop. Another 0.5–2ms of proxy latency plus another connection pool to tune, on a path with explicit latency targets.
• A config surface. Routing tables, plugin config, and credential stores that can be misconfigured. Gateway config changes have edge-wide blast radius; a bad deploy takes down every API at once instead of one service.
• A failure domain. The gateway fleet is one more thing that pages you at 3am.

The honest heuristic to offer in an interview: a gateway earns its hop when at least two of these hold — multiple distinct backend services behind one public host; multiple client types (web, mobile, third-party API consumers) needing different auth schemes; centralized quotas or API keys; edge-level versioning or canary routing. One stateless service with none of those needs a balancer, full stop.

The same discipline applies to internal east–west traffic. Service-to-service calls inside the cluster generally should not funnel back through the edge gateway — that turns it into a chokepoint and doubles its traffic for no policy benefit. Internal traffic gets client-side load balancing or a service-mesh sidecar, which is "a tiny L7 proxy per pod": gateway-like machinery, deliberately decentralized. Saying that one sentence in an interview signals you know the pattern has a scope, not just a name.

FAQ

Is an API gateway the same as a load balancer?

No. A load balancer distributes connections across identical copies of one service and only tracks which copies are healthy. An API gateway parses each HTTP request and applies policy: it authenticates the caller, enforces rate limits, and routes to different services based on path, method, or headers. In most production stacks the two are separate tiers, with a load balancer in front of a horizontally scaled gateway fleet.

Does an API gateway replace a load balancer?

Not in a full architecture. The gateway is itself a stateless service running as multiple instances, and something has to spread connections across those instances and route around a dead one. That something is a load balancer, usually L4. Managed cloud products sometimes fuse the two roles into one offering, but both jobs still exist inside it.

Is a load balancer L4 or L7?

It can be either. An L4 balancer forwards TCP or UDP traffic based on IP and port and never sees URLs or headers, which makes it extremely fast. An L7 balancer terminates TLS, parses HTTP, and can route per request, for example sending /api/* and /static/* to different pools. AWS NLB with TCP listeners is L4, AWS ALB is L7, and nginx can run in either mode.

What is the difference between an L7 load balancer and an API gateway?

Both parse HTTP, and the boundary is ownership rather than raw capability: an L7 balancer picks a target group per request, and some can even authenticate callers (AWS ALB has an OIDC listener action; nginx has auth_request). A gateway is built around the policy engine itself: it owns API-key management, per-client and per-tenant quotas, request rewriting, and versioning or canary routing as its primary job. A useful test is whether the component is the system of record for who may call what; if yes, it is doing gateway work.

In which order do the load balancer and API gateway sit in a request path?

Load balancer first, gateway second. The typical path is client to CDN to L4 load balancer to gateway fleet to backend services. The balancer's job at that position is to spread connections across gateway instances and fail over when one dies; the gateway then makes the per-request decisions before forwarding to the right service over pooled connections.

When do I need an API gateway for microservices?

When cross-cutting policy would otherwise be duplicated: multiple distinct services behind one public host, multiple client types with different auth schemes, centralized quotas or API keys, or edge-level versioning and canary routing. If you run a single stateless service that handles its own auth, a plain load balancer is the better design; a gateway there adds latency and operational surface without buying anything.