Design an LLM serving platform

Step 1 Understand it

Why interviewers ask this. It tests modern AI infrastructure: GPU cost, batching, memory limits, streaming, and fairness between teams.

Watch it happen

Requirements

Estimates

• A 70B-parameter model in bf16 needs about 140 GB just for weights, so each replica spans several GPUs; KV-cache memory then limits how many requests run at once.

API

• POST /v1/chat {model, messages, max_tokens, stream} → a stream of tokens

Data model

• Model registry (versions and hardware needs); a usage record for every request.

Start simple, then scale

1. Version 1: one model on one GPU server behind an API, handling one request at a time.
2. Version 2: static batching, then continuous batching with a paged KV cache.
3. Version 3: a gateway with quotas and routing, and autoscaled GPU pools per model.
4. Version 4: quantised tiers, prefix caching, speculative decoding and priority queues.

Picture it

flowchart LR
  C["Apps"] --> GW["Gateway<br/>auth · quotas · routing"]
  GW --> P1["GPU pool: small model<br/>batching + KV cache"]
  GW --> P2["GPU pool: large model<br/>batching + KV cache"]
  P1 -->|"stream tokens"| GW
  P2 -->|"stream tokens"| GW
  AS["Autoscaler"] -.-> P1
  AS -.-> P2

The design, step by step

1. A gateway handles authentication, quotas, rate limits and routing to a pool of servers per model.
2. Model servers (vLLM-style) use continuous batching and a paged KV cache to keep the GPUs busy.
3. GPU pools scale automatically on queue length and latency.
4. Responses stream back through the gateway; prefix caching reuses work for repeated system prompts.
5. Metrics: time to first token, tokens per second, GPU utilisation, errors, and usage per team.

Deep dives

• Prefill (reading the prompt) and decode (generating one token at a time) have different bottlenecks.
• Continuous batching adds new requests between decode steps instead of waiting for a whole batch to finish.
• Quantised, smaller or distilled models for cheaper tiers; speculative decoding for speed.
• Overload: a bounded queue, and dropping low-priority traffic first.

Common mistakes

• Handling one request at a time on an expensive GPU.
• Ignoring KV-cache memory when planning capacity.
• No per-team quotas, so one team can starve the others.

Step 2 Go further (free, optional)

Optional. The notes above are written to be enough on their own. These free resources give a second explanation and extra practice.

• Read

  Building A Generative AI Platform

  Chip Huyen's blog · free

• Read

  How continuous batching enables 23x throughput in LLM inference

  Anyscale blog

• Paper

  Efficient Memory Management for LLM Serving with PagedAttention (2023)

  Kwon et al. · arXiv · optional

Step 3 Practise

Try each exercise yourself first: on paper, out loud or in code. Then open the answer and compare it with yours.

1. ExplainSay it in plain words in under a minute, with no notes.

   Explain prefill and decode, and why they behave differently.

   Show answer

   Generating a reply has two phases.

   Prefill reads the prompt. All the prompt's tokens are already known, so the model processes them all at once, in parallel: big matrix multiplications that keep the GPU's compute fully busy. It produces the KV cache (the stored attention keys and values for every prompt token) and the first output token. Its cost grows with the prompt's length, and it sets the time to first token. It is limited by compute.

   Decode then generates the answer one token at a time, each depending on the one before. Each step does very little arithmetic per sequence (one new token), but must read all the model's weights and that sequence's whole KV cache from GPU memory. So decode is limited by memory bandwidth, not compute: one sequence alone leaves most of the GPU idle. It sets the speed of the stream (tokens per second).

   That is why they are optimised differently: decode gets continuous batching (many sequences share each pass over the weights, so the memory read is shared), and memory-efficient KV caches (paged attention) so more sequences fit. Prefill gets chunking (long prompts split so they don't stall everyone's decoding), prefix caching for repeated system prompts, and sometimes its own separate GPUs.

2. DrawSketch the diagram (or class diagram) from memory.

   Draw gateway → router → GPU pools with batching servers, the autoscaler and metrics.

   Show answer

   The gateway handles everything about who may call and how much; the router chooses a pool by model and priority. Each pool's servers batch continuously, and the autoscaler watches queue length and latency, not just GPU use, because time to first token is what users feel.

   textCopy

   Clients ── POST /v1/chat (stream) ──► Gateway
                                          auth, quotas per team,
                                          rate limits, metering
                                            │
                                            ▼
                                          Router
                                          by model, version, priority
                      ┌─────────────────────┼─────────────────────┐
                      ▼                     ▼                     ▼
               Pool: large model     Pool: small model     Pool: embeddings
               ┌───────────────┐     ┌───────────────┐
               │ model server  │     │ model server  │   each server:
               │ (vLLM-style)  │ ... │               │   bounded queue,
               │ continuous    │     │               │   continuous batching,
               │ batching,     │     │               │   paged KV cache,
               │ paged KV cache│     │               │   prefix cache
               └───────────────┘     └───────────────┘
                      ▲                     ▲
   Autoscaler ── adds/removes GPU replicas per pool
                 (on queue length, time to first token)
   Metrics ◄── TTFT, tokens/s, GPU use, errors, usage per team
   Tokens stream back: server ══► gateway ══► client

3. ApplyUse it in a real design question and say exactly where it fits.

   Estimate how many 4,000-token requests fit in 20 GB of KV-cache memory, if the model needs about 0.5 MB of cache per token.

   Show answer

   Each 4,000-token request needs 4,000 × 0.5 MB = 2,000 MB = 2 GB of KV cache when full. 20 GB ÷ 2 GB = 10 requests at once.

   What that means: the GPU can only run 10 such conversations in parallel, however much compute it has to spare; the KV cache, not compute, sets the batch size. That is why the main levers are about memory: paged attention allocates the cache as tokens are actually generated (most requests never reach their maximum, so far more than 10 fit in practice); prefix caching stores a shared system prompt's cache once for everyone; quantising the KV cache to 8 bits halves it (20 at once); and models with grouped-query attention need much less cache per token in the first place.

   pythonCopy

   kv_memory_mb = 20 * 1000                         # 20 GB of KV cache
   per_request_mb = 4000 * 0.5                      # 4,000 tokens × 0.5 MB each
   print(per_request_mb, kv_memory_mb // per_request_mb)      # 2000.0 10.0: 10 requests at once
   print(kv_memory_mb // (per_request_mb / 2))                # 20.0 with an 8-bit KV cache

4. Trade-offsSay what it costs, when not to use it, and the alternative.

   One big model for everything, or routing between a small and a large model?

   Show answer

   One big model for everything:
   • The best quality on every request, one model to evaluate, deploy and monitor, and consistent behaviour.
   • But it is expensive and slow for the many requests that are easy (classify this ticket, extract a date, a short factual answer): you pay large-model prices and latency for small-model work, and capacity runs out sooner.

   Routing between a small and a large model:
   • Much cheaper and faster on average, because most traffic is usually easy enough for the small model, and the big GPUs are saved for hard requests.
   • But you need a router: rules (by task type, by product feature) or a small classifier, and it will sometimes send a hard question to the small model, which quietly lowers quality. A cascade (try small; escalate if it is unsure) adds latency to escalated requests.
   • Two fleets to run, two models to evaluate, and answers that vary in style.

   A sensible path: start with one strong model to learn the real traffic and set a quality bar with evaluations; then route the clearly easy, high-volume tasks to a small model, and check on the same evaluation set that quality holds.

Step 4 Interview-ready, then prove it

Your interview answer

About 20 to 40 seconds, said out loud. Practise it until it sounds like you, not like notes.

A gateway handles auth, quotas and routing to a GPU pool per model. Each replica runs an inference server with continuous batching and a paged KV cache to keep the GPUs busy, and streams tokens back. Pools scale on queue length and latency, and we track time to first token, tokens per second and cost per team. Quantisation, prefix caching and speculative decoding cut cost and latency.

Follow-up questions

Tick each check when you can do it without notes