NVIDIA Dynamo Disaggregated Serving on Kubernetes: What a Credible vLLM Benchmark Requires

Why We Brought This Tool Into Our Lab#

We started evaluating NVIDIA Dynamo because aggregated vLLM deployments eventually force an awkward compromise. Prompt ingestion and token generation have different performance profiles, but an aggregated worker schedules both phases on the same GPU allocation.

That is acceptable for short prompts and predictable output lengths. It becomes harder to operate when retrieval-heavy requests arrive beside long-running generations. Prefill consumes compute in bursts, decode holds KV memory over time, and one scaling policy has to serve both behaviors.

Dynamo's disaggregated design separates those phases:

1. The frontend accepts an OpenAI-compatible request.
2. A prefill worker processes the prompt and produces the initial KV cache.
3. NIXL transfers that cache to a decode worker.
4. The decode worker generates and streams the output.
5. Prefill and decode pools can be scaled independently.

That separation is the product's actual value proposition. Dynamo is not a replacement inference engine in this configuration; vLLM still executes the model. Dynamo supplies the frontend, worker discovery, routing, transfer integration, and Kubernetes control plane around it.

The architecture is compelling when prompt and generation workloads require different replica counts, GPU types, or parallelism. It does not improve performance without adding overhead. Every request acquires another scheduling boundary and a KV-transfer operation. If transfer latency exceeds the interference avoided by separating the phases, disaggregation loses.

We therefore treated the transfer path as the first benchmark subject, not an implementation detail.

What We Could Not Verify#

We reviewed the available release and manifest examples, constructed a proposed deployment procedure, and defined benchmark controls for a future cluster run. We did not have a release-matched GPU cluster and complete CRD bundle available for a defensible performance run. In our benchmark-readiness review, we found an execution plan but no measured results to assess.

Consequently, this article does not invent TTFT, inter-token latency, throughput, GPU utilization, or cost numbers. It is a source-verified deployment and benchmark readiness review, not a disguised synthetic benchmark. Teams wanting help turning this plan into a controlled cluster run can review our AI infrastructure services or contact us with the target GPU topology.

Release-Pinned Kubernetes Setup and Validation Procedure#

We pinned the review to Dynamo 1.4.2 rather than assembling unrelated current versions of Dynamo, vLLM, CUDA, NIXL, and the operator. That pin matters because both the custom-resource schema and runtime arguments have changed across examples.

Our starting point was NVIDIA's vLLM Kubernetes template collection. The baseline disaggregated template uses a DynamoGraphDeployment with three components:

• Frontend, with type frontend
• prefill, with type prefill
• decode, with type decode

Both workers invoke python3 -m dynamo.vllm, use the same model, and configure the NIXL connector:

args:
  - --model
  - Qwen/Qwen3-0.6B
  - --disaggregation-mode
  - prefill
  - --kv-transfer-config
  - '{"kv_connector":"NixlConnector","kv_role":"kv_both"}'

The decode worker changes only the disaggregation mode in that minimal example:

args:
  - --model
  - Qwen/Qwen3-0.6B
  - --disaggregation-mode
  - decode
  - --kv-transfer-config
  - '{"kv_connector":"NixlConnector","kv_role":"kv_both"}'

We would not hand-author the full custom resource before inspecting the installed CRD. The 1.4.2 material contains both nvidia.com/v1alpha1 and nvidia.com/v1beta1 examples. Our deployment gate therefore begins with the cluster, not a copied blog fragment:

export NS=dynamo-bench
export MODEL=Qwen/Qwen3-0.6B
export IMAGE=nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.4.2

kubectl create namespace "$NS" --dry-run=client -o yaml | kubectl apply -f -

kubectl api-resources \
  --api-group=nvidia.com \
  | grep -i DynamoGraphDeployment

kubectl get crd dynamographdeployments.nvidia.com \
  -o jsonpath='{.spec.versions[?(@.served==true)].name}{"\n"}'

kubectl explain dynamographdeployment.spec.components

For gated models, create the secret without placing the token in the manifest:

kubectl -n "$NS" create secret generic hf-token-secret \
  --from-literal=HF_TOKEN="$HF_TOKEN" \
  --dry-run=client -o yaml | kubectl apply -f -

We would then copy the release-matched disagg.yaml, replace my-tag, set the model consistently on both workers, add persistent model caching if needed, and apply it:

kubectl apply --server-side --dry-run=server \
  -n "$NS" -f disagg.yaml

kubectl apply -n "$NS" -f disagg.yaml
kubectl get pods -n "$NS" -w

Server-side dry-run is non-negotiable here. Client-side YAML parsing cannot establish whether the operator accepts a particular API version or field layout.

The next gate is NIXL initialization. On a single multi-GPU node, the intended path uses CUDA IPC and NVLink rather than an external network. Worker pods still need enough shared memory because NIXL stages transfer metadata through /dev/shm. We used 16 GiB as the documented operational headroom; Kubernetes' typical 64 MB container default is inadequate.

PREFILL_POD="$(
  kubectl get pods -n "$NS" \
    -o name | grep -i prefill | head -n 1
)"

kubectl logs -n "$NS" "$PREFILL_POD" \
  | grep -Ei 'NIXL|UCX|CUDA IPC|TCP'

We expect successful NIXL initialization to produce a log line like this:

NIXL INFO Backend UCX was instantiated

That line alone does not prove RDMA or acceptable transfer latency. Cross-node operation requires us to inspect the selected UCX transport and verify RDMA resources, not merely find the string NIXL.

Once the service is available inside the cluster, this request exercises the API without contaminating the measurement with kubectl port-forward:

kubectl run dynamo-smoke \
  --namespace "$NS" \
  --rm -i --restart=Never \
  --image=curlimages/curl:8.12.1 \
  -- curl -sS \
    -H 'Content-Type: application/json' \
    http://vllm-disagg-frontend/v1/completions \
    -d '{
      "model": "Qwen/Qwen3-0.6B",
      "prompt": "Return exactly: transfer-path-ok",
      "max_tokens": 16,
      "temperature": 0
    }'

The following simulated response illustrates the smoke-test contract; it is not output from a completed GPU benchmark:

{
  "id": "cmpl-smoke-001",
  "object": "text_completion",
  "model": "Qwen/Qwen3-0.6B",
  "choices": [
    {
      "index": 0,
      "text": "transfer-path-ok",
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 8,
    "completion_tokens": 4,
    "total_tokens": 12
  }
}

A successful response is only gate one. We would also correlate the request ID across frontend, prefill, and decode logs, confirm prefill occurs before decode, and collect transfer bytes and cache events from backend metrics. An HTTP 200 cannot prove that the desired transfer path was used.

For KV-aware routing, the template sets DYN_ROUTER_MODE=kv. Its prefill workers publish cache events over ZMQ on topic kv-events and endpoint tcp://*:20080. That is separate from disaggregation itself: stage separation determines where computation occurs, while KV-aware routing determines which worker receives a request.

Schema Differences, Evidence Limits, and Operational Requirements#

The first issue was schema ambiguity. The disaggregated-serving overview includes a v1alpha1 DGD shape with services, componentType, and subComponentType. The vLLM template page uses v1beta1 with a components list and type fields. We could not responsibly present those formats as interchangeable.

Our proposed workaround is to inspect the installed CRD and use server-side dry-run to reject incompatible manifests before allocating GPUs. We also avoided combining an older runtime example with a newer operator just because both files contained the word DynamoGraphDeployment.

The second issue was incomplete benchmark evidence. For our proposed EKS benchmark, we would use concurrency sweeps, multi-turn traffic, sequence distributions, and shared-prefix tests. We have no measured results for this comparison and treat the following infrastructure strictly as a candidate topology, not a completed run:

• Two p4d.24xlarge nodes
• Eight A100 40 GB GPUs per node
• Sixteen GPUs total
• Four aggregated workers versus two prefill and two decode workers
• A separate heterogeneous experiment using L40S instances

We refused to convert that plan into fictional measurements. Any article assigning p50 TTFT or tokens per second to that setup would be manufacturing evidence.

The third gotcha is that NIXL's presence does not prove a fast data path. On one node, CUDA IPC over NVLink is the desired route. Across nodes, the deployment needs an RDMA-capable fabric, an RDMA device plugin, matching rdma/ib resource requests, IPC_LOCK, and suitable UCX settings. If logs show only TCP transports, RDMA is not active. At that point KV movement can dominate TTFT and erase the benefit of separation.

The fourth failure mode is configuration drift between worker pools. Prefill and decode must agree on:

• Model and revision
• Data type
• KV-cache data type
• Block size
• KV layout
• Relevant tokenizer and context settings

A transfer failure is preferable to the worst outcome: output that appears valid but is corrupted because the consumer interpreted incompatible KV state. We would enforce these values from one generated configuration object rather than duplicate them manually in two YAML sections.

We also found several smaller operational traps:

• The default /dev/shm allocation is too small for the expected NIXL setup.
• Multi-worker NIXL configurations need unique side-channel ports.
• KV-aware routing requires consistent hashing behavior; PYTHONHASHSEED=0 is an explicit check for the vLLM processes.
• NATS is not universally mandatory. NATS-backed events require it, while ZMQ events and prediction-based routing do not. Operator-based Kubernetes discovery also changes the coordination requirements.
• Port-forwarding is useful for a smoke test but inappropriate for measuring high-load service performance.
• We would measure model download time separately from steady-state serving performance.
• Cache overlap in a generated prompt is not the same as an observed cache hit. Capacity, routing, and eviction can still produce misses.

These are exactly the types of controls we include when evaluating tools for our infrastructure tools collection. Without them, an apparently clean benchmark often measures image pulls, TCP fallback, or routing mistakes instead of inference architecture.

Scale, Latency & Cost vs. Alternatives#

We cannot publish a winning latency number without the corresponding run. We can identify where each design should win and define the break-even test that a production team must pass.

For cost, we use successful output tokens delivered within the latency SLO. Raw generated tokens are misleading if requests time out or violate the service target.

The cost per million successful output tokens is:

cost_per_million =
  hourly_deployment_cost
  / (successful_output_tokens_per_second * 3600)
  * 1,000,000

If an aggregated deployment costs A per hour and a disaggregated deployment costs D, disaggregation breaks even only when:

disaggregated_throughput / aggregated_throughput > D / A

At equal hourly GPU cost, any reliable throughput increase can improve unit economics, provided latency and quality remain acceptable. If the disaggregated topology costs 25% more, it needs more than a 25% increase in successful SLO-compliant throughput merely to break even. That is a mathematical threshold, not a result from our test environment.

We would run four workload families before making the decision:

1. A concurrency sweep using fixed 1,024-token inputs and 512-token outputs.
2. Five-turn conversations to measure actual cache reuse.
3. Uniform, Zipf, and lognormal sequence-length distributions.
4. Shared-prefix traffic from 0% through 100% overlap.

For every run, we would retain TTFT p50 and p99, inter-token latency, request latency, successful throughput, errors, cancellations, stage queues, transfer bytes, cache hits, evictions, and GPU utilization. The aggregated and disaggregated configurations must use equal total GPU resources before introducing a heterogeneous cost experiment.

Changing disaggregation, routing, cache offload, quantization, GPU type, and replica count at once produces a demo, not an attribution-quality benchmark.

Our Final Verdict: When to Deploy, When to Skip#

Dynamo's disaggregated vLLM architecture is technically credible. It exposes the right control boundary for workloads where prompt processing and token generation need different capacity. The Kubernetes templates also reduce the amount of custom orchestration required.

We would deploy it when all of the following are true:

• Prompt lengths or retrieval payloads make prefill a measurable bottleneck.
• Decode concurrency requires a different replica count or GPU shape.
• The team can verify CUDA IPC or RDMA rather than assume it.
• Aggregated vLLM has already been measured as the baseline.
• Model and KV settings can be generated consistently across worker pools.
• The platform team can operate the Dynamo operator, routing layer, metrics, and transfer path.
• Benchmark traffic runs inside the cluster with fixed datasets and repeated trials.
• Cost is calculated from successful tokens delivered within the SLO.

We would hold off when any of these conditions apply:

• The model is small, prompts are short, and concurrency is low.
• The deployment has only one useful GPU.
• Cross-node traffic would fall back to TCP.
• The organization cannot inspect NIXL and UCX behavior.
• The service needs the smallest possible operational surface.
• Benchmark comparisons change GPU count, precision, or model configuration between variants.
• The business case depends on performance figures that have not actually been collected.

Our final rating is conditional deploy. Start with aggregated vLLM, reproduce the workload, then introduce disaggregation as one controlled variable. Add KV-aware routing in a later experiment rather than bundling it into the first comparison.

The key lesson is straightforward: splitting prefill from decode does not improve performance by itself. It enables independent scheduling and scaling but adds KV-transfer overhead and operational complexity. Dynamo is worth deploying only when measurements show that the benefits outweigh those costs.

For implementation details, we would keep the release-pinned Dynamo disaggregated-serving guide beside the cluster runbook and treat the installed CRD, runtime logs, and observed transfer metrics as the final authority.