vLLM

Overview#

vLLM is an inference engine for transformer language models. It exposes an HTTP server that speaks the OpenAI Chat Completions, Completions and Embeddings APIs, an offline Python entry point (`LLM(...).generate(...)`) for batch inference, and a low-level engine API used by production stacks such as KServe, Ray Serve, NVIDIA Dynamo and the upstream vLLM Production Stack. The project's defining premise is that LLM serving is a memory-and-scheduling problem first and a kernel-tuning problem second.

The original v0.1 release (June 2023) shipped two ideas that the rest of the field has since adopted: PagedAttention, which manages the KV cache as fixed-size blocks indexed by per-sequence block tables; and continuous (iteration-level) batching, which admits and evicts sequences between every forward pass rather than at request boundaries. Together they raised KV-cache memory utilisation from roughly 40 percent to above 95 percent and lifted achievable throughput on a single H100 by an order of magnitude on chat-shaped workloads.

By mid-2026 the project sits under the LF AI & Data Foundation, with maintainers from UC Berkeley, IBM, Meta, NVIDIA, AMD, Anyscale, Snowflake, Databricks and Yobitel. The mainline release cadence is roughly every two to three weeks, with new model architectures typically supported within days of weight publication. vLLM is the runtime against which every newer engine — SGLang, TensorRT-LLM, MAX, MistralRS — is benchmarked. Yobibyte exposes vLLM as its default inference engine; Yobitel customers reach vLLM through a managed workspace rather than building containers, registries and schedulers from raw upstream.

This entry documents the production surface: the CLI and Python APIs, the scheduling and KV-cache internals, the parallelism strategies, the deployment patterns, the limits and quotas, the observability hooks, and the practical sizing and cost models you need to operate vLLM at scale on Yobitel and beyond. This entry helps you stand up vLLM for production LLM serving with the right flags, sizing and operational practices — whether you are running raw upstream on your own cluster or consuming vLLM through Yobibyte's managed workspaces.

Quick start#

The example below deploys Llama 3.1 70B on a 4x H100 SXM5 node with FP8 weights, prefix caching, chunked prefill and a 32K context window, then issues an OpenAI-compatible chat completion. The first block installs vLLM and serves the model directly on any CUDA 12.4+ host; the second block hits the running endpoint with `curl`; the third block drives the same endpoint from Python using the standard `openai` SDK pointed at the local server.

# 1. Install vLLM and serve Llama 3.1 70B on 4x H100 SXM5
pip install "vllm>=0.8.0"

vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
    --tensor-parallel-size 4 \
    --max-model-len 32768 \
    --max-num-seqs 256 \
    --gpu-memory-utilization 0.92 \
    --quantization fp8 \
    --kv-cache-dtype fp8 \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --num-scheduler-steps 8 \
    --port 8000

# 2. Hit the OpenAI-compatible endpoint with curl
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "meta-llama/Meta-Llama-3.1-70B-Instruct",
      "messages": [{"role": "user", "content": "Summarise PagedAttention in 2 lines."}],
      "max_tokens": 128
    }'

# 3. Same call from Python using the official openai SDK
python - <<'PY'
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
reply = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-70B-Instruct",
    messages=[{"role": "user", "content": "Summarise PagedAttention in 2 lines."}],
    max_tokens=128,
)
print(reply.choices[0].message.content)
PY

How it works#

vLLM is structured as an asynchronous engine wrapped by a frontend API server. Requests enter through the FastAPI server, are tokenised, validated against `--max-model-len` and the per-request `max_tokens` budget, and handed to the LLMEngine. The engine maintains three queues: WAITING (admitted, not yet running), RUNNING (currently in the active batch), and SWAPPED (paged out to host memory under KV-cache pressure). On every step the scheduler picks the next set of sequences to run based on KV-cache availability and request priority, and the model executor performs one forward pass.

Forward execution uses FlashAttention-3 on Hopper and Blackwell and FlashAttention-2 on Ampere, with a paged-KV variant of the attention kernel that gathers K and V from the block pool via the per-sequence block table. Tensor-parallel and pipeline-parallel ranks coordinate via NCCL; the engine starts one worker process per rank with shared GPU memory for the KV pool. The KV pool size is computed at start-up as (`gpu_memory_utilization` x device memory) minus weights minus activation working set, divided by block size.

Chunked prefill changes how long inputs are processed. Without it, a single request with a 30K-token prompt monopolises the GPU for a full prefill before any decode tokens emerge, starving short interactive requests. With `--enable-chunked-prefill`, the scheduler interleaves prefill chunks (default 512 tokens) with decode tokens in the same iteration, dramatically improving p99 latency under mixed load. From v0.6 onward chunked prefill is the recommended default for production deployments.

Speculative decoding adds a draft pass per step. The engine runs a small draft model (or EAGLE-2 / Medusa heads attached to the target) to propose k tokens, then verifies them with one parallel forward of the target. Accepted tokens are committed; the first rejected token is the new ground truth. On chat workloads with a well-matched draft (Llama 3 8B for 70B target), end-to-end latency drops 1.5-3x at unchanged quality.

• PagedAttention: KV cache split into 16-token blocks (configurable), allocated on demand from a global pool. Identical prefix blocks are content-addressed and shared across sequences.
• Continuous batching: scheduler decision happens every iteration; sequences enter and leave the running batch at token boundaries.
• Prefix caching: cached blocks survive request completion until evicted under LRU; subsequent requests with matching prefixes skip prefill on shared tokens.
• Chunked prefill: prefill work split into fixed-size chunks interleaved with decode in the same step.
• Multi-step scheduling: `--num-scheduler-steps n` lets the worker execute n forward passes per scheduler invocation, reducing Python overhead by ~3-5x.
• CUDA graphs: enabled by default for decode-only batches; captures the forward pass to eliminate launch overhead.

Reference and specifications#

Every long-lived deployment is parameterised through a small number of high-impact flags. The table below is the canonical reference for the engine CLI surface as of vLLM v0.8 (June 2026). Flags marked with an asterisk are also available as `EngineArgs` fields in the Python API. Flags not listed here are either internal tuning knobs that defaults handle correctly, or specialised features documented in the upstream reference.

Workload patterns#

Three workload shapes cover the bulk of vLLM production deployments: interactive chat behind an OpenAI-compatible gateway, RAG with long shared system prompts, and offline batch inference for evaluation or labelling. Each has its own preferred set of flags. These are the same three shapes Yobibyte automates for managed customers — the flags below are what a team running raw vLLM on their own Kubernetes signs up to hand-tune; the Yobibyte console derives them from the workspace's stated SLO.

Pattern A — Chat endpoint, OpenAI-compatible. Maximise concurrent users at a target p95 time-to-first-token of 250-500 ms. Enable prefix caching for repeated system prompts, chunked prefill so a long prompt does not stall short requests, and multi-step scheduling to amortise Python overhead. Pattern B — RAG endpoint with a 4-32K shared system prompt across thousands of requests. Pattern C — offline batch scoring with no API server.

# A — chat endpoint on 2x H100 SXM5 for an 8B-class model
vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \
    --tensor-parallel-size 2 \
    --max-model-len 16384 \
    --max-num-seqs 512 \
    --gpu-memory-utilization 0.90 \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --num-scheduler-steps 16 \
    --kv-cache-dtype fp8 \
    --quantization fp8 \
    --enable-auto-tool-choice \
    --port 8000

# B — RAG endpoint, 4-32K shared system prompt across thousands of requests
vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
    --tensor-parallel-size 4 \
    --max-model-len 32768 \
    --max-num-batched-tokens 8192 \
    --gpu-memory-utilization 0.92 \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --block-size 32 \
    --quantization fp8

# C — offline batch scoring (no API server)
python - <<'PY'
from vllm import LLM, SamplingParams
llm = LLM(
    model="meta-llama/Meta-Llama-3.1-70B-Instruct",
    tensor_parallel_size=4,
    quantization="fp8",
    enable_prefix_caching=True,
    max_num_seqs=1024,
    gpu_memory_utilization=0.95,
)
prompts = open("eval-prompts.txt").read().splitlines()
out = llm.generate(prompts, SamplingParams(max_tokens=256, temperature=0))
for r in out:
    print(r.outputs[0].text)
PY

Sizing and capacity planning#

vLLM throughput is bounded first by KV-cache memory, then by tensor-core FLOPs, then by NCCL bandwidth at TP > 2. The planning model below assumes Llama-family architectures with grouped-query attention and FP8 weights and KV; FP16 / BF16 doubles every memory column. Tokens-per-second figures are mid-range observed values from InferenceBench v3 at 4K input / 256 output, mixed concurrency; treat them as planning anchors, not contractual.

KV-cache budget is the constraint that decides max concurrency. The per-token KV size is `2 x n_layers x n_kv_heads x head_dim x dtype_bytes`. For Llama 3.1 70B with GQA (8 KV heads) at FP8 that is roughly 40 KB per token. On 4x H100 with TP=4 the weights occupy ~70 GB, activations ~12 GB, leaving ~240 GB of pooled KV — about 6 million KV tokens, enough for 180 concurrent sequences at average 32K used or 1,500 sequences averaging 4K used. Tensor-parallel size choice is governed by intra-node bandwidth: TP up to 8 inside one NVLink island is well-behaved; TP across InfiniBand is almost always slower than pipeline parallelism. For two-node deployments, prefer TP=8 + PP=2 over TP=16. For three or more nodes, expert parallelism dominates if the model is MoE; otherwise pipeline with replicas behind a router beats deep PP.

Limits and quotas#

vLLM enforces a small set of hard and soft limits at the engine boundary. Hard limits reject requests with HTTP 400 at the API server; soft limits apply backpressure by extending queue depth. Operational ceilings (memory, NCCL groups, file descriptors) come from the host OS and CUDA runtime.

Observability#

vLLM exposes a Prometheus metrics endpoint at `/metrics` covering request throughput, latency histograms, GPU cache utilisation, prefix-cache hit rate, scheduler queue depth and preemption counts. The metric prefix is `vllm:`. Engine logs emit one structured line per request when `--disable-log-requests` is unset; switch to JSON output via `VLLM_LOG_LEVEL=INFO` and `VLLM_LOGGING_CONFIG_PATH` for ingestion into Loki, Splunk or Datadog.

The metrics worth alerting on in production are: time-to-first-token p95, inter-token latency p95, GPU cache usage, prefix-cache hit rate, number of preempted sequences, and request queue time. The following Prometheus rules cover the common failure modes.

• vllm:time_to_first_token_seconds — prefill latency; correlate with prompt-length histogram.
• vllm:time_per_output_token_seconds — decode latency; should be near 1 / (theoretical tok/s) when batch is full.
• vllm:gpu_cache_usage_perc — KV pool fill; consistent reading above 90 percent indicates capacity headroom is gone.
• vllm:prefix_cache_hit_rate — fraction of incoming prefill tokens served from cache.
• vllm:num_preemptions_total — sequences evicted under KV pressure; non-zero in steady state means undersized cluster.
• vllm:request_queue_time_seconds — time from request arrival to first scheduler admission.
• DCGM_FI_DEV_GPU_UTIL and DCGM_FI_DEV_MEM_COPY_UTIL — pair with vLLM metrics to distinguish compute, memory and idle bottlenecks.

# Prometheus rules for a vLLM deployment
groups:
  - name: vllm-sla
    interval: 30s
    rules:
      - alert: VLLMHighTimeToFirstToken
        expr: histogram_quantile(0.95,
                sum by (le, model_name) (
                  rate(vllm:time_to_first_token_seconds_bucket[5m]))) > 1.0
        for: 5m
        labels: { severity: warning, team: inference }
        annotations:
          summary: "vLLM TTFT p95 above 1s on {{ $labels.model_name }}"

      - alert: VLLMKVCachePressure
        expr: vllm:gpu_cache_usage_perc > 0.95
        for: 10m
        labels: { severity: warning }
        annotations:
          summary: "KV pool >95 percent full — preemption imminent"

      - alert: VLLMPreemptionSpike
        expr: increase(vllm:num_preemptions_total[5m]) > 20
        for: 5m
        labels: { severity: critical }
        annotations:
          summary: "Preemptions rising — capacity insufficient or runaway request"

      - alert: VLLMPrefixCacheCollapse
        expr: vllm:prefix_cache_hit_rate < 0.20
              and vllm:prefix_cache_queries_total > 100
        for: 15m
        labels: { severity: info }
        annotations:
          summary: "Prefix cache hit rate dropped — workload shape changed"

      - alert: VLLMGPUUnderutilised
        expr: avg_over_time(DCGM_FI_DEV_GPU_UTIL[5m]) < 30
              and rate(vllm:request_success_total[5m]) > 0
        for: 15m
        labels: { severity: info }
        annotations:
          summary: "GPU under 30 percent — investigate Python overhead or PP bubble"

Cost and FinOps#

vLLM cost economics are dominated by three levers: GPU rental rate, achieved tokens-per-second-per-GPU, and average prefix-cache hit rate. Holding model and SKU fixed, doubling sustained throughput halves the unit cost in $/M tokens. The table below uses Yobitel UK list pricing (June 2026) and InferenceBench v3 throughput anchors; substitute your own when planning.

• Spot instances cut GPU rate 40-60 percent but require autoscaling that tolerates 30-90s pre-emption notices. Pair with vLLM's draining endpoint to flush in-flight requests cleanly.
• FP8 weights + FP8 KV is the highest $/M-tokens lever available on Hopper; BF16 is roughly 1.6x more expensive at the same SLO.
• Prefix-cache hits are accounted at zero prefill cost — high-overlap workloads (multi-tenant agents, shared system prompts) can lift effective throughput 1.5-2x at no infrastructure change.
• FOCUS-conformant billing exports from Yobitel include `inference_engine` and `model_name` resource tags so $/M tokens can be sliced by tenant or product line.

Security and compliance#

vLLM ships with an opt-in bearer-token auth on the API server (`--api-key`); production deployments terminate TLS at an ingress (Envoy, NGINX, AWS ALB) and apply mTLS or signed-JWT auth at that layer. The engine itself does not enforce per-tenant quotas — those are implemented at the gateway or via the Yobitel platform's multi-tenant router. Network isolation should follow the standard pattern: the inference pod has no egress to the public internet, model weights are pulled from a private registry, and per-replica NetworkPolicy locks ingress to the gateway service account.

Prompt-injection mitigation is a layered concern that vLLM does not solve directly. The engine supports system-prompt pinning via prefix caching (the system prompt is the same blocks every request) and structured-output enforcement via Outlines, XGrammar or LM Format Enforcer. Pair these with retrieval source validation, output classifiers and rate limits at the gateway. The Yobibyte platform enforces this stack by default.

Regulatory implications are model-, data- and deployment-specific. For UK public-sector workloads, deploy on Yobitel sovereign tenancies that satisfy NCSC Cloud Security Principles and G-Cloud 14 (vLLM as the engine is a control-plane component on those tenancies). For EU GDPR, the engine processes prompt and completion data only in volatile GPU memory and the on-disk scratch path; ensure `--swap-space` resides on encrypted ephemeral storage. For US HIPAA workloads, run inside a BAA-covered VPC and disable request logging; for FedRAMP, run the FIPS-validated CUDA build and pin to NIAP-approved cipher suites at the ingress.

Migration and alternatives#

Most production migrations to vLLM come from one of four origins: HuggingFace `pipeline()` / `model.generate()`, Hugging Face TGI, NVIDIA TensorRT-LLM, or a managed SaaS API (OpenAI, Anthropic, AWS Bedrock). The first delivers the largest throughput uplift (5-10x typical at the same latency); the others are roughly at parity on throughput with different operational trade-offs.

If you are running production today on Kubernetes with TGI, the migration is essentially a container swap and a flag rename. The incumbent commands below produce comparable deployments; you can roll vLLM behind the same `Service` and shift traffic at the gateway.

# Production deployment on Kubernetes with NVIDIA GPU Operator installed
kubectl apply -f - <<'YAML'
apiVersion: apps/v1
kind: Deployment
metadata: { name: llama3-70b-vllm }
spec:
  replicas: 2
  selector: { matchLabels: { app: llama3-70b-vllm } }
  template:
    metadata: { labels: { app: llama3-70b-vllm } }
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.8.0
          args:
            - "--model=meta-llama/Meta-Llama-3.1-70B-Instruct"
            - "--tensor-parallel-size=4"
            - "--max-model-len=32768"
            - "--enable-prefix-caching"
            - "--enable-chunked-prefill"
            - "--quantization=fp8"
            - "--kv-cache-dtype=fp8"
          resources:
            limits: { nvidia.com/gpu: 4 }
          ports: [{ containerPort: 8000 }]
          volumeMounts:
            - { name: dshm, mountPath: /dev/shm }
      volumes:
        - name: dshm
          emptyDir: { medium: Memory, sizeLimit: 8Gi }
YAML

# Equivalent today on AWS (bare p5 instance with Deep Learning AMI)
AMI_ID=$(aws ec2 describe-images --owners amazon \
    --filters "Name=name,Values=Deep Learning OSS Nvidia Driver AMI GPU PyTorch 2.4*" \
    --query 'sort_by(Images,&CreationDate)[-1].ImageId' --output text)

aws ec2 run-instances \
    --image-id "$AMI_ID" \
    --instance-type p5.48xlarge \
    --user-data "$(printf '#!/bin/bash\npip install vllm>=0.8.0\nvllm serve meta-llama/Meta-Llama-3.1-70B-Instruct --tensor-parallel-size 8 --quantization fp8 --port 8000\n')"

# Equivalent on GCP A3 (H100)
gcloud compute instances create vllm-llama70b \
    --machine-type=a3-highgpu-8g \
    --accelerator=type=nvidia-h100-80gb,count=8 \
    --image-family=pytorch-2-4-cu124 --image-project=deeplearning-platform-release \
    --metadata=startup-script='vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct --tensor-parallel-size 8 --quantization fp8'

Troubleshooting#

The error table below covers the failure modes that account for roughly 80 percent of production vLLM incidents observed on Yobitel-operated fleets and InferenceBench community submissions. Each row maps an observable symptom to the underlying mechanism and the minimum-viable fix.

Where this fits in the Yobitel stack#

vLLM is the default inference engine inside Yobibyte, Yobitel's AI-native platform. Every model that lands in the Yobibyte catalogue — from Llama 3.1 8B to DeepSeek-V3 671B — is exposed first via a vLLM-backed endpoint, with TensorRT-LLM offered as an opt-in performance variant where lowest-latency SLOs justify the engine build overhead. The Yobibyte control plane handles fleet sizing, prefix-cache-aware routing, multi-LoRA tenancy, draining for spot pre-emption, and FOCUS-conformant cost attribution back to tenants.

Omniscient Compute scores vLLM continuously on InferenceBench v3, the public benchmark suite Yobitel maintains for inference engines across NVIDIA H100, H200, B200 and AMD MI300X tenancies. Each release is benchmarked at fixed input/output token mixes (chat, RAG, long-context, batch) and the results are surfaced to customers as live capacity plans — every recommended SKU and replica count on the Yobibyte console comes from an InferenceBench measurement, not a vendor datasheet.

For UK and EU sovereign workloads, vLLM runs on the Yobitel London-1 and Frankfurt-1 regions inside tenancies that satisfy NCSC Cloud Security Principles, G-Cloud 14 lot definitions and the OFFICIAL handling caveat. The combination of an open-source Apache 2.0 engine, sovereign hardware, and transparent benchmark scoring is what lets Yobitel customers deploy production LLMs without ceding control or visibility to a hosted SaaS API.