NVIDIA H100 Tensor Core GPU

Overview#

The NVIDIA H100 is the data centre GPU that turned large language models from a research curiosity into an industrial product. Announced at GTC 2022 and shipping in volume from Q4 2022, it pairs the Hopper architecture (GH100, 80 billion transistors on TSMC 4N) with HBM3 memory and a dedicated Transformer Engine — the combination that let teams train 70B-parameter models in weeks rather than months and that defined the cost-per-token economics of the first ChatGPT-era serving fleet.

The headline numbers — 989 TFLOPS BF16, 1,979 TFLOPS FP8 dense, 3,958 TFLOPS FP8 with 2:4 sparsity, 80 GB HBM3, 3.35 TB/s memory bandwidth — only matter alongside the interconnect. NVLink 4.0 and the third-generation NVSwitch ASIC give H100 the lowest-latency, highest-bandwidth fabric of any commodity accelerator. That fabric is what made the H100 era distinct from the A100 era: not just more FLOPS per GPU, but a way to make 256 GPUs behave like one. By 2026, H100 capacity is broadly available across every hyperscaler, every NVIDIA-Partner neocloud and most regional sovereign clouds; pricing has compressed from $4-8/GPU-hour in 2023 to $1.10-3.00/GPU-hour, making it frequently the best price-per-training-token GPU NVIDIA ships.

This entry is the reference for teams operating H100 at scale: full spec sheet, the sizing tables we use internally on InferenceBench, the DCGM signals to alert on, the FinOps levers that move the needle, the migration paths to and from neighbouring SKUs, and the troubleshooting playbook for the issues every team eventually hits. Yobitel NeoCloud offers H100 SXM5 capacity in UK and EU regions with NCSC OFFICIAL alignment, NVLink-locality-aware placement, and FOCUS-conformant billing — most teams reading this entry consume H100 either through NeoCloud directly or through Yobibyte's managed inference workspaces. This entry helps you decide when H100 is the right pick for your workload and how to size and price it on Yobitel NeoCloud or your own cluster.

Quick start#

The shortest path from zero to a running H100 today. Three equivalent routes are shown below: an AWS p5.48xlarge (8x H100 SXM5) launched via the EC2 API, a GCP a3-highgpu-8g (8x H100 SXM5) launched via gcloud, and a bare-metal/colo path that exposes existing H100 nodes to Kubernetes via the NVIDIA GPU Operator. Pick whichever matches your fleet, then jump to Workload pattern A to serve Llama 3 70B on the GPUs you just provisioned.

# --- Route 1: AWS p5.48xlarge (8x H100 SXM5, 3.2 Tb/s EFA) ---
# Requires the "Running On-Demand P instances" service quota uplifted from 0.
aws ec2 run-instances \
  --region eu-west-2 \
  --image-id ami-0abcdef1234567890 \
  --instance-type p5.48xlarge \
  --key-name my-key \
  --subnet-id subnet-0abc... \
  --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=500,VolumeType=gp3}' \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=h100-train-01}]'

# Verify the 8 H100s are visible once the host boots
aws ssm start-session --target i-0abc... -- nvidia-smi -L

# --- Route 2: GCP a3-highgpu-8g (8x H100 SXM5) ---
gcloud compute instances create h100-train-01 \
  --project=my-project \
  --zone=europe-west2-a \
  --machine-type=a3-highgpu-8g \
  --image-family=common-cu124-debian-12 \
  --image-project=deeplearning-platform-release \
  --maintenance-policy=TERMINATE \
  --boot-disk-size=500GB --boot-disk-type=pd-ssd \
  --metadata="install-nvidia-driver=True"

gcloud compute ssh h100-train-01 --zone=europe-west2-a --command='nvidia-smi'

# --- Route 3: Bare-metal / colo K8s — expose existing H100 nodes ---
# Adds the NVIDIA GPU Operator, which installs drivers, container-toolkit,
# DCGM exporter, MIG manager and the device plugin.
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update
kubectl create namespace gpu-operator
helm install --wait gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --version v24.9.0 \
  --set driver.enabled=true \
  --set toolkit.enabled=true \
  --set dcgmExporter.enabled=true \
  --set migManager.enabled=true

# Verify the operator brought up driver + plugin + DCGM
kubectl get pods -n gpu-operator
kubectl get nodes -L nvidia.com/gpu.product
# NAME            STATUS   ROLES    GPU.PRODUCT
# h100-node-01    Ready    worker   NVIDIA-H100-80GB-HBM3
kubectl describe node h100-node-01 | grep -E 'nvidia.com/gpu'
# Capacity:     nvidia.com/gpu: 8
# Allocatable:  nvidia.com/gpu: 8

How it works: Hopper architecture and the H100 pipeline#

Hopper introduced four innovations over Ampere that justified the generational leap, none of which were headline FLOPS in isolation.

First, the fourth-generation Tensor Core added native FP8 support (E4M3 and E5M2) at twice the throughput of FP16. Paired with the Transformer Engine — a runtime that maintains exponential moving averages of activation magnitudes (amax history) and selects per-layer FP8 vs BF16 vs FP32 — typical LLM training throughput roughly doubled at iso-precision against A100 BF16 baselines. E4M3 is used for forward activations and weights, E5M2 for gradients where extended range matters more than precision.

Second, Thread Block Clusters grouped multiple Cooperative Thread Arrays (CTAs) under a unified distributed-shared-memory namespace, letting kernels reuse data across SM groups without round-tripping to HBM. Combined with the new Tensor Memory Accelerator (TMA) — a dedicated copy engine that asynchronously moves tensor tiles between HBM and SMEM with descriptor-based addressing — this is what made Flash Attention 2 and 3 possible in their published forms. TMA is also why hand-tuned cuBLAS LT GEMMs on H100 routinely close to 80-90 % of peak.

Third, DPX instructions accelerated dynamic-programming inner loops — Smith-Waterman sequence alignment, route planning, certain reinforcement-learning search workloads — at up to 7x Ampere throughput.

Fourth, second-generation MIG (Multi-Instance GPU) added confidential-compute boundaries between instances and memory-bandwidth partitioning, letting a single H100 host multi-tenant inference with hardware-enforced isolation. MIG slices on H100 expose a fraction of SMs, a fixed share of HBM (10 GB per 1g.10gb slice up to 80 GB for a full 7g.80gb), and an isolated NVDEC/NVENC pair.

• GH100 die: 132 Streaming Multiprocessors (SMs), 528 fourth-generation Tensor Cores, 60 MB L2 cache, 50 MB combined L1/SMEM across SMs.
• Memory: 5 HBM3 stacks x 16 GB = 80 GB total at 3.35 TB/s on SXM5 (HBM2e on PCIe at 2.0 TB/s).
• Compute capability: sm_90 (sm_90a for the architecture-specific TMA and wgmma intrinsics used by CUTLASS, Flash Attention 3 and Triton's Hopper backend).
• Confidential Compute (CC-on) mode: AES-256-GCM encryption of all PCIe traffic and HBM-resident pages, attested via SPDM and NVIDIA's attestation service.

Reference: full specification sheet#

Authoritative per-SKU figures. SXM5 fills HGX-H100 baseboards and almost every cloud GPU instance; PCIe Gen5 is the drop-in card for retrofit servers; NVL pairs two PCIe boards via a 600 GB/s bridge with 188 GB HBM3 for memory-pressured inference. All Tensor figures assume 2:4 structured sparsity unless noted; dense throughput is half the sparse figure.

Interconnect: NVLink 4.0 and the NVSwitch fabric#

Every H100 SXM5 module exposes 18 NVLink 4.0 ports, each providing 50 GB/s bidirectional — 900 GB/s aggregate per GPU. That figure alone is interesting; the topology around it is what matters.

An HGX-H100 baseboard places 8 GPUs alongside 4 NVSwitch ASICs, wiring every GPU to every switch. The result is a fully non-blocking 8-GPU shared-memory fabric: any GPU can DMA into any other GPU's HBM at full NVLink bandwidth with no fabric contention. Inside one DGX H100, all-to-all collectives like AllReduce hit 450 GB/s per direction — close to the theoretical NVLink ceiling, and roughly 3x the equivalent A100 figure.

Beyond 8 GPUs, the optional NVLink Switch System extends the same topology to 256-GPU pods via external NVLink switches. The pod delivers 57.6 TB/s of bisection bandwidth — meaningfully faster than InfiniBand NDR (400 Gb/s per port x 256 ports ~= 12.8 TB/s) and the reason hyperscale training clusters increasingly look like 'one giant GPU' rather than 'a cluster of GPUs'. Beyond 256 GPUs the topology switches to InfiniBand or RoCE, and collective performance drops by an order of magnitude — sizing past 256 should account for that step.

• Per-GPU NVLink: 900 GB/s bidirectional (18 ports x 50 GB/s).
• Per-baseboard NVSwitch bisection: 3.6 TB/s (8 GPUs x 450 GB/s per direction).
• NVLink-domain ceiling: 256 GPUs, 57.6 TB/s bisection.
• Above 256 GPUs: InfiniBand NDR/XDR or Spectrum-X RoCE — plan for 5-10x latency uplift on cross-pod collectives.

Workload pattern A: Llama 3 70B inference at 32K context#

Single-replica throughput target, latency-sensitive endpoint. We size to two H100 SXM5 with TP=2 to fit the 70B weights plus a meaningful KV cache budget; FP8 reduces weight memory to ~35 GB per rank, leaving headroom for 16-32 concurrent sessions. The shortest path is `vllm serve` directly on the host — bind the OpenAI-compatible HTTP server, pin the replica to a single HGX baseboard via `CUDA_VISIBLE_DEVICES`, and the smoke-test is a single `curl`.

# 1) Install vLLM with FP8 support (Hopper requires CUDA 12.4+, driver R550+)
pip install "vllm==0.6.3" "torch==2.4.0"

# 2) Pin to the first two GPUs on the same HGX baseboard, then serve.
#    vLLM exposes an OpenAI-compatible /v1/chat/completions endpoint on :8000.
CUDA_VISIBLE_DEVICES=0,1 \
NCCL_P2P_LEVEL=NVL \
vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
  --tensor-parallel-size 2 \
  --quantization fp8 \
  --kv-cache-dtype fp8_e5m2 \
  --max-model-len 32768 \
  --max-num-batched-tokens 32768 \
  --gpu-memory-utilization 0.92 \
  --enable-prefix-caching \
  --disable-log-requests \
  --host 0.0.0.0 --port 8000

# 3) Smoke-test the endpoint
curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Meta-Llama-3-70B-Instruct",
    "messages": [{"role":"user","content":"Summarise NVLink 4.0 in one sentence."}],
    "max_tokens": 128
  }' | jq .

Workload pattern B: 70B QLoRA fine-tune#

QLoRA fine-tune of a 70B base model on 2x H100 SXM5 using `transformers` + `peft` + `bitsandbytes` + `trl`, launched with `accelerate launch`. NF4 base weights (~35 GB), BF16 LoRA adapters (~600 MB), paged AdamW optimiser state for adapters only, gradient checkpointing on every transformer block, Flash Attention 2 (FA3 is wired in via `attn_implementation="flash_attention_3"` on `transformers >= 4.46`). Peak working set lands around 70 GB per GPU at batch 2 / seq 4096.

• Launch on a 2x H100 node: `NCCL_P2P_LEVEL=NVL accelerate launch --num_processes 2 --mixed_precision bf16 train.py`.
• For multi-node: `accelerate launch --multi_gpu --num_machines N --machine_rank R --main_process_ip <head> train.py`, or switch to `torchrun --nproc_per_node 8 --nnodes N`.
• Monitor with `watch -n 2 nvidia-smi` and `tail -f out/llama3-70b-qlora/runs/*/events.out.tfevents.*` (TensorBoard).
• For higher-throughput Hopper-tuned kernels, swap the model loader for `unsloth` or wrap the same config in `axolotl` — both compile down to the `transformers` + `peft` primitives above.

# train.py — 70B QLoRA on 2x H100 SXM5
# Deps: pip install "transformers>=4.46" "peft>=0.13" "trl>=0.11" \
#                   "bitsandbytes>=0.43" "accelerate>=0.34" datasets
import torch
from datasets import load_dataset
from transformers import (AutoModelForCausalLM, AutoTokenizer,
                          BitsAndBytesConfig)
from peft import LoraConfig, prepare_model_for_kbit_training
from trl import SFTTrainer, SFTConfig

MODEL_ID = "meta-llama/Meta-Llama-3-70B-Instruct"

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, use_fast=True)
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    quantization_config=bnb,
    device_map="auto",                       # shard across the 2 H100s
    attn_implementation="flash_attention_2", # FA3 on transformers>=4.46
    torch_dtype=torch.bfloat16,
)
model = prepare_model_for_kbit_training(model, use_gradient_checkpointing=True)

lora = LoraConfig(
    r=64, lora_alpha=16, lora_dropout=0.05,
    target_modules="all-linear", bias="none", task_type="CAUSAL_LM",
)

ds = load_dataset("json", data_files="s3://my-bucket/customer-support-v3/*.jsonl",
                  split="train", streaming=False)

trainer = SFTTrainer(
    model=model, tokenizer=tokenizer, peft_config=lora, train_dataset=ds,
    args=SFTConfig(
        output_dir="./out/llama3-70b-qlora",
        num_train_epochs=3,
        per_device_train_batch_size=2,
        gradient_accumulation_steps=16,      # global batch 64 on 2 GPUs
        gradient_checkpointing=True,
        optim="paged_adamw_8bit",
        learning_rate=2e-4, lr_scheduler_type="cosine", warmup_ratio=0.03,
        bf16=True, max_seq_length=4096,
        logging_steps=10, save_steps=500, report_to="tensorboard",
    ),
)
trainer.train()
trainer.save_model("./out/llama3-70b-qlora/final")

Workload pattern C: Stable Diffusion XL serving#

Stable Diffusion XL 1.0 base + refiner at 1024x1024 on a single H100 PCIe with MIG-disabled. We use `diffusers` with `torch.compile` for the UNet and BF16 VAE; the workload is compute-bound rather than memory-bound, which makes the PCIe SKU competitive with SXM5 at a meaningfully lower hourly rate. For absolute peak throughput, a separate offline `trtllm-build` step compiles the UNet to a TensorRT engine, but the `diffusers` path below is what most teams run in production.

# sdxl_server.py — SDXL base + refiner on 1x H100 PCIe
# Deps: pip install "diffusers>=0.30" "transformers>=4.46" \
#                   "torch==2.4.0" accelerate safetensors fastapi uvicorn
import io, torch
from diffusers import StableDiffusionXLPipeline, StableDiffusionXLImg2ImgPipeline
from fastapi import FastAPI
from fastapi.responses import Response

BASE = "stabilityai/stable-diffusion-xl-base-1.0"
REFINER = "stabilityai/stable-diffusion-xl-refiner-1.0"

base = StableDiffusionXLPipeline.from_pretrained(
    BASE, torch_dtype=torch.bfloat16, variant="fp16", use_safetensors=True,
).to("cuda")
refiner = StableDiffusionXLImg2ImgPipeline.from_pretrained(
    REFINER, torch_dtype=torch.bfloat16,
    text_encoder_2=base.text_encoder_2, vae=base.vae,
    variant="fp16", use_safetensors=True,
).to("cuda")

# Hopper-tuned: SDPA attention is FA2 by default on torch>=2.4
base.unet = torch.compile(base.unet, mode="max-autotune", fullgraph=True)
refiner.unet = torch.compile(refiner.unet, mode="max-autotune", fullgraph=True)

app = FastAPI()

@app.post("/generate")
def generate(prompt: str, steps: int = 25, guidance: float = 7.0):
    latent = base(prompt=prompt, num_inference_steps=steps,
                  guidance_scale=guidance, denoising_end=0.8,
                  output_type="latent").images
    image = refiner(prompt=prompt, num_inference_steps=steps,
                    denoising_start=0.8, image=latent).images[0]
    buf = io.BytesIO(); image.save(buf, format="PNG")
    return Response(content=buf.getvalue(), media_type="image/png")

# Run with: uvicorn sdxl_server:app --host 0.0.0.0 --port 8000 --workers 1

Sizing and capacity planning#

Sizing tables we use internally to scope H100 footprints. All figures assume H100 SXM5, FP8 weights via the Transformer Engine, vLLM 0.6 with paged KV cache and prefix caching, and a healthy NVLink-local placement. Throughput is given in output tokens per second per replica at the listed concurrency; treat these as planning anchors, not contractual SLOs — verify on InferenceBench before production rollout.

• Training rule of thumb: 1 trillion training tokens x 70B parameters at BF16 needs roughly 250-350 H100-days on 64-GPU NVLink-domain clusters with Megatron-LM + Transformer Engine FP8.
• Memory ceiling for a single H100: weights + KV cache + activations + cuBLAS scratch < 78 GB. Above 78 GB, expect OOMs even with paged KV — drop precision, shrink context, or move to TP=2.
• AllReduce overhead at TP=8 inside one HGX-H100: ~6-9 % of step time for 70B BF16; jumps to 25-40 % the moment a rank crosses to a second NVLink domain over InfiniBand.
• For 500 RPS at 4K tokens output (mid-context chat traffic), Llama 3 70B FP8 needs roughly 6-8 H100 SXM5 replicas; size headroom for prefix-cache cold-start on rollouts.
• Spot/preemptible H100 capacity is viable for fine-tunes but not for production inference SLAs — eviction rates of 8-15 % per day are typical on hyperscaler spot.

Limits and quotas#

Default per-account caps you will hit. Hyperscaler quotas are vendor-defined and require support tickets to raise. Plan procurement around lead times of 8-26 weeks for committed H100 capacity in 2026.

Observability#

Production H100 observability is built around DCGM (Data Center GPU Manager) exporting Prometheus metrics, plus NCCL traces for collective hotspots and ECC counters for hardware health. The metrics below are the ones we alert on in every Yobitel-managed H100 deployment.

• DCGM_FI_DEV_GPU_UTIL — SM occupancy. Sustained < 60 % under load usually means dataloader stall, not under-provisioning.
• DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_FREE — framebuffer (HBM) used and free, in MiB. Alert when free < 4 GB on inference replicas.
• DCGM_FI_PROF_PIPE_TENSOR_ACTIVE — fraction of cycles the Tensor Cores are busy. The honest 'is the GPU actually doing math' metric.
• DCGM_FI_PROF_DRAM_ACTIVE — HBM bandwidth utilisation; pair with PIPE_TENSOR_ACTIVE to classify compute-bound vs memory-bound regimes.
• DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL — NVLink throughput per GPU; sudden drops correlate with single-port link-down events.
• DCGM_FI_DEV_GPU_TEMP / DCGM_FI_DEV_MEMORY_TEMP — die and HBM temperatures. Alert at die > 83 C (throttle threshold) and HBM > 95 C.
• DCGM_FI_DEV_ECC_SBE_VOL_TOTAL / DBE_VOL_TOTAL — single- and double-bit ECC error counts. Any non-zero double-bit error means quarantine the card.
• DCGM_FI_DEV_RETIRED_DBE / RETIRED_SBE — retired HBM pages; a steady climb predicts card failure within weeks.
• DCGM_FI_DEV_POWER_USAGE / DCGM_FI_DEV_POWER_VIOLATION — watts drawn and milliseconds throttled by power cap.
• DCGM_FI_DEV_THERMAL_VIOLATION — milliseconds throttled by thermal limit; correlate spikes with rack inlet temperature.

# Prometheus alert rules — H100 production fleet
groups:
- name: h100-health
  interval: 30s
  rules:
  - alert: H100ThermalThrottle
    expr: rate(DCGM_FI_DEV_THERMAL_VIOLATION[5m]) > 0
    for: 2m
    labels: { severity: warning }
    annotations:
      summary: "H100 {{ $labels.gpu }} thermal throttling"
      runbook: https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/feature-overview.html#thermal-management

  - alert: H100ECCDoubleBit
    expr: increase(DCGM_FI_DEV_ECC_DBE_VOL_TOTAL[1h]) > 0
    labels: { severity: critical }
    annotations:
      summary: "DBE ECC error on H100 {{ $labels.gpu }} — quarantine card"

  - alert: H100NVLinkDown
    expr: DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL < 700e9
    for: 5m
    labels: { severity: critical }
    annotations:
      summary: "H100 {{ $labels.gpu }} NVLink degraded below 700 GB/s"

  - alert: H100HBMNearFull
    expr: DCGM_FI_DEV_FB_FREE < 4096
    for: 10m
    labels: { severity: warning }

  - alert: H100TensorIdle
    expr: avg_over_time(DCGM_FI_PROF_PIPE_TENSOR_ACTIVE[15m]) < 0.30
      and on(gpu) DCGM_FI_DEV_GPU_UTIL > 70
    for: 15m
    labels: { severity: info }
    annotations:
      summary: "GPU busy but Tensor Cores idle — dataloader or kernel inefficiency"

Cost and FinOps#

H100 hourly pricing collapsed by roughly 60-70 % between 2023 and 2026 as supply caught up. In 2026 the public ranges below are typical; private commitments often clear 20-40 % under on-demand list. Pricing levers in order of impact: commitment term (1y reserved ~= 40 % off on-demand, 3y ~= 60 % off), neocloud vs hyperscaler (neocloud ~= 30-50 % cheaper at parity), FP8 enablement (~= 1.6x throughput vs FP16 baseline at the same hourly cost = same % cost reduction), and right-sizing replicas to NVLink-locality (avoiding cross-baseboard placement that doubles GPU count for the same TPS).

• Cost-per-million-output-tokens on Llama 3 70B FP8, 1x H100 SXM5 at $2.00/GPU-hr and 1,100 TPS sustained: roughly $0.50 per million tokens before margin.
• Switching from FP16 to FP8 with the Transformer Engine yields +1.6x throughput on H100 SXM5 at iso-context — a 38 % drop in cost-per-token.
• Reserving 3 years cuts effective $/GPU-hr roughly in half versus on-demand; only commit when steady-state utilisation exceeds 65 %.
• Idle replicas are the dominant overspend pattern in production inference fleets — set `minReplicas: 0` on non-critical endpoints with cold-start tolerance.
• Egress and inter-region data movement frequently exceed 10 % of total H100 bill at hyperscalers — collocate model artefacts with compute.

Security and compliance#

H100 ships with three independent isolation primitives — MIG, Confidential Compute (CC-on), and the standard CUDA process/IPC model — and the combination supports sovereign deployments under UK NCSC guidance, EU GDPR, US HIPAA and FedRAMP Moderate when paired with appropriate host hardening.

MIG provides hardware-enforced spatial partitioning: up to 7 instances per H100, each with isolated HBM, L2, NVDEC/NVENC and SM allocations. Inter-instance memory bandwidth contention is bounded by the partition. MIG slices appear to the OS as distinct PCIe devices, which means multi-tenant scheduling on Kubernetes via the NVIDIA Device Plugin is straightforward.

Confidential Compute mode (CC-on) encrypts all PCIe traffic between the H100 and the host with AES-256-GCM and seals HBM-resident pages so the host kernel cannot read them. Attestation is performed via SPDM-over-PCIe to NVIDIA's NRAS service; an attested boot binds the firmware version, driver, and the workload's measurement hash. CC-on costs roughly 3-7 % throughput on most inference workloads and is currently the only commercial GPU attestation path with FedRAMP Moderate coverage.

For Yobitel UK sovereign deployments the recommended posture is: MIG-off (full-card workloads only), CC-on, NCSC-aligned host hardening (CIS Ubuntu 22.04 LTS Level 2), NCSC Cloud Security Principles 1-14 evidence in the workspace audit log, and OFFICIAL-classification data segregation by workspace rather than by namespace.

• MIG: spatial isolation, 7 slices max, hardware-enforced memory and bandwidth partitioning.
• CC-on: cryptographic isolation, SPDM attestation, ~3-7 % throughput penalty.
• Per-replica IAM via Yobitel workspaces; encryption-at-rest for model artefacts; signed model provenance via Sigstore.
• Auditable: every Kubernetes admission event (Deployment, Job, InferenceService) lands in the cluster audit log; pair with Falco for runtime detection and Sigstore-verified container images for supply-chain provenance.
• GDPR: model weights and training data residency enforced at workspace level; cross-region inference requires explicit configuration.

Migration and alternatives#

When H100 is the right choice and when it isn't. The table below maps the practical migration paths in both directions; the code block below shows the real-world commands you can run today on existing infrastructure as a reference.

Two heuristics: pick H200 when memory pressure dominates (KV cache or weights); pick B200 only when you can absorb a new software stack and need FP4 throughput. Stay on H100 in every other case — the software lead alone usually justifies it through 2026.

# --- Equivalents you can run today against existing stacks ---

# 1) AWS: launch an 8x H100 p5 instance (note: requires quota)
aws ec2 run-instances \
  --image-id ami-0abcdef1234567890 \
  --instance-type p5.48xlarge \
  --key-name my-key \
  --subnet-id subnet-0abc...

# 2) GCP: a3-highgpu-8g (8x H100 SXM5)
gcloud compute instances create h100-train-01 \
  --machine-type=a3-highgpu-8g \
  --zone=europe-west2-a \
  --image-family=common-cu124 \
  --image-project=deeplearning-platform-release \
  --maintenance-policy=TERMINATE

# 3) Kubernetes (GPU Operator + Device Plugin): request 2 H100s
kubectl apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: { name: vllm-70b }
spec:
  replicas: 2
  selector: { matchLabels: { app: vllm-70b } }
  template:
    metadata: { labels: { app: vllm-70b } }
    spec:
      nodeSelector: { nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3 }
      containers:
      - name: vllm
        image: vllm/vllm-openai:v0.6.0
        args: ["--model","meta-llama/Meta-Llama-3-70B-Instruct",
               "--tensor-parallel-size","2","--max-model-len","32768",
               "--kv-cache-dtype","fp8_e5m2","--quantization","fp8"]
        resources:
          limits: { nvidia.com/gpu: 2 }
EOF

# 4) Direct vLLM serve (bare metal)
vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
  --tensor-parallel-size 2 \
  --quantization fp8 --kv-cache-dtype fp8_e5m2 \
  --max-model-len 32768

# 5) TensorRT-LLM engine build for H100
trtllm-build --checkpoint_dir ./hf_llama3_70b_fp8 \
  --output_dir ./engines/llama3-70b-h100 \
  --gemm_plugin fp8 --gpt_attention_plugin fp8 \
  --max_input_len 16384 --max_seq_len 32768 \
  --tp_size 2 --workers 2

Troubleshooting#

Operational issues we see most often on H100 fleets, ranked by frequency. Each has a definitive diagnosis and a fix path.

Where this fits in the Yobitel stack#

The H100 is the workhorse SKU across the Yobitel stack in 2026. Yobibyte — our AI-native platform — consolidates the open-source primitives shown above (NVIDIA GPU Operator, vLLM, KServe, Volcano, KubeRay, DCGM, Sigstore) into a single managed control plane: inference replicas, fine-tune jobs and notebooks all land on H100 (or H200 / B200) pools with NVLink-aware placement, FP8 enabled by default and DCGM alerts pre-wired. The vLLM and `accelerate` commands in this entry are exactly what Yobibyte reconciles under the hood on the customer's behalf.

Omniscient Compute — our cross-cloud capacity broker — indexes H100 SKUs across every connected hyperscaler and Tier-1/Tier-2 neocloud, normalises pricing onto the FinOps Foundation FOCUS spec, and arbitrages workloads to the cheapest region that meets the workspace's residency and compliance posture. When you ask Yobitel for 8x H100 SXM5 in the UK sovereign region, Omniscient Compute is the layer that finds it.

InferenceBench — our public, reproducible benchmarking harness — publishes H100 throughput, latency and cost-per-token numbers for every major open-weight model across vLLM, TensorRT-LLM, SGLang and TGI. The sizing tables in this entry are anchored on InferenceBench runs; the production numbers your team will see in steady state are typically within 10 % of the published figures. If you are sizing a 2026 H100 footprint, start with InferenceBench, lift the platform configuration into a Yobibyte manifest, and let Omniscient Compute pick the region.