InfiniBand NDR (Next Data Rate)

Overview#

InfiniBand NDR is the fifth high-rate generation of the InfiniBand standard (after SDR, DDR, QDR, FDR, EDR, HDR), defined by the IBTA in the 1.5 specification roadmap and timed to match the Hopper GPU launch. Each NDR port aggregates four lanes of 100 Gb/s PAM4 serial signalling for a usable line rate of 400 Gb/s per port; an HCA may present 1, 2, or 4 ports, and an aggregated NDR2 link uses 8 lanes for 800 Gb/s on a single cable.

Its relevance to AI infrastructure is direct: training a frontier-scale model requires synchronising gradients across hundreds or thousands of GPUs every optimiser step, and the time spent in those collectives directly bounds training throughput. NDR's combination of 400 Gb/s per port, sub-microsecond switch latency, hardware RDMA, and adaptive routing without packet loss is what keeps gradient AllReduce overhead in the single-digit percent of step time on H100-scale jobs.

By mid-2026, NDR is mature, broadly deployed and reasonably priced. It is the default training fabric in NVIDIA DGX SuperPOD reference architectures from 2023 onward, the dominant choice in NVIDIA-Partner neoclouds (CoreWeave, Lambda, Crusoe, Nebius, Yotta), and the safe pick for any new H100/H200/GB200 build that does not have an existing in-house Ethernet/Spectrum-X operations team. Yobitel NeoCloud uses InfiniBand NDR for tightly-coupled training pods in its UK and EU regions, with NCSC OFFICIAL alignment on the sovereign UK footprint.

This entry helps you pick the right AI fabric for your training cluster and understand what Yobitel runs on NeoCloud — what NDR delivers in practice, where it sits versus 400G Ethernet and the newer XDR generation, and how to size a non-blocking fat tree without surprises in the optics bill.

Specifications#

Authoritative figures: IBTA-defined NDR line rates plus the practical numbers an operator cares about when sizing a fabric. Throughput, latency, power and reach assume current Quantum-2 silicon and ConnectX-7 endpoints at the stated DOCA / MLNX_OFED levels.

Architecture: how NDR achieves lossless RDMA#

InfiniBand is fundamentally a credit-based, lossless, RDMA-first fabric — and NDR inherits that architecture rather than reinventing it. Three mechanisms together produce the headline behaviour:

Credit-based link-level flow control. Every link in the fabric maintains a count of receive buffers free at the downstream port. A sender cannot transmit a packet until the receiver has advertised enough credits. Buffers therefore never overflow, packets are never dropped due to congestion, and the retransmission tax that lossy Ethernet pays under load simply does not exist. This is the most important difference between InfiniBand and RoCEv2-on-Ethernet.

Hardware RDMA with GPUDirect. Remote Direct Memory Access lets one node read or write the memory of another node without involving the remote CPU. The HCA executes the transfer directly to or from registered memory pages; in a GPU cluster, GPUDirect RDMA extends this to GPU HBM via the PCIe peer-to-peer path, so an AllReduce across 1,024 GPUs never copies a tensor through host DRAM. The ConnectX-7 plus a Hopper or Blackwell GPU on the same PCIe Gen5 root complex (or NVSwitch-adjacent) is what makes this fast in practice.

Adaptive routing with SHIELD self-healing. Quantum-2 chooses among equal-cost paths on a per-packet basis using switch-local congestion telemetry, with the SHIELD mechanism rerouting around link faults within hundreds of nanoseconds. Combined with credit-based flow control, this keeps tail latency tight even when the fabric is running at 80-90 % offered load — the regime where lossy Ethernet's tail behaviour collapses.

Layered on top of these are partition keys (PKeys, the InfiniBand multi-tenancy primitive), service levels and virtual lanes (QoS), and the SHARPv3 in-network reduction engine that runs collective arithmetic inside the switch (see the SHARP entry for the details).

Form factor, cabling and optics#

OSFP twin-port is the NDR-era cage. Each cage accepts a twin-port cable carrying two independent 400 Gb/s links, or a single-port optic carrying one. This doubles port density at the switch — a 32-cage 1U switch presents 64 ports — at the cost of meaningful cabling discipline: a twin-port AOC plugged into a single-port host cage will negotiate at half the expected bandwidth and produce no error message.

For intra-rack runs (under ~3 m), passive copper DAC is the cheapest option. Beyond that, active optical cables (AOC) extend reach to ~30 m without the optic-per-end cost, and pluggable transceivers cover the longer reaches. The transceiver bill of materials is usually the largest single line item in an NDR fabric.

Software ecosystem: drivers, subnet manager, tooling#

Operating an NDR fabric is a software stack as much as a hardware deployment. The minimum supported set per cluster is below; mixing versions across hosts is the most common silent source of fabric flakiness.

• MLNX_OFED or DOCA-Host on every endpoint — `mlx5_core` driver, libibverbs, `rdma-core` utilities. DOCA-Host (DOCA 2.7+) is the supported path on RHEL 9 / Ubuntu 22.04+ in 2026.
• OpenSM (open source) or NVIDIA UFM Enterprise (commercial) as Subnet Manager. UFM adds topology visualisation, fabric-wide telemetry export, predictive failure detection, and Cyber-AI anomaly alerts; OpenSM is fine at single-pod scale.
• MLNX-OS as the switch operating system on MQM9700 appliances.
• Quantum-2 firmware aligned across all switches in the fabric. Mixed firmware causes sporadic SHARP CollNet failures and silent SHIELD-route rejections.
• Diagnostic utilities: `ibstat`, `ibhosts`, `iblinkinfo`, `ibdiagnet`, `ib_write_bw`, `ib_read_lat`. The `nccl-tests` AllReduce benchmark is the practical fabric validator.
• Optional integrations: Prometheus exporter via UFM Telemetry, Grafana dashboards from NVIDIA's `dcgm-exporter` companion repo, OpenTelemetry traces from training framework profilers (PyTorch Profiler, Nsight Systems).

# Verify NDR link rates across the host's HCAs
ibstat | grep -E "Active|Rate"
# Rate: 400 (4X NDR)
# Rate: 400 (4X NDR)

# Topology sweep + cable health check (run after every firmware bump)
ibdiagnet --pc \
  --get_cable_info \
  --get_phy_info \
  --counter_freq 5 \
  --out_ibdiagnet2_dir /var/log/ibdiagnet

# Point-to-point bandwidth test between two hosts
# server side:
ib_write_bw -d mlx5_0 -i 1 --report_gbits -F
# client side:
ib_write_bw -d mlx5_0 -i 1 --report_gbits -F <server-ip>
# Expect ~395 Gb/s payload, ~96-98 % of line rate.

# Full-cluster fabric validation
mpirun -np 256 -hostfile hosts \
  --mca pml ucx --mca btl ^vader,tcp,openib \
  -x NCCL_DEBUG=INFO -x NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3 \
  ./build/all_reduce_perf -b 8M -e 8G -f 2 -g 1

Sizing and capacity planning#

NDR fabrics are sized around the AllReduce collective that dominates data-parallel training. The default topology is a non-blocking fat-tree: every leaf has equal up- and down-link bandwidth, so any GPU can sustain full line-rate to any other GPU through the fabric. The tables below cover the common pod sizes; larger clusters add a third tier.

• Rail-optimised cabling: every GPU's HCA is mapped to a specific rail (1, 2, 3, 4 — matching the 4 HCAs per HGX H100 baseboard). Spines are colour-coded per rail. This avoids cross-rail traffic and simplifies cabling audits.
• Optics budget rule of thumb: at 1,024-GPU scale, optics cost roughly $1.2-1.6M and dwarf the switch BOM. At 4,096+ GPUs, optics can hit $5-8M and approach the GPU cost on a per-rack basis.
• Power: a fully loaded MQM9700 draws ~750 W typical (~1 kW peak). Plan 1 kW per leaf for PDU sizing.
• Cooling: switch fans are loud but standard front-to-back airflow; no liquid cooling required at NDR scale (unlike Quantum-3 which often sits in rear-door heat exchangers).
• Oversubscription: 2:1 leaf-to-spine oversubscription roughly halves switch and optics cost at the price of capping AllReduce throughput. Acceptable for inference fleets, rarely acceptable for training.
• Yobitel NeoCloud's UK training-pod reference design lands on the 1,024-GPU two-tier fat tree above (32 Quantum-2 leaves + 16 spines) as the standard sovereign-region building block; larger sovereign tenants land on the 2,048-GPU design and frontier-training reservations extend into the three-tier 4,096-GPU footprint.

Cost and TCO#

Switch silicon prices are negotiated; the figures below are list-adjacent indicative ranges in USD for new builds in early-to-mid 2026. Optic prices are softer (more vendors, more competition) and have dropped roughly 25 % year-on-year since 2024.

For a 1,024-GPU NDR build (single 8-host HGX = 64 GPUs x 16 hosts), the indicative full fabric bill of materials is roughly $2.5-3.5M (switches + HCAs + optics + cabling) — about 10-15 % of the underlying GPU spend. Quantum-2 is ~30-40 % more expensive per port than equivalent 400G Ethernet, but typically wins on operational simplicity for training-only fabrics.

Migration and alternatives#

NDR sits between two adjacent fabric choices on the bandwidth axis (HDR below, XDR above) and one alternative on the technology axis (Ethernet + RoCEv2). Choose deliberately.

• Migrating from HDR to NDR is mostly a swap of leaf and spine appliances plus HCAs and cables — NDR ConnectX-7 fits Gen5 hosts and the cabling system changes from QSFP56 to OSFP. Plan a coupled refresh, not a piecemeal one.
• Migrating from NDR to RoCEv2 means re-platforming the operations stack: PFC tuning, ECN/DCQCN tuning, switch buffer sizing, and per-flow telemetry. Budget 6-12 weeks of network-engineering time before the first production training run.
• Mixed NDR/HDR subnets are supported via Quantum-2's HDR-rate-capable ports but cap the slowest link's bandwidth across each topology level — segregate where possible.

Pitfalls and operational notes#

• Subnet Manager is mandatory and single-active. Run redundant SMs (one primary, one standby) at production scale; UFM HA is recommended above 256 nodes.
• Firmware drift is the most common silent failure: ConnectX HCAs, Quantum switches, and the SM should be kept on the same MLNX_OFED / DOCA release across the fabric. Pin a release per cluster and document the upgrade window.
• Cluster verification benchmarks (`ib_write_bw`, `nccl-tests` AllReduce) must be re-run after every firmware update — silent regressions in switch microcode have shipped more than once.
• Cable lengths and bend radius matter at NDR signalling rates. A single dirty OSFP optic in a 256-node cluster can manifest as random NCCL timeout errors hours into a multi-week training run.
• Mix lossy management traffic with RDMA on the same HCA only via IPoIB on a dedicated PKey — do not bridge generic Ethernet onto the same QSFP cages.
• SHARP is enabled per-job. The fact that the fabric supports it does not mean the job is using it; verify `NCCL_COLLNET_ENABLE=1` is set and the NCCL log shows `Connected ... using CollNet`.
• OSFP twin-port and single-port cages co-exist on Quantum-2; label every cable end and audit annually.
• Cooling: at high port utilisation, MQM9700 fan speed ramps audibly. Ensure rack-level cold-aisle containment to keep ASIC junction temperature below 95 C.