NVIDIA Spectrum-X

Overview#

Spectrum-X is NVIDIA's answer to the question of whether Ethernet can be made to behave well enough for AI training. The platform combines the Spectrum-4 ASIC (51.2 Tb/s aggregate switching capacity at 800G port speed) with the BlueField-3 SuperNIC on the endpoint side. Together they implement a set of Ethernet extensions — packet-spraying adaptive routing, AI-tuned per-flow congestion control, and performance isolation — that mimic InfiniBand's lossless behaviour while preserving Ethernet's operational familiarity, multi-vendor switch sourcing, and lower per-port cost.

The platform was introduced at NVIDIA Computex 2023 alongside the Israel-1 supercomputer, an internal NVIDIA reference build of 4,096 H100s on a pure Spectrum-X fabric. By 2024 Spectrum-X had become an officially supported alternative to InfiniBand in DGX SuperPOD reference architectures, with hyperscalers (Microsoft Azure ND H200 v5 series, Oracle Cloud OCI Supercluster), several US neoclouds (xAI Colossus, Vultr), and a growing number of sovereign / regional AI clouds standardising on it for new builds. By 2026 the platform is on its second generation of endpoint silicon (BlueField-3 in production, BlueField-4 sampling) and remains the canonical Ethernet-for-AI design.

Spectrum-X is one of the AI-Ethernet options Yobitel evaluates for next-generation NeoCloud regions, alongside InfiniBand XDR — the choice per region is driven by sovereign-skill availability (Cumulus Linux ops vs InfiniBand specialists) and tenant TCO targets. This entry helps you pick the right AI fabric for your training cluster and understand what Yobitel runs on NeoCloud, including the cost and operational differences from a Quantum-3 XDR build at the same port speed.

Specifications#

Authoritative figures for the SN5600 (flagship 800G leaf/spine appliance) and the BlueField-3 SuperNIC (the endpoint). Spectrum-X also includes the SN5400 (400G variant for mixed builds) and the older Spectrum-3 SN4000 family that still ships at 200/400G for storage-tier fabrics.

Architecture: what makes Spectrum-X different from generic Ethernet#

Spectrum-X is, in essence, RoCEv2 with three AI-specific enhancements that close most of the technical gap to InfiniBand. Operators who already understand vanilla RoCEv2 (see the RoCEv2 entry) need to layer these three behaviours on top.

1. Collective-aware adaptive routing (packet spraying). Spectrum-4 routes RoCEv2 elephant flows (training AllReduce, AllToAll) by spraying packets across all available equal-cost paths on a per-packet basis, then relies on the BlueField-3 SuperNIC at the receiver to re-order packets back into sequence before delivering to the host's RDMA engine. The switch knows the difference between elephant flows (the training collective) and mice flows (control plane, IPMI, monitoring) via DPI of the BTH; mice continue to use static ECMP hashing for stable ordering. The result: a 4,096-GPU AllToAll that would tail-bound on a static-ECMP RoCE fabric runs near uniformly across all spine uplinks.

2. AI-tuned congestion control. Stock DCQCN tunes well for storage and microservices traffic; it under-responds to the bursty, synchronised, correlated nature of training collectives. The Spectrum-X variant uses BlueField-3's hardware telemetry (per-QP one-way delay, instantaneous receive buffer occupancy, CNP rate) to feed a per-flow rate-control loop that reacts in microseconds rather than milliseconds — much closer to InfiniBand's credit-based behaviour. Tunable via DOCA but the defaults are AI-workload-aware.

3. Tenant performance isolation. In a multi-tenant pod (multiple training jobs sharing the same fabric), noisy-neighbour effects on tail latency are the killer. Spectrum-X uses per-tenant priority queues + dedicated buffer pools on Spectrum-4 to bound how much fabric capacity any one tenant can consume, plus per-tenant ECN marking thresholds. Job A's AllReduce burst no longer adds 500 us to Job B's tail latency.

Above the silicon, the BlueField-3 SuperNIC does the heavy lifting on the endpoint: re-ordering sprayed packets, executing the AI CC algorithm in hardware, and emitting per-flow telemetry to the switch for the closed-loop tuning. A Spectrum-X fabric without BlueField-3 endpoints is just RoCEv2 with extra steps; the magic is the joint optimisation.

Form factor and physical deployment#

The SN5600 is a 2U appliance with 64 OSFP cages and front-to-back airflow. Power draw at full 800G utilisation across all 64 ports is roughly 1.5 kW; rack PDU planning should assume 2 kW per leaf. The BlueField-3 SuperNIC ships in FHHL and HHHL PCIe Gen5 x16 form factors as well as OCP NIC 3.0; choice depends on the host chassis.

Software ecosystem#

Spectrum-X switches run NVIDIA Cumulus Linux 5.x or community SONiC — both standard Ethernet NOSes operationally familiar to any Ethernet team. Cumulus is the supported path for AI-fabric features (NetQ telemetry integration, NV-API automation); SONiC supports the silicon but Spectrum-X-specific tuning has fewer turnkey defaults.

• Cumulus Linux 5.x — primary supported NOS. Configured via NV CLI (`nv set / nv config apply`) or NV API (REST/gRPC). Ships with AI-fabric default templates.
• NetQ — fabric telemetry, flow monitoring, validation engine. Replaces what UFM does for InfiniBand; integrates with Prometheus, Grafana, ServiceNow.
• DOCA 2.7+ on BlueField-3 endpoints — the host kernel module + userland that exposes the AI CC algorithm, packet-spraying re-order engine, and per-flow telemetry.
• NVIDIA Air — cloud-hosted digital twin of the fabric for validation before bring-up.
• Optional: NVIDIA Mission Control (NV-MC) — top-of-rack-to-job-submission management for DGX SuperPOD-class Spectrum-X clusters.
• Standards-based observability: Prometheus exporters, OpenTelemetry traces, sFlow/IPFIX flow records. All available via NetQ or directly from Cumulus.

# Bring up a Spectrum-X leaf-spine pair with default AI-fabric template
# Apply on the leaf:
nv set system aaa user nvue-admin role system-admin password '...'
nv set interface swp1-64 link mtu 9216
nv set interface swp1-32 type swp-leaf-server         # downlink to BF-3 host
nv set interface swp33-64 type swp-leaf-spine         # uplink to spine
nv set qos roce mode lossless                         # turnkey AI-fabric defaults
nv set qos roce congestion-control algorithm spectrum-x-cc
nv config apply

# Apply on the spine:
nv set interface swp1-64 link mtu 9216
nv set interface swp1-64 type swp-spine-leaf
nv set qos roce mode lossless
nv set qos roce congestion-control algorithm spectrum-x-cc
nv config apply

# Verify Spectrum-X-specific behaviour
nv show qos roce
nv show platform congestion-control
netq show fabric utilisation
netq show events --severity warning

Sizing and capacity planning#

Spectrum-X scales from a single-rack pod (8 hosts, 64 GPUs) to a 32,000-GPU AI factory in three tiers. The reference cluster sizes below are NVIDIA-published validated designs; intermediate sizes are linear interpolations.

• Rail-optimised cabling: each BlueField-3 SuperNIC port is mapped to one of 8 rails, with spines colour-coded per rail. Identical to InfiniBand fat-tree cabling discipline.
• Oversubscription: 1:1 (non-blocking) is the default for training; 2:1 spine oversubscription halves spine count for inference / batch fabrics where AllReduce isn't the bottleneck.
• Power: 1.5 kW per leaf at full load, 1.0 kW for spine (less optic density). Plan 2 kW PDU headroom per leaf.
• Cooling: front-to-back airflow; SN5600 fits standard cold-aisle hot-aisle racks. No liquid cooling required at 800G in 2026.
• Yobitel's NeoCloud reference design for Ethernet-preferring sovereign tenants lands on the 1,024-GPU two-tier Spectrum-X build above as the standard footprint, scaling to the 4,096-GPU three-tier shape for frontier-training reservations.

Cost and TCO versus InfiniBand XDR#

Spectrum-X's commercial pitch is comparable AI training performance at 60-70 % of the cost of an equivalent InfiniBand XDR fabric. The figures below are indicative USD ranges for new builds in early-to-mid 2026; negotiated pricing varies meaningfully.

At the same fabric scale, Spectrum-X is consistently 30-35 % cheaper than InfiniBand XDR on bill of materials. AllReduce throughput is within ~5 % at large messages, AllToAll within ~10 % — the gap is small enough that the cost saving wins for most operators with existing Ethernet ops capability. The exception is hyperscale-frontier training (50k+ GPUs) where InfiniBand's slightly tighter tail latency still wins.

Migration paths#

Spectrum-X is most often deployed in one of two migration shapes: greenfield (new AI fabric, no legacy), or brownfield replacement of generic Ethernet + RoCEv2. Migration from InfiniBand to Spectrum-X is less common but increasing — driven by TCO at scale.

• Brownfield from generic Ethernet: the win is roughly 25-40 % AllReduce throughput uplift at the same wire speed, plus tail-latency reduction in multi-tenant pods. Justifies the BlueField-3 rollout for most large-fabric operators.
• Brownfield from InfiniBand: the win is 30-35 % TCO savings on the next fabric refresh. Run both fabrics in parallel during cut-over (12-16 weeks); pod-by-pod migration; revalidate every training job's NCCL performance before retiring the IB fabric.
• Spectrum-X is forward-compatible with BlueField-4 (sampling 2026, volume 2027) and the SN6000 Spectrum-5 switch (sampling 2026) — buys an extra hardware generation of headroom.

Pitfalls and operational notes#

• Spectrum-X-specific congestion control requires BlueField-3 SuperNIC on every endpoint. ConnectX-7 endpoints still get RoCEv2 with adaptive routing, but the full AI-tuned CC loop needs BF-3.
• Packet spraying assumes the receiver re-orders correctly. A misconfigured BlueField-3 (DOCA version drift) will deliver out-of-order packets to the RDMA engine, which collapses throughput silently. Pin DOCA version per cluster.
• NetQ telemetry retention defaults to 7 days. For incident post-mortem capability, raise to 30-60 days and budget the storage.
• Cable bend radius at 800G is tight; survey rack cabling plans before installation. Dirty optics produce slowly-growing symbol-error counters that flap a port days later.
• Mixed Spectrum-4 and older Spectrum-3 in the same fabric works but caps the slower side; segregate where possible.
• PFC watchdog must be enabled. Spectrum-X's congestion control reduces the need for PFC under normal load, but PFC remains the safety net; a stuck pause frame still kills a port without the watchdog.
• BlueField-3 also runs DOCA services (storage offload, security, host management) — coordinate the AI-fabric DOCA versions with the storage / security teams' DOCA expectations.