RoCEv2 (RDMA over Converged Ethernet, version 2)

Overview#

RDMA over Converged Ethernet version 2 (RoCEv2) is how RDMA gets carried over commodity Ethernet at data-centre scale. The first version of RoCE was a pure Layer 2 protocol — RDMA frames encapsulated in Ethernet, non-routable, confined to a single broadcast domain. RoCEv2 added UDP/IP encapsulation so the same RDMA semantics work across routed IP fabrics.

For AI infrastructure, RoCEv2 is the protocol that makes Ethernet a credible competitor to InfiniBand. It carries the same InfiniBand transport headers, supports the same Verbs API, and — given a properly tuned lossless or near-lossless Ethernet underlay — delivers comparable latency and throughput to native InfiniBand. The trade-off is the operational complexity of running lossless Ethernet: PFC, ECN, and DCQCN must all be configured and tuned correctly, and the network team needs the skills to diagnose pause storms, ECN mis-marking, and head-of-line blocking when something goes wrong.

By 2026, RoCEv2 underpins essentially every Ethernet AI fabric in production. NVIDIA's Spectrum-X (Spectrum-4 ASIC + BlueField-3 SuperNIC) is a RoCEv2 platform with vendor-specific extensions. Broadcom's Tomahawk-5-based AI fabrics from Arista, Cisco, and Juniper are RoCEv2 platforms. AWS EFAv2 is RoCEv2-derived. The Ultra Ethernet Consortium specifications that aim to supplant RoCEv2 for AI workloads (UEC 1.0, ratified 2025) keep wire-format compatibility for the migration path.

RoCEv2 is the lossless-Ethernet alternative Yobitel NeoCloud offers for cost-sensitive multi-node serving and for sovereign UK pods where the in-house network team operates Cumulus-Linux Ethernet rather than InfiniBand. This entry helps you operate RoCEv2 in production and reach the line-rate behaviour you paid for — the PFC/ECN/DCQCN tuning that turns a working RoCE fabric into a non-regressing one.

Quick start: enable RoCEv2 on a ConnectX-7 + Cumulus Linux switch#

The minimum config to bring up a working RoCEv2 link. This assumes a ConnectX-7 NIC with the `mlx5_core` driver on a recent Linux distribution (RHEL 9.4+, Ubuntu 22.04 LTS+) and a Spectrum-4 or Tomahawk-5 switch running Cumulus Linux 5.x. Adjust DSCP/PCP values to match your switch fabric's class-of-service plan.

# --- Host side: ConnectX-7 + mlx5 ---
# 1) Confirm RoCEv2 capability on the HCA
ibv_devinfo -d mlx5_0 | grep -E "transport|rocev2"
# transport:    InfiniBand
# active_mtu:   4096 (5)

# 2) Set the GID index for RoCEv2 over IPv4 (typically index 3)
#    show_gids prints the GID table; the v2 entries are tagged "RoCE v2"
show_gids
# DEV     PORT  INDEX  GID                  IPv4         TYPE        NDEV
# mlx5_0  1     0      fe80::... (link)     -            RoCE v1     enp1s0f0
# mlx5_0  1     1      fe80::... (link)     -            RoCE v2     enp1s0f0
# mlx5_0  1     2      ...                  10.1.1.10    RoCE v1     enp1s0f0
# mlx5_0  1     3      ...                  10.1.1.10    RoCE v2     enp1s0f0  <-- use this

# 3) Set DSCP 26 (PCP 3) for RoCEv2 traffic at the IP layer
#    Map application priority -> RoCE -> DSCP via mlnx_qos
mlnx_qos -i enp1s0f0 --trust=dscp
mlnx_qos -i enp1s0f0 --pfc=0,0,0,1,0,0,0,0      # enable PFC on priority 3

# 4) Test bandwidth between two hosts (server then client)
ib_write_bw -d mlx5_0 -F --report_gbits -x 3    # server, GID 3
ib_write_bw -d mlx5_0 -F --report_gbits -x 3 <server-ip>   # client

# --- Switch side: NVIDIA Cumulus Linux 5.x (Spectrum-4 example) ---
# /etc/nvue.d/roce.yaml
cat <<'EOF' | nv config patch --apply -
- set:
    interface:
      swp1-64:
        ip:
          neighbor-discovery:
            router-advertisement:
              enable: off
        link:
          mtu: 9216                          # jumbo, leaves headroom for VXLAN
        qos:
          pfc:
            switch-priority:
              3:
                enable: on                   # match host PFC priority
          congestion-control:
            wred-ecn:
              enable: on
              min-threshold: 150000          # bytes; tune per buffer depth
              max-threshold: 1500000
              probability: 100
    qos:
      mapping:
        dscp-to-switch-priority:
          26: 3                              # DSCP 26 -> SP3
EOF

How it works: packet structure#

A RoCEv2 packet is built up as follows, outermost to innermost: Ethernet header -> IP header -> UDP header (dst port 4791) -> InfiniBand Base Transport Header (BTH) -> InfiniBand Extended Transport Header (ETH, optional) -> payload -> Invariant CRC.

Because the BTH is unchanged from native InfiniBand, the NIC's RDMA engine can process inbound RoCEv2 packets through the same hardware path as native IB packets after the UDP/IP envelope is stripped. This is what lets a single ConnectX HCA family support both transports.

RoCEv2 packet on the wire:

+-----------+--------+--------+--------+-----+---------+------+
| Ethernet  | IPv4/6 |  UDP   |  BTH   | ETH | Payload | ICRC |
| 14 bytes  | 20/40  | 8 bytes| 12     | 4   | 0..4096 | 4    |
+-----------+--------+--------+--------+-----+---------+------+
                      dport
                      4791
                                ^------- InfiniBand transport
                                         (unchanged from native IB)

Why RoCEv2 needs lossless Ethernet#

RDMA transport assumes near-zero packet loss. The InfiniBand transport originally retransmitted entire transfers on any loss — a go-back-N style recovery that is catastrophic for large messages under congestion. Modern ConnectX-7+ NICs implement selective repeat retransmission (SR) and improved loss recovery, but even with SR, high loss still collapses throughput. The whole point of the PFC/ECN/DCQCN apparatus is to make loss a rare event.

Three mechanisms layered together produce near-lossless behaviour on Ethernet. ECN/DCQCN do the bulk of the work in steady state by throttling senders before congestion becomes severe; PFC is the safety net for transient bursts that ECN-driven backoff cannot react to quickly enough. PFC alone (without ECN) creates pause storms and head-of-line blocking under sustained load; ECN alone (without PFC) loses packets during the gap between mark and rate-cut response.

• PFC (Priority Flow Control, IEEE 802.1Qbb): per-priority pause frames that stop upstream senders when downstream buffers fill. Lowest-level loss prevention. Hop-by-hop, no end-to-end signalling.
• ECN (Explicit Congestion Notification, RFC 3168): switches mark packets in the IP header when queues build, before drops occur. End-to-end feedback signal.
• DCQCN (Data Centre Quantised Congestion Notification, SIGCOMM 2015): the end-host congestion control algorithm that translates ECN marks into per-QP rate cuts. Hardware-implemented on the NIC.

Reference: NIC sysctls, kernel module options, and verbs#

Operational reference for the NVIDIA `mlx5` driver — by far the most common RoCEv2 NIC. The relevant sysctls, module parameters, and verbs queries an operator touches in production.

# Inspect current RoCE-relevant firmware config
mlxconfig -d /dev/mst/mt4131_pciconf0 query | grep -E "ROCE|CNP|ECN"

# Persistently set DCQCN-friendly config (requires NIC reboot)
mlxconfig -d /dev/mst/mt4131_pciconf0 set \
  ROCE_CC_ALGORITHM_P1=DCQCN \
  CNP_DSCP_P1=48 \
  CNP_802P_PRIO_P1=6 \
  ROCE_NEXT_PROTOCOL_P1=4791

# Live per-port congestion counters (the canonical RoCE perf signal)
ethtool -S enp1s0f0 | grep -E "rx_pause|tx_pause|rx_congestion|ecn_marked|rx_out_of_buffer"

# Per-QP retransmission counters (raise concern if non-zero growth)
cat /sys/class/infiniband/mlx5_0/ports/1/hw_counters/np_cnp_sent
cat /sys/class/infiniband/mlx5_0/ports/1/hw_counters/rp_cnp_handled
cat /sys/class/infiniband/mlx5_0/ports/1/hw_counters/out_of_sequence

Switch configuration: Cumulus Linux, Arista EOS, SONiC#

Switch-side config differs by NOS but the concepts are identical: classify RoCE by DSCP, map to a dedicated priority queue, enable PFC on that priority, enable WRED/ECN marking with thresholds appropriate to the buffer depth. Examples below for the three most common NOSes in AI fabrics.

• DSCP 26 / PCP 3 / Switch-Priority 3 is the de facto convention for RoCEv2 on AI fabrics. Pick a different value if your fabric already uses SP3 for something else — but document the mapping cluster-wide.
• WRED/ECN thresholds depend on switch buffer depth. Shallow-buffer Tomahawk: min ~100 KB, max ~1 MB. Deep-buffer Jericho: min ~1 MB, max ~10 MB. Wrong thresholds either fail to mark (loss) or over-mark (throughput collapse).
• PFC headroom (the extra buffer reserved per pause-enabled priority) must be sized for the longest cable round-trip in the fabric. Default headroom assumes ~10 m fibre; ~100 m runs need explicit headroom uplift.
• Enable PFC watchdog. Stuck pause frames take a port out of service in seconds; PFC watchdog detects and breaks the loop within milliseconds.

# --- NVIDIA Cumulus Linux 5.x (Spectrum-4) ---
nv set qos congestion-control switch-priority 3 wred-ecn enable on
nv set qos congestion-control switch-priority 3 wred-ecn min-threshold 150KB
nv set qos congestion-control switch-priority 3 wred-ecn max-threshold 1500KB
nv set qos pfc switch-priority 3 enable on
nv set qos mapping dscp-to-switch-priority 26 3
nv config apply

# --- Arista EOS (Tomahawk-5 / Jericho) ---
qos map dscp 26 to traffic-class 3
priority-flow-control on
priority-flow-control priority 3 no-drop
queue 3 random-detect ecn minimum-threshold 150 kbytes maximum-threshold 1500 kbytes max-mark-probability 100

# --- SONiC (community / Enterprise) ---
config qos clear
config qos reload
# QoS templates live in /etc/sonic/qos.json; the AI-fabric template
# ships from the platform vendor with DSCP 26 -> TC 3 + PFC priority 3
# pre-configured. Validate with:
show pfc counters
show queue counters

Workload patterns#

RoCEv2 carries three distinct traffic patterns in an AI cluster, each with different congestion characteristics. Knowing which dominates your workload tells you which tuning lever to reach for first.

• Training collectives (AllReduce, AllToAll): bursty, large messages (8 MB - 8 GB), strongly congestion-correlated across QPs. Dominant lever: DCQCN parameters (Kmin, Kmax, Pmax, Rai, Rhai). Default DCQCN tuned for 100G; needs re-tuning at 400G and 800G.
• Storage I/O over RDMA (NVMe-oF, Lustre, GPFS): steady-state large reads/writes, less correlated across QPs. Dominant lever: PFC pause behaviour. Buffers must absorb sustained read bursts without pausing the training plane.
• Inference KV-cache transfers (vLLM disaggregated, prefill-decode separation): small-to-medium messages (1-64 MB), latency-sensitive, low-rate. Dominant lever: ECMP hashing — avoid placing prefill->decode flows on congested links. Adaptive routing helps when available.

Sizing and capacity planning#

Real-world line-rate numbers at 400G and 800G, ConnectX-7/8 endpoints, Spectrum-4 or Tomahawk-5 switches, DCQCN tuned, PFC enabled on SP3. Treat as planning anchors; verify with `ib_write_bw` and `nccl-tests`.

• At 400G, achievable single-flow throughput is 96-99 % of line rate when PFC/ECN/DCQCN are correctly tuned. Anything less suggests a misconfiguration.
• At 800G, single-flow throughput drops to 95-99 % of line rate due to packet-loss recovery overheads — verify with `ib_write_bw -F` and watch for retransmission counter growth.
• Tail latency under load is the operationally critical number. p99 above 200 us at 70 % offered load almost always indicates DCQCN under-tuning (slow rate recovery after a CNP).
• Plan 10-20 % bandwidth headroom on every link; running RoCEv2 fabrics above 80 % sustained load amplifies tail-latency variance.
• Yobitel NeoCloud's sovereign UK reference design lands on the 400G Spectrum-X (BlueField-3 + Spectrum-4) row above for Ethernet-preferring tenants — the same PFC/ECN/DCQCN profile ships in the cluster image, validated against the headline AllReduce numbers before customer access opens.

Observability#

RoCE health surfaces in three places: NIC hardware counters (Mellanox `hw_counters`), switch port counters (PFC pause counts, ECN marks, drops), and NCCL/MPI job logs. Production fabrics export all three to Prometheus + Grafana via the `dcgm-exporter`, NVIDIA UFM telemetry (works on RoCE too) or SONiC's `gnmi` exporter.

• Per-port: `rx_pause_count`, `tx_pause_count`, `rx_out_of_buffer`, `ecn_marked_packets`. Pause counts > 0 are normal; sustained growth is a problem.
• Per-QP: `out_of_sequence` (selective-repeat triggers), `np_cnp_sent` (CNPs generated by this node), `rp_cnp_handled` (CNPs received and rate-cut). Track per-job, alert on rates.
• Job-level: NCCL log lines `Connected ... using IB` confirm RoCE path active; `falling back to TCP` is the disaster mode — alert on any occurrence.
• Switch-level: PFC pause rate, ECN mark rate, WRED drop count, buffer occupancy per priority. Drops on the RoCE priority are always a red flag.

# Per-port RoCE health on a host
ethtool -S enp1s0f0 | grep -E "rx_pause|tx_pause|rx_out_of_buffer|ecn_marked"

# Per-QP counters (one per CX-7 port)
for c in $(ls /sys/class/infiniband/mlx5_0/ports/1/hw_counters); do
  printf "%-30s %s\n" "$c" "$(cat /sys/class/infiniband/mlx5_0/ports/1/hw_counters/$c)"
done

# Prometheus exporter snippet — UFM emits these for RoCE fabrics
# Sample queries:
#   rate(rx_pause_count[5m]) by (host, port) > 100
#   sum(ecn_marked_packets) by (job) / sum(tx_packets) by (job) > 0.05
#   absent_over_time(nccl_job_using_ib[5m])  -- alert on TCP fallback

Cost and FinOps#

RoCEv2 itself has zero licence cost; the protocol is implemented in NIC hardware and switch silicon you already own. The cost-versus-InfiniBand argument is about hardware bill of materials and the operator-skill premium.

• The headline TCO win for RoCE is at large fabric scale (1k+ GPUs); below 256 GPUs the per-switch savings rarely justify the operational complexity over plug-and-play InfiniBand.
• Add 6-12 weeks of network-engineering time per new RoCE fabric for PFC/ECN/DCQCN tuning. Skipping this is the most common reason RoCE deployments underperform InfiniBand in the first 6 months.
• Multi-vendor sourcing is the secondary win: RoCE switches can be Tomahawk, Spectrum, Jericho, Silicon One — InfiniBand is single-vendor.

Security and compliance#

RoCEv2 inherits IP security primitives — IPsec, MACsec, VLAN segmentation — that pure InfiniBand does not have. For multi-tenant clusters, this matters: VRF isolation per tenant on the RoCE fabric is straightforward; the equivalent on InfiniBand requires partition-key (PKey) management discipline that fewer operators have.

Confidential RDMA: NVIDIA Hopper+ with Confidential Compute mode (CC-on) encrypts PCIe and HBM traffic; the RoCEv2 wire-format itself is unencrypted, but the data inside (GPU-to-GPU tensors) is protected by the CC envelope. For host-to-host plaintext-on-the-wire concerns, MACsec at the link layer is the operational answer; full end-to-end IPsec adds CPU overhead and is rarely chosen.

Compliance: RoCE fabrics qualify under NCSC Cloud Security Principles (UK), GDPR Article 32 (technical and organisational measures), HIPAA, and SOC 2 reference designs the same way any other Ethernet fabric does. The novel concerns are around tenant-isolation evidence (VRF + PKey audit trails) and against side-channel attacks on shared infrastructure (CC-on attestation).

Migration and alternatives#

RoCEv2 is the default RDMA-over-Ethernet today, but two adjacent technologies bracket it: iWARP (RDMA over TCP/IP) below, and the Ultra Ethernet Consortium 1.0 transport above. Both matter for migration planning.

• Migrating from InfiniBand to RoCEv2: re-platforming exercise, not a swap. Budget 6-12 weeks per fabric for PFC/ECN/DCQCN tuning and operator training. Run both fabrics in parallel during cut-over.
• Migrating from RoCEv2 to UEC 1.0: the UEC transport keeps RoCEv2's UDP/4791 wire format for handshake compatibility, but adds packet spraying, selective repeat, and rich per-flow telemetry. Switch silicon upgrade (Spectrum-X SN5600 or Tomahawk 5) plus SuperNIC required; no application changes.
• Choosing iWARP over RoCEv2: only for WAN-distance RDMA (storage replication across DCs) or environments where the lossless tax is genuinely impractical. Almost never the right choice for new AI training fabrics.

Troubleshooting#

The RoCE failure-mode catalogue is large but the high-frequency entries are predictable. Map symptom to cause; verify with the listed action.

Where this fits in the Yobitel stack#

Yobitel runs both InfiniBand NDR/XDR and RoCEv2 fabrics in production: InfiniBand is the default on the H100 training pods and the GB200 NVL72 racks where single-vendor predictability matters most; RoCEv2 on Spectrum-X is the default on the H100/H200 sovereign UK pods where Cumulus-Linux-based operations align with the in-house Ethernet skill set. Both fabrics terminate into the same Yobitel GPU Cloud control plane and look identical to a Yobibyte customer — the difference is invisible above the cluster abstraction.

For customers running directly on Yobitel GPU Cloud rather than Yobibyte, the cluster image ships with the appropriate `/etc/profile.d/nccl.sh` profile: RoCE pods include the GID-index pinning, NCCL_IB_TC=106, and PFC-verified switch config; InfiniBand pods include the SHARP-enabling profile. The choice of fabric is exposed at provisioning time so workloads with specific fabric requirements can target accordingly.