MetaRoCE is a transport layer for AI infrastructure where the network becomes part of the critical path. For distributed training and inference, this is important because even minor loss or delay in the network directly impacts throughput and tail latency.
The weak point here is not a single specific link, but the behavior of the system under load. In clusters of hundreds of thousands of GPUs, the network is on the critical path because collective operations like all-reduce and all-to-all synchronize thousands of accelerators. In inference, the situation is similar: low latency between sharded parts of the model affects response times for hundreds of millions of users. If the network starts to introduce friction, compute capacity sits idle.
Meta chose MetaRoCE as the protocol that was originally designed for Ethernet and scales up to million-GPU. The core idea shifts the location of intelligence: the fabric sees packets, while the NIC sees intent. This is a compromise, but a pragmatic shift. Instead of forcing the network to be lossless and maintain frame order, the protocol accepts lossy Ethernet as the norm and shifts the responsibility for routing, recovery, and balancing to the endpoint.
The implementation revolves around several solutions. MetaRoCE marks the network into many fine-grained logical paths and maintains telemetry for each: per-path RTT, ECN state, and utilization. Packets are sprayed across different paths and can arrive out of order by design. This eliminates reorder buffers and head-of-line blocking, with data written directly to the final memory based on the destination specified in each packet. For losses, selective acknowledgment is used: if a gap appears in the 256-bit SACK bitvector, it is interpreted as a loss, and the missing packet is retransmitted along the same path.
It is also important that MetaRoCE separates transport semantics from queue behavior. Writes carry the destination address in each packet, while Sends use posted receive buffers, allowing messages to be delivered correctly without a round trip for confirmation of the write location. For congestion control, the protocol combines ECN-based sender-driven AIMD and receiver-driven fair-share rate hints. The window is maintained both at the path level and the connection level, so that a congested or broken link slows down only one route, not the entire flow.
From an operational perspective, this is a significant simplification. MetaRoCE does not require PFC, pause frames, packet trimming, in-network telemetry, credit-based flow control, or switch-side spraying. It relies on what is already available in Ethernet: ECN and ECMP. This makes the protocol compatible with fat-tree, multiplane, deep-buffer, and shallow-buffer fabrics, as well as vendor clouds where network configuration does not belong to a single team. This approach reduces dependence on specialized hardware logic and maintains optimization freedom on the fabric side.
There is also an important architectural effect for the QP model. In traditional RDMA, scaling often hits a wall with the number of queue pairs, as each additional connection brings its own state and separate congestion window. MetaRoCE separates the ordered stream above from the bandwidth below. One connection can carry many independent ordered streams, while the paths below multiply. This preserves the NIC state from growing alongside the workload’s parallelism.
In practice, the protocol has already been tested on AMD Pensando programmable NICs. On a 64-node AMD GPU cluster with RCCL collectives, MetaRoCE was compared with RoCEv2 on all-reduce and all-to-all. In initial data, it consistently showed higher throughput and lower flow completion times. With 1% packet loss, it maintained about 86% throughput, and with 10% loss, it continued to provide useful bandwidth instead of a sharp collapse. On a multiplane topology with 4 and 8 planes, throughput scaled linearly with the number of planes, and during simulated plane failures, traffic was redistributed without the involvement of the application or operator.
The conclusion of this design is quite clear. MetaRoCE does not attempt to make Ethernet perfect. It makes the network predictable under conditions of loss, uneven load, and partial failures. For AI infrastructure, this is a practical improvement because it maintains performance under normal conditions and provides the system with decent behavior when situations worsen. It is also important that Meta opens the specification, compliance suite, and software reference implementation through OCP, which affects not only a single implementation but also the possibility of inter-vendor compatibility.