MetaRoCE is an attempt to transfer intelligence from the network to the edge of the system. For AI clusters, this is important because the network often becomes the critical path for training and inference.
Meta encounters the same boundary when a cluster grows to hundreds of thousands of GPUs and is distributed across data centers and regions. For distributed AI training, the slowest part is not computation, but synchronization through all-reduce and all-to-all. For inference, the problem is different, but the logic is the same: low latency between model shards directly affects response time. Even minor network friction leaves some compute idle.
Standard RoCE relies on delivery order and lossless behavior of the network. Traditionally, PFC and packet spraying limits are used for this, which do not work well with multiplane and large Ethernet topologies. MetaRoCE chooses a different compromise: the network is acknowledged as potentially lossy, and resilience is shifted to transport and NIC. This provides high throughput, low tail latency, and simpler operation as the number of accelerators and distances between them grow.
The key idea of MetaRoCE sounds simple: the fabric sees packets, while the NIC sees intent. Instead of centralizing intelligence in switches, the protocol breaks the network into multiple logical paths with their own telemetry. For each path, RTT, ECN state, and utilization are tracked. This allows the endpoint to make decisions locally without waiting for the fabric to “get everything right” for it.
Practically, this means intentional out-of-order delivery. MetaRoCE sprays packets across many paths, and they arrive out of order. But transport treats this as the norm rather than an exception. Each packet carries a destination, so data is written directly to the final memory location without a reorder buffer and without head-of-line blocking. This is particularly important for Writes. For Sends, a binding to the posted receive buffer is used, allowing the message to be delivered correctly without the round trip needed to determine the recipient.
This approach also changes the behavior of congestion control. Each connection receives its own first-class paths, and the NIC can change the UDP source port to shift traffic to other ECMP branches. In a multiplane fabric, the choice of plane also remains with the NIC. If one path is overheated or broken, only that path suffers, not the entire connection. This is a more precise response than stopping the flow entirely. For AI workloads, where a large portion of traffic goes through collective operations, this is critical.
A separate engineering detail is the handling of loss. MetaRoCE does not require PFC and does not rely on pause frames. Each path has its own ordered sequence and its own 256-bit selective acknowledgment bitvector. If there is a gap in the bitvector, it is interpreted as loss, not reordering. Then retransmission is triggered precisely: only the missing packet is sent, and only on the path where it was lost. This is a measured but important shift. The protocol stops treating loss as an anomaly and begins to manage it as a normal system state.
On top of this, MetaRoCE combines ECN-based sender-driven AIMD congestion control with receiver-driven fair-share rate hints. Windows are maintained both per path and per connection. Therefore, the congestion mark only affects the path that received it, while the receiver returns the share of inbound bandwidth allocated to that sender in each acknowledgment. As a result, senders approach the desired speed directly, without prolonged searching. This is especially useful for incast: according to Meta, the load converges within one or two round trips and provides better fairness with lower tail latency.
There is also an architectural argument in favor of this design. MetaRoCE does not require packet trimming, in-network telemetry, credit-based flow control, or switch-side spraying. The protocol only uses ECN and ECMP, which are already present in most Ethernet fabrics. This reduces dependence on specific network configurations and makes transport applicable on fat-tree, multiplane, deep-buffer, and shallow-buffer fabrics, as well as in vendor clouds where network settings cannot be fully controlled. The trade-off is clear: complexity is shifted to the endpoint, but in return, the system becomes less dependent on fabric behavior.
At the API level, a pragmatic approach has also been chosen. One queue pair now carries both an ordered stream of messages and bandwidth. In traditional RDMA, to increase parallelism, dozens of QPs are often opened on a pair of nodes, each living with its own congestion state. MetaRoCE separates these roles. At the top, there are several ordered streams, while at the bottom, there are many paths under one congestion controller. Existing RDMA Verbs APIs and software stack remain largely unchanged, with extensions needed only for functions like multiplane.
For validation, Meta worked with AMD and implemented MetaRoCE on programmable Pensando NICs. On a 64-node AMD GPU cluster with RCCL collectives, the protocol was compared to RoCEv2 on all-reduce and all-to-all. According to the results, MetaRoCE demonstrated higher throughput and lower flow completion time. With 1% packet loss, it maintained about 86% throughput, and with 10% loss, it continued to deliver useful bandwidth instead of a sharp collapse. On 4-plane and 8-plane topologies, throughput scaled linearly with the number of planes, and during failure traffic simulation, it redistributed without the involvement of the application or operator.
So far, this appears to be an evolutionary improvement of the transport layer for AI infrastructure on commodity Ethernet. Next, Meta separately highlights scale-up, scale-across, and storage/Kv-cache scenarios, but specific results for them are not provided in the source. The only facts are that Meta plans to release the specification, software reference implementation, and compliance framework through OCP, as well as develop support for various NIC architectures. For the industry, this is an important signal: the focus is not only on performance but also on interoperability.