OpenAI demonstrated the architecture of GPT-Live as an example of how to build continuous stateful voice interaction without losing responsiveness. The key question here is not about the model, but about how to separate the latency-critical media path from everything that can wait.
The architectural challenge in GPT-Live begins where the voice session must remain continuous, while variable latency operations arise around it. If media processing, delegation, tool use, persistence, and other application logic are consolidated into one critical path, the system quickly runs into latency issues. For real-time AI, this means a simple principle: “voice must flow,” and everything else must conform to this requirement.
OpenAI chose an asynchronous boundary between the live path and the application part. Inside the live path, the media pipeline and inference loop remained. Everything that can tolerate latency was moved beyond the asynchronous RPC boundary. This is a pragmatic trade-off: less coupling in the critical loop, but more engineering discipline around state exchange and behavior consistency.
An additional layer of complexity is stateful inference. GPT-Live uses dedicated, stateful inference for each session, but the session context can migrate to another model instance if the current instance is draining or the conversation approaches the context limit. This approach helps combine state retention with operational flexibility. The system can reserve capacity on a designated instance while redistributing new and already active sessions across available capacity.
This is not a free solution. The more state is tied to live interaction, the more careful one must be regarding availability, elasticity, and operational complexity. However, in this case, the compromise seems reasonable: the dialogue model remains continuous, while the infrastructure gains the ability to scale and free up resources without breaking the session.
For the transport layer, OpenAI retained WebRTC. This is also an engineering choice, not merely a matter of inertia. WebRTC already provides a battle-tested low-latency media stack and built-in error recovery. Alternatives like RTP over QUIC look promising, but in their current form, they only cover the transport layer, not the complete media pipeline. Additionally, at the transport level, several capabilities are still lacking, including GCC congestion control and RTT-aware path selection.
Instead of replacing the stack, OpenAI opted for its simplification. The company added WARP improvements, including SPED, DTLS 1.3, and SNAP, as well as Instant Connect to reduce startup latency. An important advantage of this approach is that each improvement can be rolled out independently and its effect tested in isolation. This reduces risk and simplifies operational validation. The additional effect for the ecosystem is also noticeable: existing WebRTC applications receive enhancements without code changes.
The most illustrative moment is the silent test. Before launch, OpenAI mirrored real Voice sessions in GPT-Live, but the output was simply discarded. The system operated in an effectively read-only mode and without user credentials. This allowed live traffic to be run through the media loop and inference service without impacting customer experience. An important detail here is that synthetic tests would not have provided the same picture. The real traffic showed geographical diversity, load behavior, and failures that were not reproduced with canned speech.
It was the silent test that revealed degradation under load, which had to be addressed with targeted optimizations and bug fixes. One example is that in some regions, GPUs were not colocated with the CPUs that fed them data, creating unexpected latency. Such a defect is difficult to see in laboratory load testing if the test profile is too uniform. Production traffic proved to be more honest because it brought real variability and load distribution.
In summary, GPT-Live appears as an evolutionary improvement of the architecture for real-time AI. OpenAI did not attempt to replace the entire stack. It separated the critical path from the auxiliary logic, retained the familiar media foundation, and separately strengthened the weak points. For architects, the important aspect is not the set of technologies itself, but the way of thinking: first, maintain responsiveness, then carefully lay out state, transport, and observability around the live path.