The LLM serving platform based on vLLM and Triton addresses the challenge of unifying inference and reduces the gap between experimentation and production.
The system encountered the classic problem of scaling ML in production. Most teams use hosted APIs, but as load and latency and control requirements grow, this becomes a bottleneck. In this case, inference is embedded directly into the existing JVM-based serving system. It already handles routing, A/B testing, feature fetching, and post-processing. Adding LLM to this framework created pressure on the architecture: different models, varying resource requirements, and the need to maintain a unified API without “special cases.”
The solution was pragmatic: unify all inference through a common layer and separate execution by model type. Lightweight CPU models are run in-process to avoid network latency. Heavy models are offloaded to a GPU-backed Model Scoring Service. Triton Inference Server is used as the backend, responsible for batching and GPU scheduling. Above it is a Java control plane managing deployments, versioning, and autoscaling. A key decision was to switch to vLLM as the primary inference engine. The reason is not only performance but also alignment with the actual workload: embedding, prefill-only inference, autoregressive decoding, and custom generation constraints.
At the implementation level, the architecture is built around a single API. All models, including LLM, are accessible via a gRPC interface. Simultaneously, an OpenAI-compatible API was added for integration with the ecosystem. This reduces migration costs: transitioning from hosted models to self-hosted requires almost no code changes. However, the integration revealed a hidden issue. The response_format parameter declared in the API was lost at the vLLM level. As a result, the system could return incorrect JSON without errors. This was resolved with a patch to the frontend layer, passing parameters into guided decoding. This case illustrates a typical risk: API compatibility does not guarantee semantic correctness.
An additional layer of complexity is packaging and rollout. Triton supports various ways to package models, and the choice affects system cohesion. A strong coupling between the model and frontend complicates updates. A zero-downtime deployment strategy is used, considering that GPU instances take longer to spin up. It is recommended to make models version-agnostic to utilize cheaper rollout strategies. The versioned approach remains only for breaking changes. This is a trade-off between flexibility and operational stability.
In production, issues emerged that were not visible during the design phase. The first is observability. vLLM and Triton publish metrics separately, and some key signals are lost. For example, token throughput and KV cache utilization. The solution is a unifying proxy that aggregates metrics into a single endpoint. This allows existing dashboards to remain unchanged. The second issue is constrained decoding. Moving business logic inside the decode loop reduces costs on repeated requests but creates CPU load.
The initial implementation of constrained decoding in Python hit a wall with the GIL. The logic was executed sequentially for each request, and latency increased linearly with batch size. This is a typical example where the GPU effectively batches computations, but the CPU becomes the bottleneck. The solution became possible only after transitioning to vLLM V1. Logits processing was moved to the batch level. Critical code was rewritten in C++ with multi-threading. This eliminated the dependency of latency on batch size and stabilized throughput.
As a result, the system became more predictable under load, although exact metrics are not disclosed. The main effect is the alignment of the path from experimentation to production and a reduction in architectural exceptions. However, the cost is an increase in the complexity of the control plane and tighter coupling with the Triton and vLLM infrastructure. This is a conscious trade-off: less flexibility at the component level, more at the platform level.
This approach reflects a broader industry trend. Companies are moving from using external LLM APIs to self-hosted inference to control latency, cost, and data. However, key challenges lie not in the model but in the details: API compatibility, metrics, rollout, and behavior under real load.