Enterprise AI rarely breaks down at the model level. More often, the issue lies in the absence of a complete architecture around it: compute, storage, serving, and integration. This AI architecture determines whether the model becomes part of production or remains a pilot.
Enterprise teams often start with the question of model selection, but they encounter not a shortlist, but operational reality. If there is no clear scheme for how the model is stored, where it is executed, how it is served through an endpoint, and how it connects with existing systems, the project remains a local demonstration. This is the main challenge of enterprise AI: the model alone does not provide a ready-made capability.
Essentially, this is not about “plugging in AI,” but about assembling an infrastructure stack. The initial architecture identifies 4 layers: compute, model storage, inference and serving, integration. This is a pragmatic way to distribute responsibility across levels. It helps to see trade-offs: a managed API alleviates some operational burden, but self-hosting provides more control over the estate and data.
The first layer is compute and hardware. For LLM and most generative workloads, GPU or comparable accelerators are almost always required here because matrix operations do not scale well on CPU alone. In self-hosted inference, this typically involves GPU servers in a data center or GPU-backed instances in the cloud, often through a Kubernetes platform, such as Red Hat OpenShift with GPU operators. The size of this layer depends on the model and concurrency. If capacity is insufficient, the system degrades through elevated latency, throttling, or queued requests.
The second layer is model storage. In practice, this is not just “model files,” but a package of artifacts: weight files, tokenizer vocabulary, and configuration for architecture and serving settings. The larger the parameter count and the higher the numeric precision, the greater the disk footprint and the longer the startup. Quantized builds reduce both metrics, but this is already a trade-off between size and execution mode. For self-hosted deployments, weights typically reside in durable blob or object storage, such as S3, MinIO, or OCI images.
The third layer is the inference engine and model serving. Here, it is easy to confuse two different mechanisms, but the separation is important. The engine loads weights into accelerator memory, tokenizes input, batches requests, streams tokens, and controls concurrency limits. Serving is responsible for routing, health checks, replicas, and autoscaling, aligned with GPU capacity. The text mentions open engines vLLM, Text Generation Inference, and Ollama, and notes that Red Hat AI Inference uses vLLM at its core.
The fourth layer is integration. This is where AI ceases to be a separate system and is embedded into the enterprise architecture. Integration occurs in two directions. In the first, your application calls the served model endpoint via an OpenAI-compatible HTTP API. In the second, the model or agent framework accesses databases, internal APIs, ticketing systems, and document stores. These directions differ in risk: in the first case, the model is passive; in the second, it initiates actions, so input validation, authentication, error handling, and guardrails are needed.
For the first scenario, the source emphasizes the importance of timeouts, retries, cost controls, and audit logging. When many internal systems share a single endpoint, an AI-aware gateway becomes useful. It can enforce authentication, quotas, token-aware rate limits, and observability tailored to inference traffic. Red Hat Connectivity Link serves as such a gateway for Kubernetes and Red Hat OpenShift. This is an evolutionary improvement, not a replacement for the serving layer.
For the second scenario, the key issue becomes tool exposure. Without a common protocol, each agent-to-system connection turns into bespoke integration. Here, the Model Context Protocol, or MCP, is used. The idea is simple: capabilities are published as MCP servers, and agents discover and invoke them through a shared protocol. At the enterprise level, a governed front door with federated discovery, authentication, identity-based tool filtering, and audit is still needed. The text notes that Red Hat Connectivity Link adds an MCP gateway in technology preview on Red Hat OpenShift.
The practical takeaway from this architecture is that the hosting choice does not change the structure itself, but only who operates layers 1–3. A managed API hands off vendor compute, storage, and serving for the model. Self-hosting retains these layers on your side. Integration remains your responsibility in both cases. Therefore, the question of production AI in the enterprise is a question of the operational model, not just the quality of the model.
If we distill the analysis into a single checklist, it would be as follows: compute executes the model, storage holds and versions artifacts, engine and serving stabilize the endpoint, integration connects AI with business systems. This sequence is what distinguishes a pilot from production. And this is why enterprise AI often requires architectural discipline more than yet another choice of the “best” model.