Agent optimization in Microsoft Foundry begins not with reducing token cost, but with the price of a successful outcome. For an agentic system, this is more important because one result often requires multiple model requests.
The main issue here is not the model itself, but that a prototype can easily become the production default. In a prototype, the strongest available model is usually chosen, all possible context is included in the prompt, and it is checked whether the idea works. This makes sense during the validation phase, but does not scale economically because one agent loop can involve a dozen model requests.
Next, the system starts paying for unnecessary complexity twice. First, different AI workloads have different natures: classification, extraction, formatting, summarization, and multi-step reasoning do not require the same power. Second, each error in the loop creates new turns, repeated calls to tools, and unnecessary tokens. Therefore, the cost of a successful outcome is determined not only by the price of a single request but also by the number of turns that were avoided.
This leads to the solution: runtime optimization should distribute the load across tasks rather than sending all requests to a single frontier model. Microsoft Foundry provides four levers for this: model routing, choice of deployment strategy, prompt caching, and tuning through an optimizer. This is a pragmatic set because each lever can be applied separately, measured against a quality bar, and rolled back if the trade-off does not hold.
The first lever is the model router in Foundry Models. It evaluates incoming requests and redirects them to the appropriate underlying model through a single endpoint and single deployment. Routing modes allow shifting the balance toward cost, quality, or a compromise between them. Model subsets, which now align with Azure Policy, limit the choice of approved allow-list where compliance boundaries are needed, and built-in failover adds resilience if one model is unavailable.
The second lever is the deployment choice. Standard deployments provide flexibility and pay-as-you-go pricing. Priority processing is suitable for interactive applications where more consistent latency is important. Provisioned Throughput Units, or PTUs, assist with predictable demand and sustained throughput. For large asynchronous workloads, such as document processing, classification, and evaluation runs, Foundry offers Batch deployments, which can provide up to 50% lower costs if immediate response is not required.
Here, architectural compromise is important. The same product can contain different traffic patterns, and each has its own economic logic. Developer-facing tools can operate on Standard. Chat experiences often require Priority. Agentic applications with constant load benefit from PTUs. Background jobs like document analysis or knowledge extraction can be moved to Batch without affecting the end-user experience.
The third lever is fine-tuning and caching. Fine-tuning is useful when behavior is stable and the volume is high enough to justify the effort. In this case, a smaller model can approach a large model on a specific task because it learns the task, tone, or format. This reduces the rate and shortens prompts, but it is not a universal answer: if the task changes frequently, the economic rationale is thinner.
Agents are particularly cache effective because system instructions, tool schemas, and policy text are repeated on each turn. Prompt caching allows reuse of previously processed prefixes instead of reprocessing. It is important to maintain the structure of the prompt: stable content should come first, while volatile content should be lower. If a timestamp, user name, or another mutable fragment is placed at the top, the cache hit will not work because the exact match at the start of the prompt will be disrupted.
Caching also works at the infrastructure level. If there is a gateway in front of Foundry inference APIs, it makes sense to choose a semantic-cache-aware gateway, such as AI Gateway in Azure API Management. It helps maintain session affinity to the same endpoints and better utilize cache across sessions and users. Deterministic tool results can also be kept in a dedicated store with a TTL that corresponds to the frequency of data changes.
The fourth lever is instruction optimization and observability. The prompt optimizer rewrites system instructions according to best practices in prompt engineering and shows reasoning for each change. The agent optimizer in Foundry Agent Service goes further: it runs the agent through a dataset of real tasks, generates candidate configurations, scores them, and ranks them. It can change instructions, skills, tool descriptions, and model selection, and the dataset can be built on its own agent traces.
Without observability, no optimization works here. Foundry observability provides per-request signals: input and output tokens, cache hit rate, latency, actual served model, and evaluation scores. In this scheme, two metrics are particularly important: cost per request and cost per completed outcome. The first shows local efficiency. The second shows how much the business actually paid for a completed task, taking into account turns and retries.
The practical takeaway is simple. First, measure cost, latency, and task success together. Then maintain a standing evaluation set that any optimization must pass before release. After that, link traces, evaluation sets, budgets, alerts, and cost tagging in Azure so that regression is visible immediately, not at the end of the month. This is the engineering logic of the article: savings should not compromise quality, safety, and latency, and the right system should improve all three parameters simultaneously.