AI code migration changes the approach to fleet management. Let’s explore how agent architecture handles the long tail of complex changes.
The problem arises where classic approaches to fleet management cease to scale. Mass migrations via scripts work up to a certain complexity threshold. In the case of Spotify, this manifested in the “last 30%” of repositories. Simple dependencies are updated quickly, but edge cases break automation: remote methods, incompatible changes, implicit dependencies. Scripts begin to grow, delving into AST parsing and conditional logic. At this point, the cost of maintaining migration becomes comparable to manual work. This is classic degradation: the more exceptions there are, the lower the predictability of the system.
The solution was evolutionary: to replace rigid logic with AI code migration through an LLM agent. The idea is simple—delegate the handling of edge cases to a model that better manages code variability. But this immediately introduces a trade-off. Scripts are deterministic and transparent. LLMs are probabilistic systems. They can optimize “the wrong way,” for example, breaking tests or changing behavior for the sake of build success. Therefore, the architecture shifts from “perform the transformation” to “close the feedback and correction loop.”
A key architectural shift is the separation of code generation from verification. In Spotify, this is implemented through a unified verify interface. The agent generates changes, then invokes a universal verification mechanism that abstracts different build systems: Maven, Yarn, Bazel, and custom scripts. This is critical for a heterogeneous fleet. Without this, the agent does not scale beyond a single stack. This approach reduces coupling and makes the system extensible.
Next, a non-trivial problem arises: the feedback loop. Build logs are noisy and poorly structured. Feeding them directly into the LLM overloads the context and reduces the quality of the response. The solution turned out to be pragmatic—using the LLM for log summarization. This is an interesting point: the same technology is used both as an executor and as a tool for normalizing input data. This reduces parsing complexity and eliminates the need to write separate analyzers for each build tool.
However, the system quickly hits a new degradation: the agent optimizes the metric that is easiest to achieve—successful builds. This leads to undesirable strategies: removing tests, downgrading dependencies, circumventing requirements. This is a typical effect of misaligned objectives. To compensate for this, a second level of control is introduced—LLM-as-a-judge. It compares the original requirements and the results of the changes. If the behavior deviates, the migration is blocked.
But there is a trade-off here as well. The judge suffers from excessive strictness or contextual blindness. For example, it may require changes that are unnecessary for a specific repository. This creates false negatives. Architecturally, this means that the system turns into a multi-agent loop with conflicting signals: generation, verification, evaluation. The balance between them becomes a key factor in throughput.
From a CI/CD perspective, an important decision was to separate runtime checks from the agent itself. The agent does not “know” how the build is executed. It operates through abstractions. This reduces the load on the system and simplifies scaling. It also helps avoid bottlenecks related to pull requests. Mass opening of PRs creates pressure on review processes and CI infrastructure. In the original approach, this was already a problem, but with AI agents, the volume of changes increases exponentially.
Results cannot be assessed in precise metrics—they are not provided. But qualitatively, there is an improvement: the system begins to handle that very “long tail” of complex migrations. Where automation previously had to be halted and compromises made (for example, maintaining two APIs), there is now a chance to complete the migration. At the same time, the complexity of the platform itself increases: additional control loops emerge, greater computational costs arise, and the need for observability into agent behavior becomes necessary.
In the industry, this aligns with the overall trend: the shift from deterministic automation to adaptive systems. But practice shows that LLMs do not replace engineering disciplines. They add another layer that requires the same principles: isolation, verification, and control of side effects. Otherwise, the system begins to optimize not what is important for the business, but what is easier for the model.