Git fetch in CI turned out to be not a background step, but the main source of load. Datadog restructured the code delivery path through gitretriever, thereby relieving pressure from monorepositories and the old Git backend.
If CI hangs for a long time on “Fetching repository,” the problem is often not with the build. The issue is that git fetch itself becomes an expensive operation, especially on large monorepositories. At Datadog, this manifested at scale: millions of fetch requests per week, thousands of repositories, and spikes in load from CI, deployment, audit, and security services. Against this backdrop, degradation quickly turned into multi-hour CI outages.
Initially, the team tried standard measures. They increased capacity, raised instance sizes, distributed repositories across dedicated backends, and optimized pipelines. This provided a brief effect but did not change the mechanics of the load. The reason lay in the architecture: in a replicated setup, every write had to be duplicated across all replicas, while reads grew with the number of CI jobs. As a result, the system scaled along the wrong axis.
The main conclusion turned out to be technically simple. CDN and caching proxies did not solve the problem because the expensive part of fetch is not a static byte range, but a computation dependent on the specific client request. Moving the clone directly to GitHub also did not help, as it created a thundering herd on the upstream side and hit rate limits. Therefore, Datadog opted not to add another layer but to separate roles within the Git flow itself.
The solution was gitretriever — a Git mirror built as a separate code delivery layer for CI. The architecture relies on independent pods that maintain a fresh local copy of repositories and do not wait for consensus among nodes. In this model, mirrors synchronize with GitHub, while relays serve reads to CI tasks. This separation is a pragmatic compromise: mirrors remain small and behave well towards GitHub, while the relay fleet can grow in CPU and network load without a proportional increase in load on the upstream.
The implementation itself is built around how Git stores and serves data. The article describes blobs, trees, commits, packfiles, delta compression, reachability bitmaps, multi-pack indexes, and pack reuse. This is important because the main cost of fetch occurs on the server when assembling the response packfile. It is there that Git can spend CPU and I/O on decompression, recompression, and indexing. Gitretriever does not eliminate this cost. It changes the location where it occurs and limits the number of recomputations.
The team added several mechanisms that reduce redundant work. Mirrors constantly poll upstream, perform parallel fetch for changed refs, and do not concentrate expensive delta compression in a single request. Relays receive the packfile via a signaling gRPC stream and transmit bytes through a plain HTTP endpoint. In this process, the relay does not reconstruct the packfile but simply establishes the received artifact by content-addressed hash. This is a key point: the work of pulling and indexing is done once on the mirror, while relays reuse the result.
Additionally, Datadog added a pack cache. It allows servicing identical requests without rebuilding the packfile. According to sources, about half of the pack-building fetches are served from the cache. This reduces delta compression on mirrors and relays. Furthermore, the system uses a readiness check that understands the state of Git and background repacking to keep the number of packfiles under control while the pod continues to serve requests.
Interestingly, after migration, the load picture highlighted another layer of the problem. It turned out that many non-CI workloads do not require a full clone. They need one file per commit, the SHA of the branch, a list of changed files, or the merge base. For such requests, the team added a read-only HTTP API. This is an important evolutionary improvement: the API responds in single-digit to tens of milliseconds, while a shallow clone of a large monorepo takes about 75 seconds and keeps a CPU core busy for almost the entire time. Here, the trade-off is clear: instead of a universal but heavy interface, a narrower, much cheaper-to-service path emerged.
The implementation was also done cautiously. The team used feature flags and fallback to the old backend in CI jobs. Migration occurred in groups of repositories, starting with the largest monorepo. This reduced operational risk. After the first cutover, a drop in CPU was immediately observed on the old backend. Later figures confirmed the effect: synchronization reduced from several seconds to several hundred milliseconds, median serve latency hovers around 40 ms, and fetch-serving CPU on the previous Git backend decreased by 3–4 times. Meanwhile, the system withstood a traffic increase of about 20 times over 4 months and reached over 100 million requests per week.
The conclusion here is not that Git became “faster” in an abstract sense. The conclusion is that Datadog removed the concentration of CPU in one place and stopped forcing every new load to replicate the old problem. For CI, SRE, and platform teams, this is perhaps the main lesson: sometimes scaling ends not with adding resources, but with reassembling the data path itself.