From Code to Cache: How Ant Group's AI Delivery Pipeline Relies on Smart Load Balancing

The Shift from AI Coding to AI Delivery

Ant Group's teams now see AI generate more than 90% of their code. That number is huge, but it's no longer the bottleneck. The real challenge has moved downstream, to what they call AI Delivery—the process of getting all that machine-written code tested, built, and shipped reliably. And at the heart of that process? Caching and load balancing.

When you're running thousands of AI agents in parallel, each one spinning up environments, running builds, and hammering APIs, your infrastructure needs to be smart about what it reuses and how it spreads the load. Otherwise, you're not delivering faster—you're just burning compute and waiting on queues.

Why Traditional Caching and Load Balancing Fall Short

Ant's legacy DevOps platform was built for human-paced workflows. It worked fine when a developer triggered a build every few minutes. But AI agents don't drink coffee or take breaks. They fire off requests continuously, 24/7. Traditional caching strategies—like simple key-value stores for build artifacts—assume a relatively steady stream of requests. AI agents are anything but steady.

The load balancing problem is even thornier. With human developers, you can predict peak hours. With AI agents, you get sudden bursts of activity as multiple agents decide to run tests at the same time. You need a system that can scale out to handle thousands of concurrent executions, then scale back down just as quickly.

Inside Ant's New Architecture: ADS

Ant's answer is a new architecture they call ADS (Agentic Delivery System). It's built on a few key principles that directly affect how they handle caching and load distribution.

First, they treat the agent as a first-class citizen. That means the entire delivery pipeline is designed around agents, not humans. Second, they follow a 'weak platform' philosophy—decentralized, CLI-first, code-first. This isn't just about flexibility; it's about making sure load balancing doesn't become a single point of failure.

CLI-First: A Standardized Interface for Agents

Their ACLI initiative standardizes the CLI across all tools. Why does that matter for load balancing? Because you can't route traffic intelligently if every tool speaks a different protocol. By standardizing the CLI, they can build a uniform queueing system that distributes agent requests to the right services without custom adapters.

It also simplifies caching. With a standardized command structure, you can cache results at the command level. If an agent runs the same CLI command twice, you can serve the cached result instead of re-executing.

Poolab: The Cloud-Native Agent Environment

Poolab is their cloud-based environment for running agents. It's designed to handle ten-thousand-plus concurrent agent executions. That scale doesn't happen by accident. It requires aggressive caching of environment snapshots and intelligent load balancing across machines.

When an agent needs an environment, Poolab checks if a cached snapshot exists. If yes, it spins up a new container from that snapshot in milliseconds. If no, it builds one from scratch—but that's the slow path. The goal is to hit the cache as often as possible.

On the load balancing side, Poolab uses a scheduler that considers machine health, current load, and even network latency to decide where to place each new execution. It's not just round-robin; it's adaptive.

ADE: Keeping Test Environments Fresh and Routed

One of the trickiest parts of AI delivery is testing. Ant's ADE (Agent Delivery Environment) system handles evaluation and staging environments for agents. The acronym hides a lot of complexity: ADE makes environments 'creatable, verifiable, fresh, and routable.'

Caching plays a huge role here. Environments are cached as golden images. When an agent requests a test environment, ADE checks whether a cached image matches the required configuration. If it does, the environment is provisioned in seconds. If not, it builds a new one—but that's the exception, not the rule.

Load balancing is equally critical. ADE routes test traffic to the right environment instances, avoiding hot spots and preventing 'eval traffic from falling off.' Before ADE, teams struggled with environment drift—where the environment an agent tested against didn't match production. That's a caching problem in disguise: stale caches cause drift.

Agentic CI/CD: The Final Frontier

Ant's Agentic CI/CD pipeline is AI-driven end-to-end. It handles builds, deployments, traffic shifting, and quality gates. The user just gives an intent—like 'fix this bug'—and the system figures out the rest.

For load balancing, this means the pipeline itself can shift traffic smartly. If one service is overloaded, the pipeline can route around it. If a build cache is stale, it knows to invalidate it and rebuild. The system also monitors its own health and can automatically roll back a deployment if something goes wrong.

Caching here is about more than speed; it's about consistency. The pipeline caches build artifacts, test results, and even deployment configurations. But it also knows when to bypass the cache—like when a security patch changes the base image.

Real-World Impact: The A-Fu Harness

The first end-to-end test of this system was in a scenario called A-Fu BadCase Repair. A-Fu is an AI assistant that fixes bugs from production. The harness covers the entire flow: root cause analysis, solution design, code generation, evaluation, and release.

They've already fixed over 300 bad cases using this pipeline. The time-to-fix dropped from an average of 12.6 days to 3 days—a 3x improvement. AI handles more than 95% of the process, with humans only making key decisions.

That speed wouldn't be possible without smart caching. Each stage—from analysis to evaluation—reuses cached data where possible. The load balancing ensures that when 100 agents are running simultaneously, no single server becomes a bottleneck.

Lessons for Your Own Infrastructure

Ant's journey offers a few takeaways for anyone building AI-assisted delivery systems.

• Standardize your interfaces. If your agents can't talk to your tools in a consistent way, you can't route or cache effectively. A unified CLI is a solid first step.
• Cache everything you can, but know when to invalidate. Cached environments and build artifacts save time, but stale caches cause drift. Build in freshness checks.
• Design for bursts. AI agents are not human. They don't follow a 9-to-5 pattern. Your load balancer needs to handle sudden spikes without breaking a sweat.
• Monitor and adapt. The best caching and load balancing strategies are dynamic. They learn from traffic patterns and adjust accordingly.

Ant Group's experience shows that as AI coding becomes mainstream, the differentiator won't be how much code you can generate. It'll be how efficiently you can ship it. And that efficiency depends on the boring stuff—caching, load balancing, and infrastructure that actually works.