How I scale my AI code review system — and why every agent thinks like me
Adding a new agent to my system takes 2 days. Not because of the architecture — but because teaching it to think like me is the hard part.
When I built my 9-agent code review system, I didn't just build a tool. I built a system designed to grow.
Every new problem my team encounters is a potential new agent. The architecture — fan-out/fan-in, parallel execution, resilient fallbacks — was designed from day one to absorb new agents without breaking anything else.
But adding an agent isn't just wiring it into the pipeline. The hard part is something else entirely.
The architecture scales easily. The thinking doesn't.
Plugging a new agent into the system is straightforward. Define its domain, connect it to the Orchestrator, set its output format. Done.
What takes time is calibration — teaching the agent not just what to look for, but how to think about what it finds.
My latest addition is a Storybook conventions agent. Its job: verify that
developers use the right components from our design system instead of raw HTML
tags. <Link> instead of <a>. <Button> instead of <button>. It also
checks whether the implementation matches what's actually defined in Storybook.
Simple domain. But getting it right took 2 days.
Why 2 days for a "simple" agent?
The domain was clear from the start. The hard work was in three places:
Value gates and severity — what's a blocker and what's a suggestion? Using a
raw <a> tag instead of <Link> in a new component is different from finding
it in a legacy file nobody touched. The agent needs to understand that
difference.
Source of truth — where does the agent look to know which components exist? How does it compare what's in Storybook against what's in the diff? Getting this right required careful thought about context and how the agent accesses it.
Reading the output — every calibration cycle, I read everything the agent produces. Not to rubber-stamp it. To understand where it reasons correctly and where it drifts.
That last part is non-negotiable. You can't calibrate what you don't read.
Every agent is calibrated individually
I don't have a template I copy between agents. Each one gets its own calibration process because each domain has its own failure modes.
The TypeScript agent fails differently than the Security agent. The Storybook agent has completely different edge cases than the React agent. Treating them the same would mean accepting mediocre results across the board.
But there's a deeper reason why the system reviews code the way I do. The instructions are written so that no code is treated as an afterthought. Every variant gets the same scrutiny. A simple utility function goes through the same algorithmic complexity check as a core business component. Nothing gets a free pass.
That's a deliberate choice. The moment an agent starts skipping edge cases because something "looks simple," the whole system loses credibility.
This is also why the system reflects how I review code — not how a generic AI reviews code. I spent a week on the original 9 agents, iterating on ~2,900 lines of instructions until the output matched my own standards.
The Storybook agent went through the same process. Shorter — because I've learned what to look for — but no less individual.
What scaling actually means
Scaling this system doesn't mean adding agents quickly. It means adding agents that maintain the same standard as the ones before them.
One new problem in the codebase. Two days of calibration. One agent that thinks like me, running in parallel with the other eight.
That's the system working as designed.
Full architecture breakdown: How I built an AI code review system with 9 agents
Was this helpful?