The Most Important AI Work Doesn't Show Up in Any Metric

Every company brags about their model's stats. It can do this, it can do that. It's some percent better than the last one. Every model is amazing.

But nobody tells you what you have to set up, and what you have to prompt, to actually hit those numbers.

The hype is the same as it always was

First comes the hype. Every new model, like every new JavaScript framework back in the day, is the best one. Every one is going to replace all developers, generate perfect code, you won't need a team anymore.

And honestly? I think it just added work. It didn't take it away.

I only got this recently. I used to think you fire up Claude Code or Codex, drop a prompt, done, you've got code. And yeah, you've got code. Usually in a terrible state.

A concrete example

I needed a component to render blog posts. The AI kept pushing dangerouslySetInnerHTML, "that's how you do it, it's fine".

I dug in and pointed out: that opens the door to XSS. If the content isn't sanitized, someone can inject any script they want. The model came back with "you're right, sorry about that."

But here's the thing. If I hadn't known it was a risk, I'd have merged it with a clear conscience. The code would run. The bomb would just tick quietly until someone tripped it.

So you start setting things up

After reading way too many articles, I found out you start with an AGENTS.md. Fine. But what do you actually put in it so it helps, instead of adding more problems?

Turns out if you want genuinely solid generated code, that file has to carry the foundations of your project: how components are structured, how you type things, how the project is supposed to look. So a seemingly trivial thing, knowing how to lay out files, which linter to wire in, what to configure, actually matters a lot. That knowledge didn't go anywhere. It's still required.

One file is never enough

Then comes the next problem. If you want code that's properly styled and structured, you can't cram everything into one AGENTS.md. Overload it and the model starts making things up, hallucinating, and you get stuff that bites you later.

So you split it: separate file, separate agent. You add value gates to tell the model what actually matters versus what's noise. And those instructions have to be well formatted and deliberately chosen, because they load into the model's context and eat up a chunk of your tokens right off the bat.

The prompt itself is work

The prompt has to be right too. "Generate a component that does X" isn't enough, because the model glances at the instructions once and ignores them the next time.

You have to tell it explicitly: read the instructions in this file, then spin up an agent to verify what you just did. Better yet, go into planning mode and run a few iterations with it. That's when you get the best result.

A word on the shortcuts

Sure, there are ready-made skills you can install as a plugin. But that starts to look like another layer, this time wired straight into an agent that has access to your repo, your terminal, and sometimes your secrets.

And after everything we've seen lately, attacks on npm, PyPI, IDE extensions, I trust installing random helpers less and less without checking what they actually do. Spending the time to write your own skills is, in my opinion, simply safer.

The part nobody measures

And that's the point. All this work around setting up the model is the real challenge, and it's the one nobody talks about.

The model's stats are real. They show what it can do. But to actually get there, to reach the quality those benchmarks promise, you have to put in hours of work that don't show up in any chart, any velocity metric, any "X% better than the last model."

The benchmark is the ceiling. Getting anywhere near it is the invisible work.