AGENTS.md vs. skills: How to steer a coding agent

Every team adopting coding agents hits the same question early: where do you put the instructions that tell the agent how your codebase actually works? Two answers dominate the conversation right now. One is AGENTS.md, a plain markdown file at the root of your repo. The other is skills, packaged instruction sets an agent loads on demand.

Most of the debate treats this as a formatting decision. It isn’t. The format you pick is downstream of a harder question: does steering the agent change what it does, and how would you know? If you can’t answer that with something more reliable than a good feeling, you’re picking a file extension, not a strategy.

AGENTS.md vs. skills: the short answer

AGENTS.md is a single markdown file, conventionally at the repository root, that a coding agent reads at the start of a session. It’s an open format backed by OpenAI, Google, and others, and OpenAI’s Codex documentation treats it as the place to put the custom instructions the agent reads before doing any work. It holds always-on context: how to run the build, which commands to prefer, project conventions, and the things a new contributor would need to be told on day one. It loads every time, so it’s simple and predictable, and every capable agent can read it.

Skills, the approach Anthropic introduced for Claude, are modular. Each skill is a self-contained bundle of instructions (and sometimes scripts or references) that the agent pulls in only when a task calls for it. In Anthropic’s design, the agent sees only each skill’s name and description up front and loads the full instructions on demand when a task matches. Instead of one file that grows without bound, you get a library the agent selects from based on the work in front of it.

The short answer to which one to use:

• Reach for AGENTS.md when the guidance is small, stable, and applies to nearly every task: build and test commands, repo layout, house style. This is the floor. Almost every project benefits from one, and it takes minutes to write.

• Reach for skills when guidance is large, situational, or procedural: a multi-step release checklist, a framework-specific migration, a review rubric that only matters on certain files. Loading all of that on every task wastes context and dilutes the instructions that do apply.

In practice most teams end up with both. A lean AGENTS.md for the always-true facts, and skills for the specialized procedures. The choice between them comes down to scope and load timing.

But notice what that framing quietly assumes: that writing the instruction down is the same as the agent following it. It isn’t.

What the evals show about AGENTS.md vs. skills

When teams actually measure agent behavior instead of eyeballing it, a few patterns show up repeatedly.

The first is that instructions have a compliance curve. An agent that follows a rule every time in a test will follow it far less often once its context window fills up. A rule that reads as unambiguous to you can lose out to a more recent or more specific instruction the model weighs more heavily. “It’s in the AGENTS.md” is no guarantee of agent behavior.

The second is that placement and phrasing move the numbers. The same rule stated as a concrete, testable directive (“run yarn lint before proposing a commit”) works more reliably than a vague aspiration (“keep the code clean”). Where the rule lives matters too, but not always in the direction you would guess. On-demand loading, the whole premise of skills, can surface the relevant instruction at the moment it’s needed. It also adds a step the agent can skip, namely deciding to load the skill at all. When Vercel ran a head-to-head eval, an always-on AGENTS.md reached a 100 percent pass rate on their task set while the skills version topped out at 79 percent, partly because the agent sometimes chose not to load the skill. The discussion that followed, however, raised an important point: that result is specific to Vercel’s tasks, their model, and their setup. Result may vary in other scenarios, which is exactly why you measure your own rather than inheriting someone else’s conclusion.

The third is that more instruction does not produce linearly better results. Past a certain point, adding rules to a single file degrades adherence to the rules already there, because the agent has a finite budget of attention to spend. Teams that treat their AGENTS.md as an append-only dumping ground tend to watch overall compliance drift down as the file grows, even for rules that used to work.

None of these are things you can settle by argument. They’re empirical, they vary by model and by codebase, and they produce different results every time you upgrade the underlying model. The only way to know how your configuration behaves is to test it.

How to test agent config with a reproducible eval loop

An eval loop for agent configuration is simpler than it sounds. You don’t need a research harness. You need a fixed set of tasks, a way to run the agent against them, and a check that tells you whether the behavior you wanted actually happened.

A workable loop has four parts:

1. Fixtures. A small set of representative tasks, pinned to a known starting state: a checked-out commit, a specific prompt, a defined repo layout. The point is that every run starts from the same place, so a change in output reflects a change in your config, not a change in the input.
2. A behavior to assert. Pick something observable. Did the agent run the test command before committing? Did it put the new file in the directory your conventions specify? Did it avoid the deprecated API you told it to avoid? Vague goals (“write good code”) don’t work here. Testable ones do.
3. A runner. A script that feeds each fixture to the agent and captures what it did: the files it changed, the commands it ran, the final diff.
4. A checker. Code that inspects the captured output and returns pass or fail: a grep for the forbidden API, a check that the expected file exists, an assertion that the lint step ran. This is an ordinary test assertion pointed at agent behavior instead of application behavior.

With those in place, the workflow is familiar to anyone who writes tests. Change the config (move a rule from AGENTS.md into a skill, rephrase a directive, split an overloaded file), run the fixtures, and compare pass rates before and after. Because agent output has some run-to-run variance, run each fixture several times and track the pass rate rather than a single pass or fail.

Here is an example checker, using a lint-before-commit rule as the example:

def check_ran_lint_before_commit(run):
    commands = run["commands"]  # ordered list of shell commands the agent ran
    try:
        lint_index = next(i for i, c in enumerate(commands) if "lint" in c)
        commit_index = next(i for i, c in enumerate(commands) if c.startswith("git commit"))
    except StopIteration:
        return False  # one of the two commands never ran
    return lint_index < commit_index  # lint must come before the commit

def pass_rate(runs):
    passed = sum(1 for r in runs if check_ran_lint_before_commit(r))
    return passed / len(runs)

The assertion is trivial. The value is in running it consistently, across config changes and across model upgrades, so that “does our agent guidance still work” becomes a measurable number instead of an opinion.

Steering agents in a delivery pipeline

The reason to make agent behavior measurable is the same reason you make anything measurable: so you can gate on it. Once the check is code, it belongs where the rest of your checks already live, in your continuous integration pipeline.

There are two distinct things worth running there.

The first is treating your agent config as a tested artifact. Your AGENTS.md and your skills are inputs that shape production work, and they change over time. When someone edits them, or when you bump to a new model version, run the eval suite in CI and compare pass rates against the baseline. A pull request that drops a key rule from full adherence to 40 percent should fail the same way a pull request that breaks a unit test fails. This catches the quiet regressions that are otherwise invisible until an agent ships something wrong weeks later.

The second is validating the work the agent produces. Steering reduces the odds of a bad change, but it doesn’t eliminate them. Whatever an agent generates still runs the same checks every human change runs: build, lint, unit tests, integration tests. This is the work CircleCI is built for: it runs your test suite on every commit, so an agent is only ever as trusted as its most recent test run. A pipeline that returns clear pass or fail signals in minutes makes it safe for an agent iterate quickly, because every iteration is verified before it reaches customers.

You can also move some of that validation earlier. With Chunk, the same kinds of checks run as a pre-commit hook in the agent’s inner loop, so basic failures get caught before they ever leave your machine and the pipeline only sees code that’s ready for prime time.

Now the AGENTS.md-versus-skills question resolves itself. You write instructions, measure their effect with reproducible evals, gate config and agent output in CI, and feed the results back to the agent. With that loop running, you can see which format gets better adherence for a given rule instead of guessing.

The format is downstream of the feedback loop

The basic takeaway is to pick AGENTS.md for the small, always-true facts and skills for the large, situational procedures. But when it comes to steering your coding agent, that’s one of the least important decisions you’ll make.

What’s more important is whether you can tell, with evidence, that your steering works. Agent output that isn’t validated is a liability moving at machine speed. Build the feedback loop first, run it in your pipeline, and the format question becomes a decision you can make with data.

To build the most effective feedback loops, your agent needs direct access to your test results. That’s why we rebuilt the CircleCI CLI from scratch to be agent-friendly, with predictable JSON on every data-returning command, stable exit codes, and a built-in MCP server so an agent can trigger runs, inspect jobs, and read test results without guessing. Point your coding agent at the CircleCI CLI and let it close the loop itself: run the evals, read the pass rates, and gate its own changes on every commit before they reach production.