Write AGENTS.md instructions you can actually test

Start with a failure you have actually seen

Imagine an agent repeatedly editing generated API types instead of the source schema. A general reminder to follow best practices does not tell it which file owns the behavior. A useful instruction identifies the source of truth, the generated output, and the command that connects them.

Write down the observed mistake before writing the rule. That gives you a reason for the instruction and a candidate task for testing it. If you cannot name a mistake, a repository convention, or a decision the instruction clarifies, consider whether the file needs it yet.

Illustrative rule — adapt paths and commands to your repository:

For API shape changes, edit api/schema.yaml.
Regenerate src/generated/api.ts with npm run generate:api.
Do not manually edit generated API types.
Run npm run test:api before handing off the change.

Give each rule an observable outcome

The example above creates several checks: was the schema changed, was the generated file produced by the generator, and did the API tests pass? Those checks are more useful than asking a reviewer whether the agent seemed to understand the architecture.

Separate rule adherence from task success. An agent can follow the command sequence and still implement the wrong behavior. Conversely, an agent might solve a task without needing a rule that does not apply. Your evaluation should record both the functional result and the applicable instructions followed.

Avoid rules that require subjective grading when a concrete boundary would do. Instead of asking for clean code, identify an existing module to extend, a public interface that must remain compatible, or a test that captures the required behavior.

Keep the scope visible

The AGENTS.md format supports repository guidance and nested files for subprojects. Put instructions near the work they govern, and check your particular agent’s loading and precedence behavior before relying on that arrangement.

In a repository with a frontend and a data pipeline, the frontend’s component checks should not silently become a requirement for a data-only change. State when a rule applies. In a shared root file, use wording such as ‘When changing the public API…’ so the condition is part of the instruction.

Also look for duplicated guidance. If two files describe the same workflow differently, decide which file owns the guidance and link to it where appropriate. A shorter file is not automatically better, but contradictory copies are difficult to maintain and evaluate.

Test the proposed edit against the old version

To evaluate an instruction change, keep a copy of the previous file and the proposed file. Run the same reviewed task from the same starting commit with each version. Keep the agent, model, permissions, task prompt, and budget fixed, and start a fresh session for every attempt.

For the schema example, choose a task that really needs an API shape change and a second task where that rule should not apply. The first tests whether the instruction helps; the second checks whether it causes unnecessary work. Repeat the comparison before treating one result as a pattern.

Save the actual instruction text and file paths alongside your results. A hash identifies a snapshot but cannot reconstruct its contents. Record failures and timeouts too, so a polished successful run does not hide the cost of earlier attempts.

Use a small review checklist

Before merging the instruction, ask: What specific problem does this address? Which tasks should load it? How will we check that it was followed? Does following it improve the task outcome? What would make us remove or revise it?

When results are mixed, keep that uncertainty in the record. You might narrow the instruction’s scope, rewrite an ambiguous command, or gather more representative tasks. Treat the file as maintained project guidance rather than a growing archive of every past incident.

The Sharper web planner can inventory supported instruction files and prepare historical task candidates for this comparison. It does not execute the agent or prove that a rule works. Use the local runner and evaluation guide to turn that brief into a controlled run.