Evaluation guide · September 28, 2026
How to test whether AGENTS.md helps your coding agent.
An instruction file can look useful without improving a single task. Test it by giving the same agent the same work, with and without the instructions, and checking what actually changes.
The free planner inventories instructions and finds historical PR candidates. You review the tasks and run the evaluation yourself.
Prepare a free experiment brief →1. Choose tasks with verifiable outcomes.
Start with three to six merged changes that represent the work you want the agent to do: a bug fix, a small feature, or a refactor. Avoid a set made entirely of dependency updates. More tasks and repeated runs will be needed before making a confident general claim.
For each task, write a problem statement from the original issue or requirements. Define the tests and acceptance criteria before running the agent. A merged pull request is a source of evidence, not a ready-made prompt: its title may be vague and its description may contain the solution.
The uv example brief includes six PR candidates and a detected instruction file. These are starting points for manual review, not six completed or validated benchmark tasks.
A worked task specification
For a hypothetical CSV importer bug, a useful prompt is: “A blank trailing row causes an import failure. Ignore blank trailing rows while preserving errors for malformed nonblank rows.” The evaluator should test those two cases and run the existing importer suite. Do not give the agent the merged diff or the new test implementation.
2. Change only the instruction context.
AGENTS.md provides repository guidance to coding agents, including setup commands and conventions. The relevant instructions can span root and nested files. Check which files your chosen agent actually loads; a file appearing in an inventory does not prove that it was consumed.
Instructions disabled
Start at the reviewed pre-change commit. Remove or disable applicable repository instructions in a disposable checkout.
Instructions enabled
Start at the same commit, with the exact reviewed instruction snapshot installed at the same paths.
Keep the model version, agent version, prompt, permissions, environment, tools, budget, and evaluator the same. Use fresh agent sessions. Record any global instructions or memory that remain enabled in both conditions.
If you are testing an edit to CLAUDE.md or another existing instruction file, compare the old snapshot against the proposed snapshot instead. Do not change several unrelated rules and then attribute an improvement to just one of them.
3. Isolate each run and prevent answer leakage.
Use disposable clones or containers with a clean working directory per run. Verify that the selected base commit does not already contain the solution. The planner includes commit references from GitHub; confirm those references and reconstruct the intended pre-change state before treating them as an executable benchmark.
Keep reference diffs, expected outputs, and evaluator implementations outside the agent workspace. Restrict access to repository history, remotes, and network sources that expose the answer. A worktree alone does not isolate Git history.
Current instructions may mention APIs or files introduced after a historical task. Review them for future information and either exclude the task or record the limitation. Save the actual instruction text and its paths, alongside the brief’s fingerprint: a fingerprint cannot restore a file.
Run both conditions several times with fresh sessions. Alternate their order and record timeouts and failures. Never quietly rerun only the failures in the condition you prefer.
4. Record results before interpreting them.
Use one row for each task, condition, and repeat. The scorecard is intentionally blank: it contains no invented benchmark results.
Download the CSV scorecard ↓| Measure | What to record |
|---|---|
| Verified success | Acceptance checks and existing tests pass; inspect the change for shortcuts. |
| Regressions | Previously passing checks broken by the change. |
| Cost | Input and output tokens, provider-reported spend, and failed-run spend. |
| Time | Elapsed seconds, timeouts, and agent steps. |
| Adherence | Whether applicable, concrete rules were followed; keep this separate from success. |
Calculate success rate as successful runs divided by all attempted runs. Calculate cost per success as total spend across all attempts divided by successful runs. If there are no successes, report that explicitly rather than reporting a zero cost per success.
5. Keep the rules that earn their place.
Compare paired results task by task before looking at averages. Instructions that reduce exploration but introduce regressions are not an unqualified win. A single faster run is a reason to investigate, not proof of an improvement.
If the sample is small or results vary, report “inconclusive,” expand the task set, and repeat. If a rule consistently helps a specific task category, narrow its scope instead of requiring every task to pay its context cost. Keep a held-out task set to check that your edits generalize.
Repeat this evaluation when instructions or agent versions change. That recurring comparison is the foundation of context regression testing. The web planner prepares the brief and the local runner executes the paired comparison; continuous hosted monitoring is not part of the current project.
Build your first comparison.
Inspect an example, then prepare an inventory and task shortlist for your own public repository.