How to choose useful tasks for a coding-agent evaluation

Choose the decision before choosing the tasks

Be explicit about what you want to learn. Testing a new instruction about database migrations calls for different tasks than deciding whether an agent can handle frontend maintenance. Write a one-sentence question, then use it to include or exclude candidate work.

For an initial instruction comparison, a handful of reviewed tasks can expose obvious problems in your setup. It cannot establish broad performance across a codebase. Plan to expand the set once the prompts, environment, and checks are working reliably.

Look for a recoverable starting point

A merged pull request gives you a real change to investigate, but it is not automatically an evaluation task. You still need the intended pre-change commit, the dependencies needed to run it, and a problem statement that makes sense from that point in time.

Check that the original behavior is reproducible before asking an agent to fix it. If the application cannot start because a service is gone or a dependency cannot be installed, the task may measure environment repair instead of the capability you intended to test.

Inspect commit references rather than trusting their names. Confirm that the starting snapshot does not already contain the solution, and keep your reference implementation outside the agent’s accessible workspace.

Write the requirement without giving away the patch

Use the original issue or a carefully reconstructed requirement as the prompt. Describe the expected behavior and relevant constraints. Do not paste a merged diff, solution-bearing review comments, or implementation notes that would make the task easier than the original work.

For example, consider a hypothetical date filter that incorrectly excludes records on the final day. A task can specify the inclusive date range and timezone expectation without naming the line to change or supplying the comparison operator.

Illustrative task card:

Problem: Records on the selected end date are missing.
Expected: Include all records within both selected calendar dates.
Constraint: Use the account’s configured timezone.
Checks: Start boundary, end boundary, outside range, empty result.
Starting point: A reviewed commit where the bug is reproducible.

Build checks that allow more than one valid solution

Grade the required behavior and the regressions that matter. A test that compares the agent’s output to the exact reference diff can reject a correct alternative. A test that only checks whether the application starts can miss the actual defect.

For the date-filter example, prepare fixtures on each boundary and outside the range. Include a timezone case because that constraint is part of the requirement. Then run the relevant existing tests to check that the fix did not break another filter behavior.

Keep the evaluator separate from the agent’s edits and inspect unexpected passes. A change that disables a failing check is not the same as a fix. If a check is ambiguous or incorrect, repair the evaluation and rerun both comparison conditions.

Include work where the instruction should not matter

A rule can help one category of work while creating unnecessary steps elsewhere. Include some representative tasks outside the rule’s scope so that you can observe this tradeoff. Label the task categories before the runs rather than selecting favorable groups afterward.

Keep a few additional tasks out of the instruction-writing loop. Once you have revised the rules using your development set, evaluate them on those held-out tasks. Repeatedly tuning to the same examples can make a rule look more useful than it is on new work.

Record every attempt and the limits of the result

Agent outputs can vary across attempts, so repeat each task with fresh sessions and fixed conditions. Anthropic’s evaluation guidance distinguishes the task, each trial, the grading logic, and the actual outcome. That separation is useful when diagnosing whether a failure comes from the agent or the test setup.

Record successes, failures, timeouts, elapsed time, and cost. Compare the same tasks across both conditions. If one condition fails every attempt, say so; do not report a zero cost per successful task when there were no successes.

A candidate list is the start of this process. Sharper Than AI prepares a shortlist from public pull-request history and inventories the instruction files. You still review each task, define the evaluator, and run the agent yourself. Our evaluation guide and blank scorecard help make those steps explicit.