In a coaching session, a team's agent passed every step check and still shipped a broken report. When they switched to checking the output file, they caught the problem in the first run.
That is the whole lesson, and it is easy to miss. This post is for engineers and team leads writing their first evals. You'll get one rule for where to put your checks, and one small exercise to test it on your own agent.
Every check passed and the report was still broken
The team had done what most of us do first. Their checks were of two kinds: structural checks and content quality checks.
Every one of those checks passed. The agent still shipped a broken report. It had a typo, and a required Objectives section was missing.
That gap is the whole story. The checks graded the agent's work before the report reached anyone. The reader of the report only cares whether the report works.
Step checks answer a different question than the one you care about
A step check asks, "Did the agent do what I expected?" An output check asks, "Is what it made correct?" These sound close, but they fail independently.
The team's case shows it. Their structural and content quality checks passed, yet the finished report had a typo and no Objectives section. Whatever each check verified along the way, only the finished file could say whether the report was complete and correct.
My own inference, not Anthropic's: step checks also pile up. Each one encodes a guess about how the agent should work, and every guess is another place for the eval to be wrong.
Switching to the output file caught the problem in the first run
The team switched to checking the output file, the thing the user actually receives. That check caught the broken report in the first run.
The defects were ordinary: a typo and a missing required Objectives section. The earlier checks had passed with both present.
Anthropic's eval guidance points the same way
Anthropic's engineering team defines an outcome as "the final state in the environment at the end of the trial". Their example is a flight-booking agent: the agent may say it booked a flight, but the outcome is whether a reservation exists in the database.
They are also direct about step-checking. They write that checking exact step sequences "results in overly brittle tests," and that it is often better to grade what the agent produced, not the path it took.
That matches what I saw, with one extension that is mine and not Anthropic's. Brittleness is not the only cost of step checks: a check on the route can pass while the result fails.
Where step checks still earn their place
I don't want to overstate this. Output checks have limits, and I have seen this on one team in one session, not across a large sample, so treat it as a strong hint rather than a proven rule. I also can't say whether an output check would have missed something the original checks caught.
First, an output check tells you that something failed, not why. When the file is wrong you still need the transcript to find the cause. Anthropic makes the same point: after confirming outcomes, it is often useful to also grade the transcript.
Second, some outcomes are hard to inspect. In my view, if the agent's job is a side effect you can't easily read back, a step check may be the only guard you have. That is my opinion, not something the team or Anthropic tested.
So keep step checks, but demote them. Use them to diagnose a failure, not to declare success. When a run fails, the step checks and the transcript help you find where it went wrong. When a run passes, only the output tells you whether to trust it.
One thing to try this week
Pick the single artifact your agent hands to a person: a report, a file, a record. Write one check that opens that artifact and tests the thing the reader depends on. Then run it against a run your current evals already call a pass.
Write the check from the reader's side. Ask what a person would reject on sight: a required section that is absent, a misspelled word, an empty field. Turn each answer into an assertion on the finished file, and keep the list short enough to read in one sitting.
If it passes, you've gained confidence. If it fails, you've found a real defect the current checks missed.
Check the thing your user receives
Green step checks tell you the agent followed a plan. They don't tell you the plan produced something that works. Grade the output, and use the steps to explain what went wrong.
If you try the exercise, I'd like to hear what it caught. Reply with your result, even if the answer is that nothing was wrong.