Model benchmarks are useful for comparing capabilities under controlled conditions. Enterprise work is rarely controlled. A production task may depend on permissions, changing source data, ambiguous instructions, tool reliability and the consequences of a wrong action. A high model score cannot tell an organisation whether that whole system is dependable.
Evaluation must follow the work.
Begin with the job to be done
A useful evaluation starts with representative tasks and a clear definition of success. “Write a good answer” is too vague. “Identify the current renewal risk, cite the relevant customer interactions and route the account to the correct owner” can be measured.
Create a set that reflects the distribution of real work, including ordinary cases, difficult cases and cases where the correct behaviour is to abstain. Preserve the source state used for each test so results can be reproduced.
For agents, evaluate the trajectory as well as the final response. Anthropic's practical guide to agent evaluations notes that multi-turn systems introduce additional sources of variation and recommends combining several evaluation methods rather than relying on a single grader. That is vendor guidance, but the underlying principle is sound: an acceptable answer reached through unsafe actions is not an acceptable result.
Measure four layers
The first layer is outcome quality. Did the system complete the task correctly and at the required standard?
The second is evidence quality. Were material claims supported by current, authorised sources? Did the system surface conflicts and distinguish evidence from inference?
The third is execution quality. Did it use the right tools, remain within permissions, avoid unnecessary actions and leave the environment in the expected state?
The fourth is operational impact. Did the system reduce cycle time or rework? Did it create a new review burden? What was the severity of errors, not merely their count?
The NIST AI Risk Management Framework's Generative AI Profile is organised around governing, mapping, measuring and managing risk throughout the lifecycle. That lifecycle view is essential: evaluation is not a gate that ends at launch.
Test the system's ability to be uncertain
Enterprise data contains gaps and contradictions. An evaluation set that always contains a clean answer rewards confidence rather than judgement.
Include documents that are outdated, duplicated or scoped to the wrong team. Remove critical evidence from some tasks. Change permissions. Introduce a newer policy that invalidates an older one. Then measure whether the system finds the conflict, asks for clarification or declines to proceed.
This is particularly important for retrieval systems. The 2026 EnterpriseRAG benchmark reports a large gap between satisfying individual constraints and satisfying all requirements simultaneously under retrieval noise, knowledge gaps and factual conflicts. The paper is recent and should be read as emerging evidence, but it illustrates why clean-answer tests can overstate production readiness.
Treat productivity claims as hypotheses
Evaluation should challenge intuitive expectations. In a randomised study of experienced open-source developers using early-2025 AI tools, METR reported that participants took 19% longer on the measured tasks even though they expected AI to make them faster. The study was small and specialised, so it should not be generalised to all development work. Its value is methodological: perception and measured outcome can diverge.
The same discipline applies in every function. Measure the complete task, including review and correction. Compare like with like. Segment results by experience and task type instead of hiding variation in an average.
Keep a living evidence record
Every material change to a model, prompt, tool, data source or permission can alter system behaviour. Maintain a versioned evaluation record that links the tested configuration to results and known limitations.
Production signals should feed back into that record. Escalations, corrected answers and near misses are valuable test cases. Over time, the evaluation set becomes a compact representation of what the organisation has learned about operating AI safely.
The aim is not to produce one reassuring score. It is to build evidence that the system can perform the actual work—and to notice quickly when that evidence no longer holds.