Anthropic Simon Searchers Meet smevals, a Smaller Bet on AI Evaluation

Simon Willison released smevals after years of eval experiments, creating a fresh contrast with Anthropic’s more formal guidance on testing AI agents. The anthropic simon connection matters because both sides now emphasize the same problem. A model score says little unless teams also test prompts, tools, system instructions, and the harness surrounding that model.

Willison built smevals with Jesse Vincent’s Prime Radiant applied AI research lab. The project runs small evaluation suites across multiple configurations, grades their outputs, and generates reports for closer inspection.

The release challenges a common assumption about AI evaluation. Teams do not always need a large benchmark platform before they can ask a useful question. They need a focused task, repeatable configurations, explicit checks, and enough visibility to understand failures.

That makes the primary contest smaller and more practical than Anthropic versus another model provider. It is focused local evaluation versus heavyweight, generalized evaluation infrastructure. The former favors speed and inspectability, while the latter supports broader experiments and more complex environments.

What smevals changed for small AI evaluations

smevals turns a specific product question into a portable directory of tasks, configurations, and grading rules.

Willison announced smevals on July 31, 2026. His smevals overview describes it as a tool for running small eval suites across different model configurations and grading the resulting outputs.

The basic workflow begins with uvx smevals docs. That command gives a coding agent the project documentation, allowing the agent to study the format before building an evaluation suite.

This approach treats documentation as operational context. Instead of asking users to memorize every configuration field, the project expects a coding agent to read the instructions and help create the files.

An evaluation then lives inside a directory containing YAML files. YAML is a human-readable data format often used for configuration. Those files describe the question, tasks, model configurations, and grading behavior.

A user can run the same suite against multiple models. Willison’s example compares named GPT and Claude configurations through repeated -m arguments.

That command structure is important. It frames model choice as one variable inside a larger experiment, rather than treating the model as the entire product.

smevals also separates running from grading. The run command records what happened when a configuration attempted a task. The grade command later applies defined checks to those recorded results.

That separation creates a useful audit boundary. Teams can preserve raw behavior, revise their grading logic, and examine how a different rubric changes the interpretation.

The tool offers two reporting paths. The serve command starts a local web interface, while build produces static HTML that can be hosted elsewhere.

Willison demonstrated the workflow with a haiku evaluation. The report checked whether models produced exactly three nonempty lines and ranked configurations using the resulting grades.

A haiku benchmark is intentionally modest. It still illustrates a serious evaluation principle: narrowly defined requirements often reveal differences that broad preference scores cannot explain.

The release also introduces a consistent vocabulary. An eval contains tasks, while a configuration defines the model and other variables under examination.

A run records one configuration attempting one task. A grader produces a grade by applying checks, including deterministic checks or custom checker scripts.

Those custom checks can inspect strings, validate formats such as XML, or call another model for judgment. This range allows one suite to combine objective constraints with more subjective quality assessments.

Nothing in that workflow establishes smevals as an Anthropic product. The anthropic simon association comes from overlapping interest in agent evaluation and Claude configurations, not corporate ownership.

The immediate change is therefore accessibility. A developer can now package a small evaluation question without first adopting an extensive evaluation service or building a custom dashboard.

Why Anthropic Simon interest now centers on the harness

The model is no longer the only meaningful unit of comparison because the surrounding agent harness can change the result.

Anthropic defines an agent harness as the system that processes input, coordinates tool calls, and returns results. Its agent eval guidance distinguishes that layer from the evaluation harness that runs and grades experiments.

This distinction helps explain why smevals supports configurations beyond a model name. A configuration can also include different system prompts, model parameters, or agent harnesses.

Suppose two coding products use the same underlying model. One gives the model better repository context, while another provides stronger tools and clearer completion checks.

A model-only benchmark would treat those systems as equivalent. A configuration-level evaluation can show that their actual behavior differs.

The pressure falls on AI product teams that still select models using public leaderboards alone. Those rankings can help narrow a field, but they rarely reproduce a product’s exact prompts, tools, permissions, and data.

Agent behavior also unfolds across multiple steps. A system may call a tool, modify state, interpret the result, and decide whether to continue.

One early error can affect every later action. That makes an agent evaluation different from checking whether a chatbot answered one question correctly.

Anthropic’s guidance says teams evaluate the model and agent harness together when they evaluate an agent. That view closely matches the configuration model used by smevals.

This overlap is the real anthropic simon story. Both approaches move attention away from isolated model intelligence and toward the complete system that users experience.

The timing also reflects a growing operational problem. Models, prompts, and harnesses change independently, yet product teams still need to identify what caused a regression.

A new model may improve reasoning while changing output style. A revised system prompt may reduce verbosity but weaken instruction following. A harness update may expose better tools while introducing state errors.

Without controlled configurations, those changes become tangled. Teams see that a product feels different, but they cannot attribute the difference confidently.

Anthropic describes this condition as operating without enough visibility. Teams wait for user complaints, reproduce failures manually, patch one issue, and risk creating another regression.

smevals offers a smaller response to the same problem. It does not attempt to reproduce every production condition. It gives teams a structured way to isolate a question before expanding the experiment.

That matters for knowledge-intensive work. An engineering team might test whether an assistant finds the correct internal specification before generating code.

The test could compare two retrieval prompts, two model versions, or two tool policies. Teams maintaining a searchable knowledge base face similar questions whenever document access changes.

The resulting comparison is more useful than asking which model is best. It asks which complete configuration performs a defined task under stated conditions.

The mechanism is separation, not a smarter score

smevals gains clarity by keeping tasks, execution, grading, and reporting separate enough to inspect independently.

Many evaluation products promise a single score that makes comparison easy. That convenience can hide the decisions that produced the score.

smevals takes a more decomposed route. The eval states the larger question, and each task presents a specific challenge.

Configurations then describe the systems attempting those tasks. A run captures the attempt, while a grader evaluates the saved result through one or more checks.

This architecture sounds like ordinary testing because much of it follows ordinary testing logic. Inputs, conditions, outputs, assertions, and reports remain recognizable concepts.

Language-model behavior complicates each component. The same prompt can produce different answers, while several different answers may all satisfy the user.

A useful check therefore needs to match the requirement. Exact string matching suits a fixed token, but it performs poorly when several phrasings are valid.

Structural checks offer another option. A team can validate JSON, XML, line counts, required sections, or files created inside an agent environment.

Model-based graders handle less deterministic qualities. Another model can assess whether an answer follows a rubric, includes necessary reasoning, or meets a style requirement.

However, an AI judge does not turn a subjective question into objective truth. It introduces another model, prompt, and set of assumptions into the evaluation.

Separating grading from execution makes that limitation easier to investigate. A team can preserve the same runs and compare several grading methods without paying for every task again.

It can also inspect disagreements. If a format checker passes while an AI judge fails, the report reveals two different dimensions rather than averaging them immediately.

The reporting layer matters for the same reason. Aggregate scores help readers scan results, but individual runs reveal why one configuration succeeded or failed.

Willison’s haiku example illustrates this balance. A leaderboard provides the summary, while recent runs, task details, tags, and grader information expose the underlying evidence.

Static HTML adds another practical benefit. A team can publish a result without maintaining a live evaluation service.

The uvx entry point lowers setup friction as well. According to the official uv tool guide, uvx runs a packaged tool inside a temporary isolated environment.

That design suits short investigations. A developer can try the command without making a persistent global installation the first requirement.

The coding-agent workflow reduces another setup cost. The agent can read the project documentation, propose YAML files, and help refine the test.

Human review remains necessary. An agent-generated suite can encode vague expectations, omit hard cases, or create checks that merely reward its own assumptions.

The tool therefore does not eliminate evaluation design. It shortens the distance between a question and the first executable version of that question.

That difference matters. Teams often postpone evaluation because their imagined first step includes databases, dashboards, tracing systems, and a large golden dataset.

smevals proposes a narrower first step: encode one real uncertainty and run it across a few controlled configurations.

Small eval suites challenge heavyweight frameworks

The strongest case for smevals is not feature breadth but the ability to start with a bounded question and retain the evidence.

The evaluation market already includes broader open frameworks. The UK AI Security Institute’s Inspect platform supports datasets, solvers, scorers, agents, sandboxes, model providers, and detailed transcripts.

Its Inspect documentation presents a task as a combination of a dataset, a solver, and a scorer. The solver can make one model call or operate a multi-turn agent with tools.

Inspect also supports complex security evaluations and isolated execution environments. Those capabilities fit organizations running formal benchmarks or testing agents that modify external state.

Promptfoo approaches the problem from prompt and application testing. Its configuration format covers providers, prompts, test cases, assertions, and variables.

The official evaluation workspace shows how YAML can define providers, prompts, and expected behavior. That makes Promptfoo a relevant comparison for teams already treating prompts like testable code.

smevals enters this field with a smaller declared scope. Its advantage depends on whether that small scope stays coherent as users request more features.

A focused suite can be easier to review. Every task can relate directly to a product decision, and every configuration can represent a change a team might actually ship.

That focus also improves failure analysis. A test named around a concrete user need tells developers more than an abstract capability category.

Consider an assistant that prepares weekly product updates. A small suite might test whether it cites the correct meeting notes, distinguishes decisions from proposals, and avoids unsupported claims.

The configurations could vary the retrieval prompt, the model, and the document-selection tool. Graders could check citation presence, source identity, and factual consistency.

A public benchmark would not answer that product question. It lacks the team’s documents, expected workflow, and definition of a useful update.

Heavyweight frameworks remain valuable when the environment itself requires simulation. Browser agents, coding agents, and customer-service systems often need stateful tasks with reproducible databases or sandboxes.

Small YAML suites do not automatically recreate those conditions. They need compatible runners, scripts, fixtures, or other harness components.

This is why the primary opponent is a route, not a specific company. The choice is between starting locally with a narrow question and starting with generalized evaluation infrastructure.

Neither route wins every case. The smaller route wins when setup cost prevents teams from testing anything.

The broader route wins when the test must control complex state, capture full trajectories, enforce isolation, or operate continuously inside deployment pipelines.

The most useful progression may connect both. A team can discover valuable cases through smevals, then migrate mature tests into a larger regression system.

That progression only works if artifacts remain legible. Tasks, configurations, outputs, and grading rules must be clear enough for another engineer to reproduce.

smevals appears designed around that portability, but adoption will determine whether the convention holds. Tooling becomes harder to replace once custom graders and runners accumulate.

The project’s small size is therefore both the selling point and the test. It must add enough capability for real agents without recreating every complex evaluation platform.

What the scores still cannot settle

A repeatable suite can expose behavior, but it cannot guarantee that its tasks, graders, and samples represent production reality.

The first uncertainty concerns coverage. A compact suite can answer a narrow question well while overlooking rare failures that matter more than its average score.

Teams may also write tasks around known success cases. Coding agents asked to generate evals can produce plausible variations without discovering the surprising edge cases found by real users.

Production incidents should therefore feed back into the suite. Complaints, failed traces, support tickets, and manual reviews can reveal scenarios that synthetic task generation missed.

The second uncertainty is nondeterminism. Models can produce different outcomes across repeated attempts, even when the configuration appears unchanged.

One pass per task cannot distinguish a dependable configuration from one that happened to succeed. Repeated trials become essential when output variance affects the decision.

Anthropic’s evaluation guidance recommends examining success rates across multiple trials. It also warns that a model can find a valid solution the evaluator did not anticipate.

That creates a difficult failure mode. A rigid grader may punish a creative result even when the result better serves the user.

The opposite problem occurs with model graders. A generous AI judge may accept fluent output that violates an important hidden requirement.

Human calibration helps identify these errors. Reviewers should inspect passes and failures, compare grader decisions, and revise rubrics when the judge rewards the wrong behavior.

The third uncertainty concerns contamination between the system and evaluator. When coding agents help write tasks, prompts, and checks, their preferences can shape the benchmark.

Using a related model as the grader can deepen that effect. The test may favor familiar phrasing or reasoning patterns without measuring actual usefulness.

This does not make model grading invalid. It means the grade should remain traceable to a rubric, judge configuration, and review process.

The fourth issue is statistical confidence. A three-task suite can identify an obvious format regression, but it cannot support broad claims about model quality.

smevals calls itself a small eval suite, and readers should preserve that boundary. Its reports compare the tasks that were run, not every capability of the models involved.

Teams should avoid converting a local result into a universal ranking. “Configuration A passed eight product cases” is supportable. “Model A is better” usually is not.

Costs and latency require similar care. A configuration that scores higher may use longer prompts, more tool calls, or a slower reasoning mode.

If those factors matter to the product, the suite needs to record and compare them. Quality scores alone cannot determine the best shipping choice.

Security also changes the evaluation design. An agent with shell, browser, or database access needs isolated environments and checks on final state.

A transcript may show that an agent claimed success. The real outcome depends on whether it created the correct file, changed the intended record, or avoided prohibited actions.

These limitations are not an argument against small evals. They define where a small suite remains trustworthy.

The anthropic simon convergence is useful precisely because neither approach treats an aggregate number as the finish line. Runs, traces, outcomes, and grader behavior all deserve inspection.

Three signals will show whether the Anthropic Simon overlap lasts

smevals will matter beyond its launch if teams use it to compare real harness decisions, calibrate graders, and preserve repeatable evidence.

The first signal is the range of published evaluation suites. Haiku formatting proves the workflow, but agent developers need examples involving tools, state, and multi-step completion.

Suites that compare prompts alone would keep smevals close to established prompt-testing tools. Suites that compare coding or research harnesses would support its broader positioning.

The judgment strengthens if users publish reproducible agent cases with visible runs and checks. It weakens if examples remain limited to short text formatting tasks.

The second signal is grader calibration. The project supports deterministic checks and more complex checker scripts, including model-based assessment.

Users now need methods for comparing those grades with human judgment. Useful reports should expose disagreements rather than concealing them inside one score.

The case for smevals strengthens if teams can rerun grading, inspect rubrics, and document why graders changed. It weakens if leaderboards become detached from the underlying evidence.

The third signal is integration with everyday development. A local experiment creates insight once, while a regression suite protects future changes.

Watch whether teams run smevals after model updates, prompt edits, tool changes, and harness releases. Repeated use would show that small suites can become durable engineering assets.

Integration does not require every team to build an elaborate platform. A shared repository, reviewed YAML, saved runs, and a consistent release check may be enough.

The signal weakens if suites become stale after the initial comparison. An outdated benchmark can create confidence without reflecting the current product.

For developers and enterprise buyers, the practical action is simple. Identify one decision currently being made through intuition, then define the smallest test that could challenge it.

That decision might involve Claude versus GPT, but it could also involve two system prompts or two retrieval strategies. The configuration should reflect what users actually experience.

Treat the first result as evidence, not a verdict. Inspect failures, question the grader, add cases from real work, and repeat trials where behavior varies.

The lasting anthropic simon lesson is not that one small tool solves AI evaluation. It is that model choice, prompt design, and harness behavior must be tested together.

Which product decision is your team still making from demos, leaderboards, or instinct? Turn that uncertainty into a focused suite, preserve the runs, and see whether the evidence changes the answer.