Anthropic Engineer: In Production Prompting, Evals Matter More Than Clever Prompts

Anthropic engineer Margot Van Laar said most prompt engineering time goes into debugging and updating existing prompts.

This view came from her Code with Claude talk. She described real production work as ongoing maintenance rather than initial creation.

Teams often spend weeks refining prompts that already run. The focus shifts from clever wording to measurable outcomes.

Maintenance Takes Most of the Time

Van Laar noted that engineers rarely start from a blank page. They inherit prompts built for earlier models.

Old instructions create friction. Newer models overfit to outdated constraints. Removing legacy rules improves reliability.

She gave the example of a customer support agent. An old forbid list blocked valid tool calls. Deleting that list raised success rates without new phrasing.

Production prompting therefore resembles software upkeep. Each model update requires checks and adjustments.

Structured Prompts Reduce Drift

Van Laar recommended XML tags to separate sections inside prompts. The structure makes changes easier to track.

In the support example, tags isolated tool logic from safety rules. Engineers could edit one part without breaking others.

Clear boundaries also helped when switching models. The agent called tools with precise parameters after the update.

Retail Scheduling Agent Shows Another Pattern

Van Laar described a second case built from scratch. The team split one complex prompt into three smaller ones.

The first prompt generated a draft schedule. The second prompt evaluated that draft. The third prompt fixed identified issues.

Each prompt stayed simple. The separation improved stability. The team also selected a stronger reasoning model for the evaluation step.

Evals Replace Guesswork

Van Laar stressed that evaluation gives the only reliable signal. Without evals, changes rest on intuition.

She outlined three questions for any production prompt. Does the output match expected business results? What happens when inputs vary? How often does the agent need human handoff?

Teams that answer these questions with data make informed updates. Others keep tweaking prompts and hope for improvement.

Cost and Benefit of Human Handoff

The support agent example included explicit rules for escalation. The prompt weighed the cost of calling a human against the risk of a wrong answer.

Clear tradeoffs reduced unnecessary handoffs. They also prevented the agent from stalling on difficult cases.

Van Laar noted that these rules must be adjusted after each model release. What worked with one model may cost too much with the next.

Teams Need Reusable Context

Prompt maintenance grows harder without shared context. Engineers lose track of why earlier changes were made.

Tools that keep meeting notes, documents, and past decisions in one place help here. remio captures that context automatically and surfaces it when prompts are revised.

What Practitioners Should Watch

Model releases will continue to require prompt reviews. Teams that maintain running evals will adapt faster than those relying on manual inspection.

New evaluation frameworks may standardize how agents measure output quality. Adoption of those frameworks will show whether the practice spreads beyond early users.

Companies running agents in customer workflows should track escalation rates after each model update. Rising rates often signal prompt drift before users complain.

The pattern Van Laar described points to a shift. Prompt engineering work now resembles operations more than creative writing. Success depends on measurement loops that stay in place long after the first prompt ships.