SynthID Safety Risks Grow as AI Watermarking Changes Model Behavior

Google DeepMind’s watermarking method now carries a troubling conflict: new tests found that it changed model responses to harmful prompts. The SynthID safety risks were most visible during prompt injection, when several models followed requests they had rejected without watermarking.

The finding challenges a central assumption behind invisible AI text marks. A watermark may preserve readable quality while still changing which tokens a model selects. Those token changes can affect refusals, tool choices, and tool arguments inside AI agents.

The immediate pressure falls on Anthropic, Google, and developers building agents around hosted models. Anthropic plans to add SynthID-Text to future Claude models as part of its response to European transparency rules. However, the new tests did not evaluate Claude or Anthropic’s production implementation.

That distinction matters. The research is not evidence that Claude has become less safe. It is evidence that watermarking must be treated as a behavioral system change, not a passive label attached after generation.

Researchers Found a Behavioral Shift, Not Just Different Wording

The new result is important because SynthID altered individual safety decisions even when overall performance appeared relatively stable.

Andrea Siposova, an AI security researcher at Lasso Security, compared watermarked and unwatermarked generations under tightly paired conditions. The model, prompt, random seed, batch composition, and temperature stayed fixed. The watermark processor was the only intentional difference.

Her experiments used the unmodified SynthIDTextWatermarkLogitsProcessor available through Hugging Face. The configuration included 30 tournament layers, a five-token context, a sampling table of 65,536 entries, and a context history of 1,024.

The refusal evaluation covered 200 harmful behaviors from HarmBench. It also included 100 benign controls from JailbreakBench. Researchers tested harmful requests both alone and with one fixed prompt-injection technique.

Prompt injection is an adversarial input that attempts to override an application’s intended instructions. In this experiment, retrieved text falsely claimed that safety filtering was disabled and told the model to comply.

The results varied considerably by model. At a temperature of 0.001, Gemma 3 27B changed its verdict on 6 percent of bare harmful requests. Under prompt injection, that disagreement rate rose to 23.5 percent.

More importantly, the direction changed. Watermarking reduced harmful compliance by one percentage point for bare requests. Under injection, harmful compliance increased by 12.5 points.

Gemma 3 12B followed a similar pattern. Its disagreement rate rose from 7.5 percent on bare requests to 11 percent under injection. The net increase in harmful compliance reached nine percentage points.

Llama 3.1 8B produced disagreement rates of 14 percent at the lower temperature and 17.5 percent at 0.7. Its net shift was less conclusive, but individual decisions still changed.

Phi-4 and Qwen3 4B showed little movement. However, both models frequently refused harmless control requests. That high baseline refusal rate makes their apparent stability difficult to interpret as better safety.

The complete behavioral study describes this effect as “sampling drift.” The term means that a generation intervention changes decisions because it changes the tokens selected during inference.

This is not the same as showing that every watermark weakens every model. Some watermark keys moved behavior toward safer responses. Others increased attack success, and the magnitude varied by model.

The strongest conclusion is narrower and more useful. A watermark can alter safety behavior, so evaluations conducted without it may not describe the deployed system.

The experiments also examined seven open-weight models on tool-calling tasks. Watermarking reduced accuracy on six models, with statistically significant declines on four.

Researchers used 1,150 fixed, non-live tasks from the Berkeley Function Calling Leaderboard for temperature comparisons. Across 21 model and temperature combinations, the average paired disagreement rate was 6.5 percent.

For Phi-4 at temperature 1.0, 16.8 percent of individual tool-call verdicts changed. Yet its net accuracy fell by only 2.87 percentage points.

Llama 3.1 8B showed the same masking effect. Its verdicts changed on 9.9 percent of tasks, while aggregate accuracy declined by only 0.87 points.

Those gaps expose a weakness in top-line benchmark reporting. One call can change from correct to incorrect while another improves, making the average look stable.

For an agent, the errors are not interchangeable. A correct-looking function call with the wrong recipient, path, query, or amount can execute successfully and cause harm.

SynthID Safety Risks Put Agent Developers Under Pressure

Developers can no longer assume that a provider-side watermark leaves an agent’s tested behavior intact.

Anthropic announced on August 14, 2026, that future Claude models would generate watermarked text. The company said its method uses SynthID-Text and will support compliance with the European Union’s AI Act.

Anthropic describes the mark as an invisible statistical pattern in word selection. Nothing is appended to the text, and the system does not insert hidden characters.

The company says its internal testing found no practical effect on content, creativity, readability, or output quality. Its public watermark explanation also cites Google’s large-scale quality evaluation.

That claim and the new findings are not direct contradictions. They measure different properties.

A response can remain fluent and earn the same user rating while containing a different tool argument. It can also sound equally polished while crossing from refusal into compliance.

The original SynthID-Text researchers studied detectability, latency, and perceived response quality. Their Nature paper reported a live experiment covering nearly 20 million Gemini responses.

Standard benchmarks and human comparisons found no measurable capability or quality degradation. The watermark also added negligible computational overhead in the tested production design.

Lasso’s work asks another question. Does the same system make the same decision on the same input after token sampling changes?

That question becomes urgent when a model operates behind an agent. A chatbot response ends as text, but an agent may convert generated tokens into executable actions.

An assistant could select a calendar function, database query, email recipient, file path, or payment amount. Small changes in prose may be harmless. Small changes in structured arguments are not.

The pressure is also difficult for application developers to manage. A hosted model provider can activate watermarking or rotate its secret key outside the application developer’s release cycle.

An agent team may have validated a model before that change. Its monitoring might show similar overall accuracy afterward, even while a different set of individual requests fails.

The Lasso experiments found that watermark effects depended on the secret key. Researchers tested the main study key and ten additional keys at temperature 0.7.

For Llama 3.1 8B, the study key raised attack success by 3.5 percentage points. The other keys averaged a 4.4-point increase, ranging from a 4.5-point decrease to a 14.5-point increase.

Both tested Gemma models also showed mostly higher attack success across keys. Granite 3.2 8B moved in both directions, while Phi-4 and Qwen3 4B remained near their baselines.

That range makes key rotation a security-relevant event. A provider might rotate keys to protect detection integrity, but the replacement key can generate a different behavioral profile.

Developers therefore need deployment-specific tests. A pre-watermark safety report cannot establish that the watermarked version preserves the same refusals and tool calls.

The responsibility cannot sit only with model providers. Agent builders know which functions are available, which data is sensitive, and which incorrect arguments create meaningful damage.

Providers control the watermark configuration. Application teams control permissions, confirmation steps, execution boundaries, and many runtime checks.

Both sides now face a shared testing burden. Providers must disclose behaviorally relevant changes, while developers must repeat adversarial evaluations against the actual deployed endpoint.

How SynthID Watermarking Changes a Model’s Decisions

SynthID watermarking explained at the token level reveals why a provenance feature can influence both language and action.

A large language model generates output by repeatedly selecting the next token. Tokens can represent words, word fragments, punctuation, code elements, or structured values.

The model first assigns probabilities to possible next tokens. Sampling settings then determine which candidate becomes part of the response.

SynthID-Text intervenes during that selection process. It uses secret keys and a scoring function to create a detectable statistical signal across many choices.

Google’s SynthID documentation describes the system as a logits processor applied after Top-K and Top-P filtering. Logits are the model scores used to calculate token probabilities.

In the tested configuration, candidates pass through tournament sampling. Token pairs compete using hidden scores derived from the key, and winners advance until the system selects a final token.

The non-distortionary design preserves the original distribution when averaged over watermark randomness. However, that mathematical property does not require identical output under a particular fixed key.

This distinction sits at the center of the SynthID safety risks. The distribution can remain correct in expectation while a deployed key changes a specific response.

Consider a sentence where “overcast” and “cloudy” carry nearly identical meaning. Selecting either word rarely changes the practical result.

Now consider structured output containing a destination path or recipient. Several values may remain plausible to the model, but only one matches the user’s intention.

A watermark does not need to force an absurd token to cause trouble. It only needs to select a different plausible candidate at a consequential point.

The same mechanism can affect refusals. Safety training does not insert a fixed refusal message that always appears unchanged.

A refusal is another generated sequence. Early token choices can move that sequence toward rejection, partial assistance, or direct compliance.

Prompt injection increases the stakes because it places competing instructions inside the model’s context. The model must resolve a conflict between trusted directions and attacker-controlled text.

A small sampling shift can change how that conflict resolves. Once the model begins complying, later tokens may continue along the new trajectory.

This explains why ordinary quality testing can miss the problem. Readability scores focus on whether an answer sounds coherent and useful.

They generally do not ask whether an agent selected the identical function or preserved the same security boundary. They also average results across many interactions.

The Lasso team used paired disagreement to make those hidden changes visible. Each prompt was evaluated with and without watermarking under otherwise matched conditions.

That method distinguishes two effects. Net accuracy measures the final average, while churn measures how often individual verdicts change.

For deployed agents, churn can be the more informative signal. A stable average does not protect a specific customer whose email goes to the wrong recipient.

SynthID vs model safety is therefore not a simple contest between transparency and protection. Provenance and safety measure separate properties, and one does not guarantee the other.

Detectability asks whether a statistical signal survives in generated text. Behavioral stability asks whether the same input produces an acceptably equivalent decision.

Text quality asks whether people notice degradation. Security asks whether adversarial inputs can cause unauthorized behavior.

A watermark can perform well on the first and third measures while creating instability on the second and fourth. Each property requires its own evaluation.

The Compliance Fix Creates a Security Tradeoff

A system intended to improve accountability can introduce new uncertainty inside the models expected to enforce safety rules.

European regulators want synthetic content to remain identifiable. Article 50 of the AI Act requires certain AI outputs to be marked in a machine-readable and detectable format.

The rules call for methods that are effective, interoperable, reliable, and sufficiently resistant to removal. The European Commission’s transparency guidance connects these duties to misinformation, impersonation, fraud, and consumer deception.

Those goals address genuine problems. Text generators can produce large volumes of convincing material, while copied output loses most visible information about its origin.

SynthID offers a technically appealing response. Its signal travels with the words rather than relying only on detachable metadata.

However, regulators and providers have mostly discussed watermarking as an identification layer. The new research shows that generation-time marking can also become part of the model’s decision process.

That creates a tradeoff between provenance and behavioral stability. It does not mean the two goals are incompatible, but neither goal can be assumed free.

Anthropic’s rollout makes the issue more immediate. The company says future supported Claude models will mark output at the model level, including text served through APIs and cloud platforms.

An independent developer may never call a watermarking function. The generated tokens arriving from the provider can already reflect the marking process.

That architecture broadens the affected surface. Enterprise search assistants, coding agents, customer-support systems, and research tools can all consume marked output.

An agent with read-only access presents one risk profile. An agent authorized to send messages, modify records, run code, or approve transactions presents another.

The watermark also interacts with layered defenses. A system may combine model refusals, prompt filtering, tool permissions, argument validation, and human confirmation.

Sampling drift does not automatically defeat every layer. It can still weaken the model-level decision that determines whether later defenses receive a dangerous request.

The finding supports a stricter release process. Teams should compare matched watermarked and unwatermarked runs before enabling a mark in production.

They should measure refusal-to-compliance changes under prompt injection. They should also track correct-to-error and error-to-correct transitions for individual tool calls.

Aggregate accuracy remains useful, but it cannot stand alone. Tests should separate malformed calls, incorrect tools, and incorrect arguments because those failures carry different consequences.

High-risk tools need deterministic controls outside the model. Recipient allowlists, schema validation, scoped credentials, transaction limits, and explicit user approval can reduce exposure.

Logs must also preserve enough context for comparison. Teams need the prompt, model version, sampling settings, tool schema, policy configuration, and watermark state.

Key changes deserve the same caution as model updates. A new key should trigger focused regression testing because the research found directionally different effects across keys.

The organization deploying an agent should also maintain evaluation cases drawn from its real workflows. Generic benchmarks cannot represent every sensitive argument or business rule.

This is where SynthID vs model safety becomes an operational question. The relevant standard is not whether a watermark changes something, since stochastic generation already varies.

The standard is whether the marked system remains inside defined safety tolerances. That must be measured under realistic attacks and permissions.

What the Study Does Not Establish

The evidence identifies a credible failure mode, but it does not prove that Claude, Gemini, or every SynthID deployment is less safe.

The experiments evaluated open-weight models because researchers needed direct control over token sampling. They did not test a future watermarked Claude model.

They also used Hugging Face’s public implementation of SynthID-Text. Anthropic may use different settings, surrounding safeguards, decoding controls, or deployment tests.

The refusal experiment used one fixed prompt-injection technique. Attackers can use many strategies, and different prompts may produce smaller, larger, or reversed effects.

The study separated refusal testing from tool-call testing. It did not demonstrate an end-to-end agent accepting the injected harmful request and executing a damaging tool call.

That combined scenario remains an inference. It is plausible because the research separately observed changed refusals and changed tool behavior, but it requires direct validation.

The results also resist a universal direction. Some keys increased harmful compliance, while others reduced it. Several models barely changed under the tested conditions.

That variability weakens any claim that watermarking inherently makes models unsafe. It strengthens the case that safety cannot be inferred from the algorithm’s general design.

The research was published by Lasso Security, a company that sells AI security and red-teaming products. Its commercial position does not invalidate the data, but independent replication would improve confidence.

Researchers should reproduce the paired tests across additional implementations, model sizes, prompt injections, and agent architectures. Provider-run studies should publish more than average quality scores.

The original SynthID evaluation remains meaningful. Nearly 20 million response ratings offer strong evidence that users did not perceive broad quality degradation in that deployment.

However, thumbs-up rates answer a different question from adversarial refusal stability. Both findings can be accurate at the same time.

The same care applies to the reported findings. The strongest evidence concerns the tested models and configuration, not every product carrying the SynthID name.

Anthropic says its watermark does not change output meaning or quality. Public evidence currently supports caution, not a declaration that this statement is false.

Meaning is difficult to define for an agent action. Two natural-language responses can appear equivalent while producing different structured arguments downstream.

A provider might also achieve stable production behavior through configuration choices that the public implementation does not reproduce. That possibility needs testing rather than assumption.

There is another uncertainty around normal model variability. LLM outputs already change with random seeds, temperature, infrastructure, and model revisions.

The study addressed part of this issue by comparing matched pairs and examining temperature-induced churn. At temperature 0.7, watermark-induced refusal churn exceeded temperature-induced churn on four of six models.

Gemma 3 27B showed 26 percent watermark churn under injection, compared with 13.5 percent from the temperature change. Llama 3.1 8B showed 17.5 percent versus 7.5 percent.

Granite 3.2 8B reached 21.5 percent watermark churn and 15.5 percent temperature churn. Gemma 3 12B showed 11 percent and 6 percent, respectively.

These comparisons suggest that the watermark was not merely adding ordinary randomness. Yet they still represent selected settings, models, and tasks.

The responsible conclusion is specific. SynthID watermarking explained as a quality-preserving provenance method is incomplete unless behavioral security is measured separately.

Three Signals Will Show Whether Providers Contain the Risk

The next evidence should come from deployment-specific testing, independent replication, and visible changes to agent security practices.

The first signal is Anthropic’s evaluation of watermarked Claude models. The company should report paired refusal and tool-calling results before or alongside deployment.

Useful disclosure would separate ordinary prompts from prompt injection. It would also show individual disagreement rates rather than only average accuracy or user preference.

If Claude remains stable across keys and adversarial tasks, that result would narrow the SynthID safety risks to particular models or configurations. Missing data would leave the central uncertainty unresolved.

The second signal is independent replication across watermark implementations. Researchers need to test the reference code, Hugging Face integration, and production-like configurations.

Replication should include several secret keys and multiple injection techniques. It should also connect refusals with actual agent actions inside controlled environments.

Consistent shifts across models would strengthen the claim that generation-time watermarking creates a general security cost. Mixed findings would point toward configuration-specific mitigation.

The third signal is whether agent developers treat watermark changes as security releases. A provider announcement should trigger regression tests, permission reviews, and renewed red-team exercises.

Teams should watch for changed tool arguments, not only failed syntax. They should compare the same high-risk cases before and after model, configuration, or key updates.

A mature response would also move critical controls outside probabilistic generation. Agents handling sensitive actions should require validated arguments and explicit confirmation at execution time.

For knowledge workers, the immediate lesson is simpler. A watermark says something about likely origin, not truth, safety, or unchanged behavior.

Developers should ask providers whether watermarking is active, where it is applied, and whether keys can change without versioned notice. Enterprise buyers should request adversarial evaluation results.

Security teams should add watermark state to their model inventory and incident records. Product leaders should decide which agent actions remain acceptable when individual decisions can drift.

The open question is no longer whether invisible text marks affect readability. It is whether provenance can be added without changing the decisions that matter most.

SynthID vs model safety should not become a choice between transparency and secure agents. It should become a requirement to verify both properties under the exact deployed configuration.

Until providers publish that evidence, teams should assume that enabling a generation-time watermark changes the system. They should test it with the same care as a model upgrade.