AI Watermarks Risk Drifting Agent Behavior

Overview

Researchers have identified a subtle but significant risk in the growing practice of embedding invisible watermarks into large language model outputs. A recent study, The Provenance Tax, reveals that these security features can alter how AI agents behave, potentially causing them to call incorrect tools or fail to refuse harmful requests.

The findings come as major providers like Anthropic begin implementing SynthID-Text, a watermarking system based on Google DeepMind’s technology. While designed to ensure content provenance and comply with emerging regulations like the EU AI Act, the process of embedding these marks changes the fundamental way models generate text.

"Such a watermarking procedure can therefore affect both what the model says and what an agent does. We call this behavioral effect sampling drift."

This drift occurs because watermarking algorithms bias token selection during generation. In simple chat interfaces, this might result in minor stylistic variations. However, when the model is integrated into an agent capable of taking actions through software tools, these variations can lead to concrete operational failures.

The Mechanism of Sampling Drift

Text watermarking is not a new concept, but its deployment at scale has significant regulatory implications. Article 50(2) of the EU AI Act requires providers to mark synthetic text in a machine-readable format. To meet this standard, systems often use generation-time watermarking rather than post-processing, as the latter is easier to remove.

Anthropic’s integration of SynthID-Text utilizes a method called tournament sampling. Unlike ordinary sampling, which selects tokens directly from the model’s probability distribution, tournament sampling introduces a random seed generator and a scoring function. This process subtly biases which tokens are chosen, particularly in situations where the model is uncertain about the best next word.

The study emphasizes that while this configuration is "non-distortionary" in expectation—meaning the average output quality remains unchanged—a specific watermark key can still cause individual generations to diverge from unwatermarked versions.

For a standard conversational AI, this divergence is often negligible. However, for an AI agent, the stakes are higher. Agents rely on precise token sequences to determine which tools to invoke and what arguments to pass. A slight change in token selection can turn a valid function call into an invalid one, or alter a critical parameter such as a file path or a recipient address.

Methodology and Experimental Design

To quantify this risk, the researchers employed a paired experimental design. They compared the behavior of watermarked and unwatermarked models under identical conditions, varying only the presence of the SynthID watermark processor.

The study utilized the following datasets:

• BFCL v4 Single-Turn AST: Used to evaluate tool-calling performance across live and non-live tasks, including relevance and irrelevance checks.
• HarmBench: A collection of 200 harmful behaviors used to test whether the model properly refuses dangerous requests.
• JailbreakBench: 100 benign controls to ensure that harmless requests were still answered correctly.

The experiments tested models at various temperatures (0.001, 0.7, and 1.0) to see if the level of randomness affected the degree of drift. Crucially, the team also tested harmful requests both in their raw form and under a fixed prompt injection technique, simulating real-world adversarial scenarios.

Findings: Tool Calling and Safety Refusals

The results confirmed that sampling drift is a practical reality, affecting both tool execution and safety guardrails.

Impact on Tool Calling

The study found that watermarking could change the specific tools an agent selected or the arguments it generated. Because structured outputs like JSON contain highly predictable elements (such as braces and keys) alongside less predictable values (such as queries and numbers), the watermark’s bias often shifted the uncertain parts of the output.

While the overall performance metrics might not show catastrophic failure, the paired disagreement analysis revealed that watermarked and unwatermarked runs often produced different results. This means that an agent that works correctly without a watermark might fail intermittently once the feature is enabled, depending on the specific random seed applied.

Impact on Safety and Refusal

Perhaps more concerning was the effect on safety behavior. The research indicated that watermarking could weaken a model’s refusal to answer harmful prompts. When combined with prompt injection techniques, this weakening became more consequential.

A model that would normally refuse a dangerous request might instead comply, or it might provide a partial compliance, when operating under a specific watermark key. This highlights a dangerous intersection between provenance tracking and security: the very mechanism designed to trace AI output can inadvertently make the AI more vulnerable to manipulation.

Implications for Developers and Safety

The study suggests that the benefits of content provenance should not be assumed to come without cost. For developers building agents on top of watermarked models, the risk is twofold: functional reliability and security robustness.

Aggregate scores may obscure these issues because positive and negative drifts can cancel each other out in broad statistics. Therefore, the authors recommend that evaluation frameworks prioritize paired disagreement analysis over simple accuracy metrics when assessing watermarked systems.

Furthermore, the research calls for greater scrutiny of how watermarking interacts with prompt injection. As agents gain more autonomy and access to external tools, even small behavioral shifts induced by watermarking could lead to significant unintended actions. The findings urge the industry to consider whether the technical solutions used for compliance are robust enough for the increasingly autonomous agents they power.