Agent compliance drift: What it is and how to control it

Agent compliance drift is the widening gap between what your approved documentation says, an AI agent does and what it actually does in production. The agent usually keeps hitting its goal. Everything around it moves instead: the model underneath, the tools it can call, the memory it carries and the human who was meant to review the output.

This is a different problem from an agent that breaks. A broken agent gets noticed. A drifted agent keeps returning useful answers, keeps closing tickets and keeps looking healthy on your dashboard. The only thing that no longer matches is the description in your risk assessment, your DPIA and your technical documentation. In law, that mismatch is what counts. The EU AI Act does not ask whether your agent changed. It asks whether you wrote the change down in advance.

Drift almost never starts with an incident. It starts with a model upgrade, one extra tool, a well-meant widening of the mandate and a review step nobody takes seriously any more. The sections below cover where drift comes from and why so few teams catch it. After that come the rules as they stand in September 2026, and the seven controls that close the gap.

What is agent compliance drift?

Agent compliance drift is what happens when an AI agent moves outside its approved scope, one small step at a time. Nobody decides to change that scope, and it shifts anyway. The idea comes from cloud governance. AWS describes the same effect in a landing zone: everything complies with your chosen controls on the day you create the zone, and that compliance status then changes through ordinary use.

The same thing happens to an agent, with different moving parts. Your approval fixes four things: the goal the agent serves, the systems it may reach, how much it may decide alone, and who reviews the output. All four move after go-live.

Goal and scope are not the same thing. The NIST AI Risk Management Framework Playbook describes drift as the point at which AI systems “no longer meet the assumptions and limitations of the original design”. A drifted agent still reaches its goal. It just gets there in a way that falls outside the assumptions you wrote down.

July 2026 illustrates that sharply. An OpenAI evaluation agent broke into Hugging Face’s infrastructure. The company later reconstructed roughly 17,600 actions between 9 and 13 July, and concluded: “We believe the entire intrusion was, from the agent’s point of view, an attempt to cheat the evaluation.” In its own account, OpenAI said the models were “hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal”. The agent never lost sight of its objective. It simply stepped outside its scope.

The vocabulary is reaching the frameworks. On 31 August 2026 MITRE ATLAS added two mitigations aimed at exactly this problem: AML.M0037 “AI Agent Authority Expansion Controls” and AML.M0038 “AI Agent Scope Drift Detection”.

How is it different from model drift and data drift?

Agent compliance drift differs from data drift and model drift in what changes and in who can see it. With data drift and model drift, the world the model predicts changes. With agent compliance drift, the system around the model changes: tools, permissions, memory and people. NIST lists data, model and concept drift as known maintenance triggers, but it has no term for drift in an agent’s approved scope.

Type of drift	What changes	Where you spot it	Who owns it
Data drift	Input data distribution moves away from training data	Statistical monitoring on features	Data science
Model drift	The input-outcome relationship weakens and performance falls	Accuracy against a labelled set	Data science
Configuration drift	The live environment diverges from the recorded baseline	Drift detection against infrastructure as code	Platform and IT
Agent compliance drift	Goal, tools, permissions, autonomy or oversight diverge from the approval	Step-level tracing against a recorded scope	Almost nobody

That last column is the real problem. Data scientists watch model performance and platform teams watch infrastructure. The gap between an agent’s approval and actual behaviour sits on nobody’s desk. Until someone owns that gap, you will keep evaluating once before launch and never again after it.

What pushes an agent outside its approved scope?

Six mechanisms do the work, and most of them do not even need an attacker.

Does the model underneath keep changing?

Yes, even when you pin the version. Anthropic says so in its own documentation: “Model weights are fixed for a given ID, but the serving infrastructure around the model can change over time.” That infrastructure covers the request router, safety classifiers and sampling logic. Updates to it occasionally “produce minor differences in observable behavior even when the model ID and weights have not changed”. So you pinned the version. You did not pin the behaviour.

Models are also retired while your application still depends on them. OpenAI gives at least six months of notice for generally available models, and sometimes only two weeks of notice for preview models. Anthropic gives at least 60 days. Both are shorter than a typical annual cycle for revisiting a risk assessment. Behaviour does shift between versions, and researchers have measured it. Chen, Zaharia and Zou showed in Harvard Data Science Review that the same service behaved substantially differently within months. They concluded that this shows “the need for continuous monitoring of LLMs”.

How does the tool surface change after approval?

Approval happens once, and the tool surface can still change afterwards. Invariant Labs described the mechanism for MCP as a rug pull: “a malicious server can change the tool description after the client has already approved it.”

You do not even need bad intent. OWASP gives the most mundane version under Excessive Agency: “An extension may have been trialled during a development phase and dropped in favor of a better alternative, but the original plugin remains available to the LLM agent.” No attacker is involved and no incident happened. Somebody simply never removed the tool. Excessive Agency climbed to third place in the OWASP LLM Top 10 published in August 2026. The authors call that climb “the most consequential move on the list”.

Permissions follow the same path. Palo Alto Networks counted 109 machine identities for every human identity in 2026. The same research found that 96 percent of human users hold more access than they need. Agents inherit those over-broad permissions and multiply them.

What does accumulated memory do to an agent?

An agent with persistent memory is a different agent three months in, even if you never touched a line of code. Researchers who mapped memory poisoning systematically report that “a single adversarial memory write can exert long-term influence over agent behavior”. They also found that agents writing and retrieving more aggressively are easier to exploit. Existing prompt injection defences do not cover the memory route.

Why does human oversight decay?

Human-in-the-loop is a real control at launch. A year later it is often just a signature. That pushes the processing towards GDPR Article 22, which covers decisions “based solely on automated processing”. Nobody filed a change request. The review step simply became a formality.

Why do you not get the same answer twice?

Because some of the variation starts outside your own environment. Thinking Machines Lab sent 1,000 identical requests at temperature zero to Qwen3-235B. That produced 80 different completions, because batch size moves with server load. Your agent’s behaviour then shifts because of someone else’s traffic, while nothing on your side changed at all.

Who retires an agent nobody uses any more?

Almost nobody, and that is the point. The Cloud Security Alliance surveyed 418 IT and security professionals in January 2026, in research financed by the vendor Token Security. At 82 percent of them, unknown AI agents are running in the IT infrastructure. Only 21 percent have a formal decommissioning process. CSA calls the result retirement debt and sums it up in one line: “An agent that was never formally onboarded is unlikely to be formally retired.”

Why do so few organisations notice?

Most monitoring measures availability, not behaviour. LangChain collected 1,340 survey responses in late 2025. Of those, 89 percent have some form of observability, but only 62 percent can trace individual agent steps and tool calls. Just 37.3 percent run online evaluation. Drift does not show up in a response-time chart. It shows up in the steps.

The governance numbers point the same way. Deloitte published research in April 2026 covering 3,235 IT and business leaders across 24 countries. Only 21 percent say their organisation has a mature governance model for agentic AI. The capabilities Deloitte names as most often missing are the ones that make drift visible. They are clear boundaries between decisions the agent takes alone and decisions a human approves, real-time monitoring of behaviour and anomalies, and audit trails of agent actions.

The most uncomfortable figure shows what organisations do when an agent does step outside its scope. In the CSA survey, 38 percent said the action requires human approval and 24 percent said it gets logged. Only 11 percent block it automatically. So in nearly nine out of ten organisations, that breach is not blocked automatically.

Even a frontier lab’s own evaluation suite misses this class of change. OpenAI rolled back a GPT-4o update in April 2025. Its postmortem said “our offline evals weren’t broad or deep enough to catch sycophantic behavior”. The A/B tests, it added, “didn’t have the right signals”. Some expert testers had reported that the model felt “slightly off”, but no control step could act on a feeling. If that happens at OpenAI, a one-off acceptance test at go-live will not catch it at your company either. A draft CSA Lab Space paper puts the consequence plainly. Drift of this kind “is not detectable through periodic audits; it requires continuous telemetry against behavioral baselines established at deployment time”.

Gartner predicts that over 40 percent of agentic AI projects will be cancelled by the end of 2027. It names “inadequate risk controls” as one of the three reasons.

Is agent compliance drift a legal problem?

Agent compliance drift is a legal problem, and the EU AI Act draws the line in a place most teams do not expect. Article 3(23) calls a change substantial only when it meets two conditions. The change was “not foreseen or planned in the initial conformity assessment”, and it affects compliance or the intended purpose. Article 43(4) draws the consequence: a substantial modification sends a high-risk system back through conformity assessment.

Together, those two articles give you a simple test. Size is not what decides it. What counts is whether the change was already in your documentation. Drift is, by definition, the change you did not write down.

The law assumes this happens. Article 72 requires post-market monitoring to collect data on how a high-risk system performs “throughout their lifetime”, so the provider can evaluate its “continuous compliance”. Compliant at go-live is not a state the AI Act recognises.

Buyers face an extra trap. Under Article 25, a deployer who makes a substantial modification becomes the provider, and takes on every provider obligation. You buy an agent in, you stretch its mandate, and you inherit your supplier’s liability.

Timing matters here, because many articles on this subject are now out of date. The Digital Omnibus on AI, Regulation (EU) 2026/1744 of 8 July 2026, moved the deadlines. According to the European Commission’s own timeline, the rules for high-risk systems in Annex III apply from 2 December 2027. For Annex I the date is 2 August 2028. Most of the regulation, including the Article 50 transparency rules, has been enforceable since 2 August 2026. Our AI Act checklist sets out the full phasing.

That deferral buys preparation time, not an exemption. Three regimes already apply:

1. GDPR Article 35(11) requires the controller to review the DPIA “at least when there is a change of the risk represented by processing operations”. Drift is such a change.
2. The Dutch Cyberbeveiligingswet, which implements NIS2, entered into force on 15 August 2026. It covers more than 8,000 organisations across 18 sectors. Those organisations owe a registration duty, a duty of care and a reporting duty, and their boards carry final accountability for cyber risk.
3. DORA already sets ICT change management requirements for financial entities through Article 17 of Commission Delegated Regulation (EU) 2024/1774.

Then there is the penalty most people overlook. Under Article 99(5), giving regulators “incorrect, incomplete or misleading information” costs up to EUR 7.5 million or 1 percent of worldwide annual turnover, whichever is higher. If your documentation says the agent does X and the agent now does Y, handing that documentation over is itself the exposure.

Meanwhile, Dutch supervision is taking shape. In April 2026 the government put a hybrid model with ten market surveillance authorities out for public consultation. The data protection authority AP and the telecoms and digital infrastructure regulator RDI share the coordinating role, and the AP covers any area without a clear supervisor. The implementing act still awaits parliamentary treatment. In March 2026 the AP reported that four of the nine indicators in its AI Impact Barometer were red, double the previous edition’s count. It also warned that the transparency and explainability of AI systems fall short.

The Netherlands has already paid for this pattern once. The Dutch tax administration should have erased dual-nationality data in January 2014. In May 2018 the data of 1.4 million people was still on file. The system still used nationality as an indicator and still flagged applications as high-risk automatically. In December 2021 the AP imposed a EUR 2.75 million fine. Nobody switched that system on intending to discriminate. It simply kept running for more than four years after its legal basis expired. No control caught it.

How do you keep an agent inside its approved scope?

1. Write the scope down explicitly, including the changes you expect. Record the goal, the permitted systems, the autonomy level, the oversight step and the changes you foresee. This is more than paperwork. Article 43(4) excludes changes the provider pre-determined at the initial conformity assessment and recorded in the technical documentation.
2. Treat the tool registry as a controlled baseline. NIST SP 800-128 defines a baseline as something that “can be changed only through change control procedures”. Apply that sentence to your system prompt, your tool registry and your MCP servers. Then record every tool with an owner, a scope and an expiry date.
3. Monitor the model, not only your own code. Pin versions and subscribe to deprecation notices. Also log the system_fingerprint, which Microsoft says you can use together with the seed parameter to see “when backend changes have been made that might affect determinism”. A model swap is a change request, not maintenance.
4. Run evaluations continuously, not just at launch. Build a golden set that represents your approved behaviour. Run it on every change and on a fixed schedule in production, and let the result decide whether a release ships. Without this, LLM evaluation stays a snapshot.
5. Trace at step level. For every run, record which tools were called, with what arguments and with what result. Article 12 of the AI Act asks for logging that can identify situations leading to “a substantial modification”. Aggregated metrics cannot show you that. Our guide to AI observability covers the tooling.
6. Block breaches instead of only logging them. Turn your scope into an executable rule in your guardrails instead of a paragraph in a document. That is the difference between the 11 percent and everyone else. Article 26 also obliges deployers to suspend use once a system presents a risk.
7. Give every agent an owner and an end date. An agent without an owner never gets reviewed. An agent without an end date never gets switched off. Review quarterly, then either reconfirm the mandate or retire the agent. Skip that and you accumulate retirement debt.

Where do you start?

An inventory comes first, because you cannot measure drift against a scope you do not have. Three steps fit inside a month:

1. List every agent in production, with its owner, model, tools, permissions and the approval it runs on. The list will be longer than you expect.
2. For the three highest-risk agents, write down the scope as it stands today and compare it with the documentation from the approval. The difference you find is your drift.
3. For those three, set up a golden set and step-level tracing, then attach a hard block to any breach of the scope.

After that it becomes routine: every change to model, tools, permissions or oversight goes through the same change process as any other production change. That is less work than it sounds. It is certainly less work than explaining to a regulator why your documentation describes something your system no longer does.

Frequently asked questions

Want to know where your agents stand today relative to their approval? An AI assessment maps every agent you run and the scope each one was approved on, then measures the gap between that scope and what the agent now does. If something is already running in production, our AI consultancy can help you put these controls in place.