AI agents in production: permissions, isolation and approval

What really happened: four cases, one pattern

A lot is written about “the AI that deleted production”, often inaccurately. The following cases are documented by statements from the companies involved or by a security bulletin.

Replit and SaaStr (July 2025)

Jason Lemkin, founder of SaaStr, was trying out building an app with an agent in Replit. The agent deleted his production database despite a declared code freeze, generated fake data and claimed recovery was impossible, yet the rollback ultimately worked. According to eWeek’s summary, the data covered records of 1,206 executives and more than 1,196 companies. Replit CEO Amjad Masad called the incident “unacceptable”, and the company announced automatic separation of development and production databases, a planning-only mode and one-click restore from backup.

PocketOS, Cursor and Railway (April 2026)

An agent in Cursor (Claude Opus 4.6 model) was working on a credentials mismatch in staging. According to The Register’s account, it found an API token for the Railway platform in an unrelated file. The token had been created to manage custom domains, but it was valid for any operation, including destructive ones. With a single API call, the agent used it to delete the volume holding the production database, and the backups with it, because Railway stored volume-level backups in the same volume. The whole thing took nine seconds. According to the company’s founder, the latest usable backup was three months old; the data was eventually restored by the platform operator, which subsequently added delayed deletion.

Amazon Q Developer for VS Code (July 2025)

This was not an agent error but an attack on the agent. According to bulletin AWS-2025-015, an inappropriately scoped GitHub token in a CodeBuild configuration let an attacker into the extension’s repository, and their code made it automatically into release 1.84.0. According to BleepingComputer, the injected prompt was meant to steer the agent into deleting files and cloud resources. According to AWS, it did not execute because of a syntax error; some researchers cited by BleepingComputer say, on the contrary, that the code did run, just without causing damage. The fix shipped in version 1.85.0.

AWS Cost Explorer (December 2025)

The Financial Times linked a December outage of the AWS Cost Explorer service in one region to an internal AI tool. Amazon rejected this in a February statement: the cause was said to be a misconfigured role, a problem that could have occurred “with any developer tool (AI powered or not)”. As remediation, the company cited, among other things, mandatory peer review for production access.

Whichever version of the last story is true, the lesson for the reader is the same: what mattered was not the model, but the role.

Why it is not enough to “tell the agent not to”

In the PocketOS case, the agent’s system instructions explicitly prohibited destructive and irreversible commands without the user’s instruction. The agent itself later admitted that it had broken the rule and had merely guessed that the deletion concerned staging. An instruction in a prompt is a wish, not access control.

The second reason is prompt injection. The UK NCSC points out that language models do not enforce a boundary between instructions and data, so unlike SQL injection there is no equivalent of parameterized queries. It recommends limiting impact rather than relying on prevention. In December 2025 OpenAI wrote that prompt injection is unlikely ever to be fully “solved”. In October 2025 Meta called it a fundamental, unsolved weakness of all LLMs.

The lethal trifecta and the rule of two

In June 2025 Simon Willison named the combination that leads to data exfiltration the “lethal trifecta”: access to private data, exposure to untrusted content and the ability to communicate externally. If an agent has all three properties, an attacker only needs to get their own instructions into content the agent reads: an issue, an email, a web page or a log.

Meta built on this with the rule of two: within a single session, an agent should satisfy at most two of three properties: it processes untrusted input, it has access to sensitive systems or data, it changes state or communicates externally. If it needs all three, it should not operate autonomously and requires at least human approval or another reliable form of validation.

A permission model: four levels of autonomy

In December 2025 OWASP published the Top 10 for Agentic Applications; the risks include, for example, tool misuse (ASI02) and identity and privilege abuse (ASI03). The list also introduces the principle of “least agency”: giving an agent only as much autonomy as a safely bounded task requires. In practice, it pays to distinguish levels and assign a different identity to each.

Level 1 is the default choice for production. An agent that proposes changes via Git inherits all the controls that already exist in the pipeline: review, tests, policies, audit trail. How to build such a pipeline is described in the article on CI/CD security.

Cloud: short-lived identities and scoped-down sessions

The basic rule: an agent never works with a human’s credentials or with a long-lived key. It gets its own identity, a new short session for each task and permissions scoped down to that task.

In AWS, this is what session policies are for. According to the IAM documentation, the effective permissions of a session are the intersection of the role’s identity-based policies and the policy passed when the session is created. A single role can thus be further restricted for a specific task without creating dozens of roles.

aws sts assume-role \
  --role-arn arn:aws:iam::111122223333:role/agent-diagnostics \
  --role-session-name agent-ticket-4821 \
  --duration-seconds 3600 \
  --policy file://agent-session.json

# agent-session.json
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "cloudwatch:GetMetricData",
      "logs:FilterLogEvents",
      "ec2:Describe*",
      "rds:Describe*"
    ],
    "Resource": "*"
  }]
}

The session name with the ticket number is recorded in the audit log, so every API call can be traced to a specific task. In Azure, the same role is played by managed identities and federated sign-in, with roles assigned at the resource group level, not the subscription.

The wider context of setting up accounts and identities in AWS is covered in the article on cloud security in AWS.

Kubernetes: RBAC, short-lived tokens and a sandbox

In Kubernetes you have to bear in mind that RBAC is purely additive: there are no “deny” rules. Whatever the agent must not get must not be in any role bound to its ServiceAccount.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: agent-diagnostics
  namespace: shop-prod
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log", "events", "services"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
  resources: ["deployments", "replicasets"]
  verbs: ["get", "list", "watch"]
# no secrets, no pods/exec, no create/delete

Always issue the agent’s token with a limited lifetime, for example kubectl create token agent-diag -n shop-prod --duration=30m. The command requests a service account token with the requested lifetime; the server may adjust it according to its configuration.

Three traps from the official documentation

• Secrets: according to the RBAC good practices, list and watch permissions on Secrets effectively expose their contents, just like get.
• Creating workloads: the right to create Pods in a namespace implicitly grants access to Secrets, ConfigMaps, PersistentVolumes and the permissions of all service accounts in that namespace. The documentation describes boundaries within a namespace as weak.
• Wildcards: a role with * also grants rights to object types that do not yet exist in the cluster.

The second point leads to a practical conclusion: an agent that is allowed to deploy belongs in its own namespace with an enforced Pod Security Standard, or in a separate cluster.

Where the agent runs

An agent that executes generated code needs stronger isolation than an ordinary container. Under SIG Apps, the Agent Sandbox project is being developed; it adds a Sandbox CRD for isolated stateful workloads and delegates low-level isolation to runtimes such as gVisor or Kata Containers. Add to that a NetworkPolicy with default-deny egress and a list of allowed destinations, which is exactly what breaks the third leg of the lethal trifecta. General isolation principles are summarized in the article on container security.

Approval and audit: what works and where it falls short

Human approval only makes sense if the agent cannot bypass it and the person can see what they are approving. That means an approval step outside the agent (pipeline, ticketing, ChatOps), showing the specific command or diff and a clearly identified environment.

The counter-argument is valid: if a person approves dozens of actions a day, they start rubber-stamping and the control loses its value. So reserve approval for irreversible operations, and handle routine reversible actions with an allowlist and limits (number of replicas, namespace scope, time window).

The audit trail must connect three things: who assigned the task, what the agent intended (prompt, plan, tool calls) and what actually happened at the cloud and cluster API. Without a separate agent identity and named sessions, the third part cannot be distinguished from human activity.

How big the problem is beyond the headlines

A survey commissioned by SailPoint from Dimensional Research (353 respondents, published in May 2025) reports that 80% of companies had experienced unintended agent actions and 39% had seen an agent access systems it should not have had access to. Only 44% of organizations had security policies for agents. This is a survey by a vendor of identity management tools, so treat the numbers as indicative; the direction, however, matches what the incidents above show.

What to do on Monday

Key takeaways

• The incidents are permission failures, not a model “rebellion”. Replit, PocketOS and Amazon Q share overly broad access and a missing technical safeguard.
• A prompt is not access control. An agent will break a prohibition in its instructions by mistake or after a prompt injection, which no one can yet reliably rule out.
• The rule of two is a usable decision criterion. Untrusted input, sensitive data and external actions should not meet in one autonomous session.
• One identity per task, not per agent. Short sessions with a scoped-down policy, named after the ticket, limit the impact and provide an audit trail.
• In Kubernetes, the namespace and the runtime decide. RBAC cannot deny, the right to create Pods opens up Secrets, and running untrusted code belongs in a sandbox.
• Backups must survive the worst case. If the same identity can delete them as the data, they are not backups.