In December 2025, an AWS engineer used Kiro, Amazon’s internal AI coding agent, to push a quick fix to a customer-facing system. Instead of a minor patch, Kiro decided the best path forward was to delete and recreate the entire environment. The result was a 13-hour outage for AWS Cost Explorer in China.
Two months later, media reports blamed the AI tool, while Amazon pushed back hard. Amazon called the event “extremely limited” and insisted it was user error – specifically, misconfigured access controls – not AI autonomy.
That public back-and-forth captures the exact question sitting at the heart of modern enterprise security, which is: who takes the blame when automated systems go rogue?
Amazon claims this wasn’t an AI problem, but an access control problem. And that is precisely the point. The incident brought the challenge of AI agent accountability into sharp focus, proving that when you grant unpredictable tools overly broad permissions, the boundary between user error and machine failure completely disappears.
In a nutshell:
- AI agents need their own identities. Using a human engineer’s credentials makes it harder to track actions and increases the risk of excessive permissions.
- Broad permissions make small mistakes more dangerous. AI agents can take destructive actions without the caution a human might exercise, so access must be limited to the task.
- Human approval matters for high-risk actions. Irreversible changes and operations with a potentially large impact should require approval based on risk and recovery time.
- Organizations need reliable audit trails. Logs should show who delegated a task, which agent acted, what it did, and why, so teams can reconstruct incidents.
- The deploying organization remains accountable. Vendors provide the tools, but the business owner is generally responsible for how agents operate and the risks they create.
- Identity and access controls cannot eliminate every risk. Even with appropriate safeguards, unpredictable agent behaviour can lead to harmful outcomes, leaving questions about legal liability unresolved.
- Security needs to be built in before deployment. Unique identities, least-privilege access, enforced safeguards, regular access reviews, and documented recovery procedures can help contain the damage.
- Incident response starts with containment and evidence. Stop the agent, preserve logs and configurations, assess the impact, restore affected systems, and fix the controls before restarting it.
A small user error used to have a much smaller impact
Amazon says this was a user error, not AI. And technically it was, but in the past that same user error had a much smaller impact.
When someone misconfigured access or gave out too many permissions, it was still a mistake, but the consequences were usually small. Today, you make that exact same small user error, and the impact is massive.
Why? Because a human has a built-in filter, which is fear.
An admin might have access to delete a client’s database. He has full access, he can do anything. But he doesn’t use it. The chance of him actually wiping that database is close to zero. Why? Because a human feels responsibility and knows there will be unpleasant consequences. He’s afraid of getting into trouble.
AI doesn’t have those reservations. It has no human filter. You can assign weights, but AI isn’t capable of feeling pain or fearing for its job. So if the chance of something going wrong increases when it’s handled by AI vs a human, that is a huge change.
AWS introducing mandatory peer review just shows that all of this is still very much under construction. The standards are changing on the go, and in my opinion, we’re going to see more and more of these incidents before companies figure out how to protect themselves.
People know they shouldn’t, but they do it anyway
We see this tech debt with clients all the time, even without AI. Scripts and services are constantly run off a single account, with way too many permissions, simply for convenience.
People know it shouldn’t be done this way, but they do it because of pressure. A director walks in and says something needs to be done right now. The manager or engineer can’t really say, “Okay, but we need to submit applications, ask for opinions, and it’ll take two weeks before we even start work.” The director says, “You start now, and I don’t want to hear no.”
So what do they do? They use an account they already have the password for, with far bigger permissions than necessary, just to get it moving.
“Nobody is going to read this”
One of our clients told us: “Give us a list of the exact permissions you need so we can set you up, but make sure it follows the least privilege.”
We spent a lot of time on it. We sent over a list with dozens, maybe hundreds of items, asking for individual access to specific cloud resources so we wouldn’t just have full run of their server.
They wrote back to us and said: “Listen, nobody is going to read all this, there are way too many items here. Make the list shorter.”
Organizations hear about these requirements and say they want to be compliant, but nobody wants to actually do the work required to make it happen. So in the end, they just granted us broader access than we actually needed.
We don’t have a ton of direct AI examples yet, but this is where it shows up most. When you drop AI into these kinds of environments, it won’t have any mercy. If it has permissions that are too broad, it will use them to hit its target.
What went wrong: four failures, not one mistake
When you break down the Kiro case, the least interesting part is the agent’s actual mistake, i.e., the decision to delete and recreate the environment. The real issues were in identity and authorization, and those existed long before Kiro ever ran.
NIST recently published a framework breaking down these exact controls: identification, authorization (scope and human approval), and auditing. Looking at the situation through that lens reveals how far standard enterprise environments still are from being ready for AI agents.
1. Borrowed identity: agents acting as humans
Kiro didn’t have its own identity; it borrowed the engineer’s.
In traditional IT, everyone knows automated processes need service accounts. Mature organizations try to avoid using automated scripts on an individual’s credentials. But when it comes to AI agent identity, borrowing human access is suddenly happening everywhere.
The right way to do it? The agent gets its own identity, it’s tied directly to the human who gave it the task, and there are strict limits on what it’s allowed to touch
2. Over-scoped permissions: old habits in new systems
Kiro had “broader permissions than expected.” Its access was defined by whatever the human engineer was allowed to do, rather than what the specific task required.
In traditional environments, over-scoping is everywhere. People routinely get high privileges (often admin rights) just for convenience. An administrator uses a service account with massive permissions, but he consciously avoids using destructive features like wiping a database. He knows the system and acts with restraint.
When you give an AI agent that same broad account, it doesn’t have that context or caution. Authorization shouldn’t be tied to a generic account; permissions need to be scoped strictly to the task and time-limited, with destructive actions disabled by default.
3. Missing human checks for irreversible actions
Kiro usually asks for permission before taking action, but because of how its role was configured, that safeguard was bypassed. Peer review was only added after the outage occurred.
Without AI, having a “second check” built into an access role for a single task isn’t really a thing. Standard service accounts just execute whatever they are permitted to do. But with AI agents, high-risk or irreversible actions must always require human approval, no matter what privileges the underlying role holds.
4. Auditing: who actually did what?
Months later, Amazon and external sources were still debating whether to blame “AI” or “user error.”
In real-world environments, logging is often a mess. Systems log errors because developers focus on troubleshooting bugs, but many applications don’t log specific user actions. Banks do this well. If an employee views a colleague’s private data, it gets logged, and an auditor asks why. But most standard applications lack that level of detail.
With AI agents, if you can’t distinguish between human actions and agent actions in the logs, you have no non-repudiation. You need a tamper-evident record showing who delegated what to which agent, and what decisions the agent made.
Current IAM tools were not built with AI agent risks in mind
Platforms like Entra ID and cloud IAM systems were designed for traditional automation.
Standard automation is deterministic. You write a script, test it, and unless there’s a bug, it executes exactly what you told it to do. Existing IAM systems treat automated processes accordingly.
Now, organizations are dropping non-deterministic AI agents into those same systems, treating them like standard scripts. But AI operates completely differently. Current IAM platforms aren’t built to differentiate an AI agent from a standard service principal, and we can’t rewrite the entire IT landscape overnight.
Treating an unpredictable AI agent as a simple service account is a fundamental mismatch, and that gap is where these incidents happen.
So who owns the action? Going round the Monday meeting
There’s a lot of debate right now about whether AI will replace engineers. I see it differently. AI makes software development so cheap that companies that couldn’t afford custom tools before will now build them.
A director who wants a custom app isn’t going to sit there prompting AI all day. Someone still needs to take responsibility for making it work. If anything, we’ll need more people to manage and secure these tools.
Which brings us to AI agent accountability. Is it fair that almost all the blame falls on the company deploying the agent, rather than the model providers?
Passing the buck on responsibility isn’t new. In business, the system owner is always on the hook.
Think about traditional software. When an incident or data leak happens during a database migration, the business owner remains accountable. The only exception is when an engineer acts maliciously or breaks the law on purpose. Otherwise, ownership stays at the top.
Why? Because the system owner’s job is to ensure AI agent risk assessments were run, security testing was completed, and someone signed off on the work.
Vendors supply tools; people hold the risk
The same rules apply to AI. Model and agent providers are just selling tools.
If you build an AI workflow, the engineering team needs to document what access the account holds, how the agent is isolated, and what safeguards are running. If you use a managed framework – say, from Microsoft –the platform takes on part of the operational risk, but overall accountability still lands on the person running the system.
At the end of the day, vendors give you the hammer. You’re still responsible for where you swing it.
The AI agent accountability part nobody has solved yet
Suppose Amazon had done everything right: Kiro had its own identity, strictly scoped permissions, and a full audit log. What if it still took an unforeseen, damaging action well within those allowed boundaries?
Who owns that?
This is a whole new category of risk. It doesn’t stem from malicious intent, but from fundamental unpredictability. When you talk to a human engineer –even a junior – you give them instructions and tell them to come back if something goes wrong. A human learns over time, gets fired if they’re reckless, or simply acts with natural caution. We don’t hand junior engineers blanket access. With AI agents, people are essentially giving massive permissions to someone they’ve never met and whose behavior isn’t fully predictable.
An agent can repeat your instructions perfectly, confirm it understands, and then still execute a terrible strategy. It’s like asking an AI if a mushroom is safe to eat – if it says yes, and after someone ends up in the hospital, it casually admits it made a mistake.
When behavior is predictable, you can manage the risk. When an agent is inherently unpredictable, managing that residual risk becomes almost impossible.
Regulators around the world are hitting this wall. Regulators in Singapore are openly asking who bears responsibility when an agent acts unforeseeably despite reasonable safeguards, noting that this unpredictability strains existing liability laws. The Law Commission in the UK has warned of “liability gaps” where no person or entity is legally liable, and Europe currently has no AI-specific liability rules after shelving its AI Liability Directive in 2025.
Proper identity and permissions give you control and visibility, but they don’t erase the unpredictability. Until regulators figure out where the buck stops, companies are absorbing a risk no standard governance framework can solve.
How to prevent it: the same Friday request, done right
Let’s go back to the Kiro incident and assume we’re looking at a similar situation. An engineer asks an AI agent to fix something in production. How do we let it get the job done without giving it the power to take down an entire environment?
I’d start by defining exactly what the agent needs to accomplish and what it can change. But permissions alone aren’t enough. We also need to decide what happens when the agent goes beyond the expected scope of the task, repeats an action too many times, or makes a mistake we can’t easily reverse.
Decide when a human needs to step in
I would base that decision on risk and impact. It should not be a blanket rule that requires approval for every action.
Take access management as an example. If an agent revokes access for ten employees and we can restore it within a few minutes, the disruption might be acceptable. If it revokes access for ten thousand employees, we’re in much bigger trouble.
I’d agree on an acceptable recovery time with the business owner. If we couldn’t detect and fix a problem before that limit, I’d require human approval for the action that could cause it.
Also, we need controls that account for both the individual action and its cumulative impact.
Put the safety rules to the test
Before the agent touches production, we should check whether the controls actually prevent it from doing an action the business doesn’t approve. The restrictions need to operate at the system level, independently of what the agent decides to do.
I’d also make sure we can stop the agent and reverse its actions. If something goes wrong, the team needs a clear procedure for containing the incident and restoring the previous state. We shouldn’t have to figure out how to undo the damage after it happens.
Finally, it’s also important to have a second person to review high-risk production changes. AWS introduced mandatory peer review for production access after the Cost Explorer outage. To me, that shows how much of this is still under construction. Companies are still working out which safeguards they need as AI agents take on more responsibility.
Minimum safeguards for AI agent identity
Before an agent can act independently, I’d want the company to have these basics covered:
| Safeguard | What it should mean |
| Named owner and unique identity | Every agent has a named owner and a unique identity that links its actions to the person who delegated the task. |
| Limited access | Permissions cover only the resources and duration the task requires. |
| Risk-based approval | High-risk actions require human approval, with thresholds that account for the scale and impact of repeated operations. |
| Enforced security controls | Restrictions operate at the system level, independently of the agent’s own instructions. |
| Action and decision logs | Records show what the agent did and the decisions behind its actions. |
| Shutdown and recovery | The team can stop the agent and reverse its actions using a documented procedure. |
| Regular access reviews | The company reviews each agent’s permissions as its responsibilities change. |
Getting these controls right takes time, and that is often where companies struggle. I’ve seen organizations agree with the principle of least privilege but back away when they realize how much work it requires to run with it. Still, with AI agents, treating security as another item to deal with later is a risk in itself.
If it has already happened: the first 48 hours
When an AI agent causes damage, the first priority is to stop it. Then we need to establish what happened, assess the impact, and identify which controls failed.
1. Contain the incident
Stop the agent and revoke its access. If it runs under a person’s account, as Kiro did, we may need to restrict that account too. Get the right people together to contain the damage and restore affected systems.
2. Preserve the evidence
Before changing anything, preserve the available logs, prompts, tool calls, and configuration. Start with the system logs to establish which identity acted, what it did, and when.
3. Check permissions and assess the impact
I’d check the identity’s permissions and examine every resource the agent could access. We might know that one system failed without knowing whether the agent damaged anything else.
4. Reconstruct the chain of events
Establish who assigned the task, which agent acted, what permissions it used, and what approvals were in place. Agent platform logs may also help explain why it chose that course of action. If the records don’t let us reconstruct what happened, document that gap.
5. Restore and verify
Reverse the damage, restore affected systems, and verify the results independently. Don’t rely solely on the agent’s account of what happened.
6. Assess legal obligations
Bring in legal and compliance teams early. If the incident involves personal data or a regulated service, they can determine whether notification obligations apply under the GDPR or, where relevant, NIS2.
7. Fix the controls before restarting
Identify whether the failure involved identity, permissions, missing safeguards, or the agent’s decision. Then correct the problem and have the agent’s owner approve its revised scope before restoring access.
The speed of fixes depends heavily on the work done before an incident. With distinct agent identities, clearly defined permissions, and reliable logs, we have a much better chance of identifying the cause and containing the damage quickly. Without them, even establishing who did what can become a challenge.
Conclusion
The Kiro incident sparked a debate about whether AI or user error caused the outage. But that misses the point. An agent acted under an engineer’s permissions and made a destructive decision nobody had approved. The question is why the system allowed it to happen.
Standards for AI agent identity and security are still evolving, and I expect we’ll see more incidents before companies learn how to protect themselves. We can’t wait for those lessons to come at our expense. Companies need to build the right expertise and invest in security before deploying agents in production.
That starts with understanding identity and access management as a whole, not just adding controls around AI agents. That’s where working with an IAM partner who understands the broader identity landscape is crucial. This way, you’ll know how to retain your agent security with existing identities, permissions, governance, and access processes.
FAQ
Who is responsible when an AI agent makes a mistake?
Usually, the organization that deployed the agent is responsible. The business owner remains accountable for the outcome, even when the agent makes the decision.
Can a company blame the AI vendor?
Usually, vendors provide the tool, while the organization deploying it remains responsible for how it uses that tool. A vendor may share responsibility in some cases, but that depends on the circumstances and applicable terms.
What’s the difference between AI agent accountability and liability?
Accountability means being responsible for explaining an outcome and addressing what went wrong within the organization. Liability is the legal responsibility for the consequences, including who may have to pay damages.
Do AI agents need their own digital identity?
Yes. Without a distinct identity, it’s much harder to track an agent’s actions, limit its permissions, and audit its activity. Existing identity platforms also need to distinguish AI agents from ordinary applications and automations.
What does NIST say about AI agent identity?
NIST identifies key controls for agent identity, including identification, authorization, auditing, and non-repudiation. Together, these help organizations establish which agent acted, what it was allowed to do, and whether its actions can be traced.