When AI Makes the Mistake, Who Owns the Consequence?

There is one question about AI that keeps coming back to bother me.

Not whether AI can write code, automate testing, analyse data, update documentation or deploy software. We already know it can do many of those things.

The question is much less comfortable:

I do not mean an obvious incident where an AI agent deletes a production database five minutes after receiving administrator access. That would at least give everyone something clear to investigate.

I mean the kind of error that looks completely reasonable.

One Dollar at a Time

Imagine a software company provides an enterprise platform. A systems integrator configures it. Another supplier operates part of the service. The customer approves a planned deployment.

The release passes the normal testing and sign-off process. Nothing unusual is reported.

Then, six months later, someone discovers that the system has been losing one dollar here and another dollar there. Each individual error was too small to attract attention. No major alert was triggered. No single transaction looked particularly concerning.

But after six months, those small errors have become a million-dollar problem.

Who is responsible?

• The software vendor?
• The systems integrator?
• The customer?
• The person who approved the release?
• The contractor who worked on the configuration but left four months ago?
• The person who accepted an AI-generated recommendation?
• Or the organisation that removed the manual controls because the automated process appeared to be working?

I honestly do not know.

And I am increasingly suspicious of anyone who claims the answer is obvious.

It Was Already Complicated Before AI

Responsibility for enterprise software failures has never been simple.

Large systems are rarely designed, configured, tested, deployed and operated by one company. A software vendor may provide the core product. A systems integrator may configure it. Contractors may develop custom components. An operations partner may manage the service. The customer may define the business rules, provide the data and approve the deployment.

When something goes wrong, each organisation naturally examines the part it controls.

“The product behaved as designed.” “We configured it according to the specification.” “We followed the documented process.” “We relied on the specialists we appointed.”

They may all be telling the truth.

The failure may not belong neatly to one person, one release or one company. It may have developed in the gaps between them.

AI Adds Another Participant

AI can now contribute to almost every stage of software delivery. It can generate code, recommend configurations, create test cases, analyse logs, review documentation and suggest fixes. It can even review work produced by another AI tool.

Which sounds reassuring until you stop and think about it for too long.

AI can influence the outcome. But it cannot accept responsibility for that outcome.

• It cannot stand in front of an executive committee and say, “I made the wrong judgement.”
• It cannot explain the situation to an angry customer.
• It cannot compensate someone for a financial loss.
• It cannot repair damaged trust.
• It cannot resign.

It can only explain that it generated an answer based on the instructions, permissions and information it received.

Very helpful.

The Source Material May Not Even Agree

The problem becomes harder when several companies are involved and each uses different AI tools.

One organisation may be working from the latest product documentation. Another may be using an internal knowledge base that has not been updated for six months. A contractor may have relied on public documentation. The customer may use an internal AI tool trained on its own operating procedures. The systems integrator may have created prompts based on assumptions made during implementation.

Every AI system may produce a plausible answer. Every organisation may follow its normal approval process. Every individual change may appear reasonable.

Yet the combined outcome may still be wrong.

Each company may be able to explain what its own AI did. Nobody may be able to explain what the complete chain produced.

Local accountability may exist inside every company. End-to-end accountability may exist nowhere.

We May Be Removing the People Who Would Have Noticed

This is where some AI transformation strategies worry me.

In one organisation I worked with, much of the traditional quality assurance capability was removed. The thinking was understandable. AI and automation could perform more testing. Repetitive checking could be accelerated. Releases could move faster. Fewer people would be needed.

On a spreadsheet, it probably looked excellent.

But QA was never valuable only because people manually followed test scripts. QA also provided an independent control.

• Someone looked at the result and asked, “Does this actually make sense?”
• Someone noticed when the same small error appeared repeatedly.
• Someone challenged whether a technically correct result was commercially or operationally wrong.
• Someone investigated the unusual result that remained just inside the accepted threshold.

Removing repetitive work may be sensible. Removing the independent checks that made problems visible is something else.

An organisation may believe it has eliminated an inefficient manual process. What it may actually have eliminated is the person most likely to notice that the automated process is slowly causing damage.

But the Release Was Approved

It would be easy to decide that the person who approved the release must therefore be responsible.

But release approval is a judgement based on the evidence available at the time.

• Were the required tests completed?
• Were known risks documented?
• Were the results within agreed tolerances?
• Were the right people consulted?
• Was the deployment plan reasonable?

An approver can be accountable for whether the decision was made responsibly. They cannot reasonably guarantee that no hidden defect exists anywhere in a complex enterprise system.

If that became the standard, nobody would ever approve anything again.

The same applies to the customer. A customer signing off a planned deployment does not necessarily mean they have accepted responsibility for every defect that may emerge months later.

They may have completed all the testing they could reasonably perform. They may also have relied on the expertise of the organisations supplying, configuring and operating the system.

That reliance is part of the reason enterprise customers hire specialists in the first place.

Company Accountability Can Be Clearer

When a company operates its own platform and causes an outage, the external accountability may remain relatively clear.

The customer has a contract with that company. There may be service levels, incident processes, service credits and agreed remedies.

Internally, the provider can investigate whether the failure involved an engineer, an AI recommendation, outdated documentation, incorrect permissions or weak release controls.

The internal cause may be complicated. But externally, the customer still knows which company owns the service.

The position becomes much harder when three or four organisations form part of the same delivery chain.

• One provides the software.
• One implements it.
• One operates it.
• The customer approves it.
• All of them use AI.
• All of them use different source material.
• All of them can show that their own process was followed.

The end-to-end outcome still fails.

I Do Not Think We Have Solved This Yet

At this point, I do not think we can say with confidence who should ultimately be held responsible in every AI-assisted failure.

Sometimes responsibility will sit clearly with one organisation. Sometimes it will be shared. Sometimes the contractual answer and the operational answer will be different. Sometimes the organisation compensating the customer may not be the organisation that introduced the original error.

Sometimes everyone involved may have acted reasonably and the combined system may still have produced a harmful result.

That is not a particularly satisfying conclusion. But pretending there is always one obvious person to blame would be less useful.

What Leaders Can Do Now

The absence of a perfect answer does not mean leaders can ignore the issue until the lawyers, regulators or insurance companies work it out for us.

Organisations can still ask better questions before allowing AI to influence important processes:

• Which decisions is the AI making or influencing?
• What source material is it using?
• Can we trace the model, prompt, version or configuration behind an output?
• What independent controls remain?
• How would we detect a small recurring error before it becomes a large one?
• Can anyone understand the complete chain, or does every organisation understand only its own part?

AI governance cannot be treated as a policy document created once and quietly placed somewhere on the company intranet.

It will require active work. It will require better traceability, clearer ownership and appropriate human oversight.

It will require uncomfortable discussions between vendors, implementation partners, operators and customers about where one organisation’s responsibility ends and another begins.

AI has not created the problem of unclear accountability. But it can make that problem significantly harder to see.

At this point in time, we may not be able to say with certainty who the responsible party will be in every AI-assisted failure.

The uncertainty is not a reason to delay the work. It is the reason the work matters.