The AI Question Nobody Wants to Own

One of the distinctive features of AI systems in complex organizations is how effectively they diffuse accountability. A traditional software system has an owner: a team that built it, a manager who approved it, an executive who signed off on it. When something goes wrong, there is a relatively clear line of responsibility. AI systems, especially those built on third-party models or deployed through layered vendor relationships, make that line much harder to trace.

Consider a hospital that deploys an AI diagnostic support tool built by a startup that licenses a foundation model from a large tech company. When the tool produces a bad recommendation that contributes to a patient harm, who is responsible? The hospital, for the deployment decision? The startup, for the product design? The foundation model company, for the underlying capability? The physician who followed the AI's suggestion? The insurer who incentivized the cost-cutting decision that led to the AI deployment in the first place? Each party can point somewhere else, and often does. In that environment, the question of who has authority to shut the system down is entangled with the question of who bears responsibility for its harms, and nobody wants to own either one.

This diffusion is not always intentional, but it is often convenient. Organizations benefit from ambiguity about AI accountability because ambiguity protects them from being the party that bears costs when something goes wrong. The result is governance structures where everyone has influence over AI systems and nobody has clear authority over them, especially not the authority to shut them down at the cost of short-term disruption.

There is actually more than one kind of shutdown authority, and they create different problems. Operational shutdown authority is the ability to turn the system off right now, in response to an immediate harm. Strategic shutdown authority is the ability to decide that a system should not be running at all, even if it is currently producing acceptable outputs. Most organizations handle the first reasonably well, at least in principle. The second is where things get complicated.

A system can be technically running fine and still be causing harm at a slower, less visible rate. Predictive policing tools, automated content moderation, credit scoring algorithms, and hiring filters have all been documented producing racially or socioeconomically biased outcomes in ways that are statistically clear but operationally invisible in any given instance. Nobody experiences a system failure. The system is working exactly as designed. The question of whether it should continue running is not about operational performance but about values, about whether the organization is willing to accept the harms it is producing in exchange for the efficiency it is gaining.

That is a strategic question, and it requires strategic authority. In practice, that authority often does not exist in a usable form. The people with technical access to turn the system off are not the people who made the strategic decision to deploy it. The people who made the strategic decision are insulated from the operational reality of its harms by layers of reporting and aggregation. And the people who can see the harms most clearly are often the ones with the least institutional power to act on them.

The organizations getting this right are the ones that treat shutdown authority as a governance design question, not an emergency response question. They decide before deployment who has the authority to turn the system off, under what conditions, and what process they need to follow. They document those decisions and make them visible to the people most likely to need them. And they create reporting channels that connect the people who see harm early with the people who have authority to act on it.

This sounds bureaucratic, and it is, but the bureaucracy serves a purpose. It makes authority legible. When a problem emerges, there is a clear path to the person who can actually make the call rather than an organizational maze where everyone is technically responsible and practically nobody is. It also creates a forcing function for the organization to think through, before deployment, what kinds of harms would justify shutting the system down. That is a conversation most organizations skip because it is uncomfortable, but skipping it is what produces the scenario where a system runs for months producing harm while people argue about who is supposed to do something about it.

Regulators are starting to impose this kind of thinking from outside. AI governance frameworks in the EU and elsewhere are beginning to require documented accountability structures for AI systems, including clear statements of who bears responsibility for specific kinds of failures. That external pressure is useful, but it should not be the only mechanism. The organizations that will handle AI governance well in the long run are the ones that want clear accountability, not just the ones that are required to have it.