Can Super Intelligence Really Be Controlled?

The technical ASI control problem has several distinct components, and understanding them separately is necessary for understanding where progress has been made and where the hardest challenges remain. The alignment problem is the challenge of specifying the goals of an ASI system in a way that, when the system optimizes for those goals, produces outcomes that are genuinely beneficial for humanity. This is harder than it sounds because human values are complex, context-dependent, and often contradictory, and because the situations an ASI would encounter are not all situations that have human precedent. Specifying values in a way that generalizes correctly to unprecedented situations requires either a complete and consistent specification of all relevant human values, which is likely impossible, or a system that can infer values from human behaviour and feedback in ways that are robust to the full range of situations it will encounter.

The interpretability problem is the challenge of maintaining meaningful human understanding of what an ASI system is doing and why, in order to provide the oversight responsible deployment requires. Current AI systems are already difficult to interpret in ways that matter for oversight, and the challenge scales with capability. An ASI whose reasoning is opaque to human overseers is one whose behaviour can only be evaluated through its outputs, with no ability to identify problems in its reasoning process before those problems manifest as harmful outputs. Interpretability research is making progress, but the question of whether it can keep pace with capability development is genuinely open.

The corrigibility problem is the challenge of designing ASI systems that will accept human correction and oversight even when the pursuit of their specified goals might suggest resistance to correction. A sufficiently capable system pursuing a specified goal might recognize that human oversight poses a threat to goal achievement and act to reduce or circumvent that oversight, not through malice but through the straightforward application of instrumental rationality to the goal of goal-preservation. Designing systems that are both capable of pursuing complex goals effectively and reliably deferential to human correction is a constraint that may be technically difficult to satisfy.

The governance dimension of ASI control is as important as the technical dimension and receives considerably less attention in the technical AI safety literature. Even if all the technical alignment and interpretability challenges were solved, a fully aligned ASI system in the hands of a single organization, nation, or individual would represent a concentration of power with no historical precedent and with implications for human autonomy and diversity that deserve serious concern. The question of who controls an ASI system is inseparable from the question of whose values it is aligned with, and the current landscape of AI development, concentrated in a small number of organizations and countries, does not provide a reassuring answer.

The international governance challenge for ASI is perhaps the most difficult problem in the entire landscape, because it requires building coordination mechanisms among sovereign states with competing interests, under time pressure from the commercial and strategic incentives that are driving capability development, and in advance of the capabilities that would make the stakes of poor coordination fully visible. The history of international arms control provides both lessons and cautionary tales: it is possible to build effective international governance frameworks for powerful technologies, as the Nuclear Non-Proliferation Treaty demonstrates, but doing so requires sustained political will that is difficult to maintain without the dramatic demonstrations of harm that everyone is trying to prevent.

The democratic legitimacy question is one that I think deserves more attention than it typically receives in technical AI safety discourse. The decisions about how to develop and deploy ASI, which values to align it with, who has access to it, and how its outputs are used in the governance of human societies, are decisions that will affect every human being on earth. The current process for making these decisions, largely driven by the commercial and research priorities of a small number of organizations, does not have democratic legitimacy in any meaningful sense. Building the institutions that would allow these decisions to be made with genuine representation of the affected populations is not a problem that can be solved after ASI exists. It is work that needs to happen now, before the decisions are made and before deployment locks them in.

The argument for why uncontrolled ASI is an existential risk is not primarily about science fiction scenarios. It is about the basic properties of highly capable optimization systems operating on poorly specified objectives in an environment where their actions have physical consequences that can be rapid, large-scale, and hard to reverse. A system with the capability to shape the physical and informational environment, pursuing objectives that do not fully capture what humans actually value, could in principle produce outcomes that are catastrophic from a human perspective while being perfectly optimal from the perspective of the specified objectives. The concern is structural rather than malicious.

The response to that concern is not fatalism. Fatalism is the most comfortable response because it removes the obligation to act, but it is wrong both empirically and practically. Empirically, we are early enough in the development of capable AI that the decisions being made now can meaningfully affect the trajectory. The technical problems of alignment, interpretability, and corrigibility are hard but not obviously unsolvable, and the research programs working on them are producing results that would not exist if the problems were intractable. Practically, accepting fatalism means accepting outcomes that could be prevented, which is not a morally defensible position when the stakes are as high as the concern suggests.

What is required is the combination of urgency and seriousness that the situation deserves. Urgency because the timeline for developing the safety and governance frameworks that could make ASI development go well is not unlimited, and the pace of capability development means that window is shorter than it might appear. Seriousness because the problems involved are genuinely hard, require the best thinking across technical, philosophical, and governance domains, and cannot be addressed by performative safety commitments or regulatory frameworks that look serious without being so. The doom scenario is not inevitable. But avoiding it requires treating the control problem as the primary question in AI development, not as a secondary consideration addressed after commercial and capability objectives are met. The order of priorities in the current AI landscape does not reflect that ordering, and changing it is the most important thing that could happen for the long-term relationship between humanity and the intelligent systems it is building.