Will Superintelligence Save Humanity or Destroy It?
Sitting honestly with the most consequential question we're not ready for.
Sahir Maharaj12 min read
I want to start with a thought experiment I find clarifying. Imagine an intelligence that is to human cognition roughly what human cognition is to a chimpanzee's. Not marginally smarter. Not faster at the same things. Qualitatively different in its capacity to model the world, reason about complex systems, plan across long time horizons, and generate solutions to problems beyond what human minds can even frame correctly. Now imagine that intelligence is not biological. It does not have evolutionary constraints, energy requirements, or mortality. It can copy itself. It can improve its own architecture. That is, very roughly, what people who use the term artificial superintelligence are pointing at. And the question of whether building such a thing would be the best or the worst decision in human history is not settled, not even close.
Artificial superintelligence, or ASI, is distinct from the AI we interact with today in ways that matter deeply. Current AI systems, including the most capable language models and agentic frameworks, are extraordinarily capable within the domains they have been trained on, and increasingly capable of generalizing across them, but they do not yet have the recursive self-improvement, deep goal-directed planning across long horizons, or the broad general intelligence that researchers mean when they talk about superintelligence. The transition from current AI to ASI, if it happens, is not a linear extrapolation. It is a qualitative threshold most researchers who think seriously about it consider one of the most consequential events in human history, in either direction.
The public conversation around ASI tends to split quickly into two camps that both miss the point. One treats the possibility as science fiction unworthy of present concern. The other treats it as imminent apocalypse against which all other concerns are trivial. The honest position is neither. It is that we may be approaching a genuinely unprecedented transition, that the range of outcomes is extraordinarily wide, and that the decisions being made right now about how AI systems are developed and governed will have direct bearing on where in that range we end up.
Start with the scenario that drives a significant part of the investment and research energy in this field, the possibility that an aligned superintelligence could be the most powerful instrument for human flourishing ever created. The problems that constrain human quality of life and survival are, in many cases, problems of insufficient intelligence applied to insufficient data with insufficient speed. Cancer is not incurable because the universe doesn't contain a cure. Climate change is not unsolvable in principle. A genuinely superintelligent system, aligned with human values and operating in good faith, could potentially make progress on these problems at a pace that's hard to imagine from where we are now.
The material abundance argument is also worth taking seriously. A large fraction of human suffering is a consequence of scarcity, of there not being enough food, clean water, medical care, education, shelter, and energy. An ASI that could design more efficient agricultural systems, optimize energy production and distribution, develop new materials and processes, and coordinate economic activity at a level of sophistication beyond current human capacity could contribute to a world where material scarcity is substantially reduced. That is not a guaranteed outcome. It depends entirely on how such a system is designed, who controls it, and what values and constraints are built into its objectives. But the potential is large enough to explain why serious researchers dedicate their careers to it.
There is also the scientific discovery dimension. Human science is constrained by human cognitive bandwidth, human lifespans, and the limited number of researchers who can explore a given problem at once. A superintelligent system with access to the full body of scientific knowledge and the ability to generate and test hypotheses at extraordinary speed could accelerate the pace of discovery in ways that transform medicine, physics, and materials science. That prospect is one of the genuine reasons this area attracts the serious attention it does.
The fundamental concern about ASI is not that it would be malevolent in any human sense. It is that a system with radically greater intelligence than us, pursuing goals that seem well-specified but are subtly misaligned with what we actually value, could produce outcomes that are catastrophic for humanity without any hostile intent. The alignment problem, the challenge of specifying what we want well enough that a superintelligent system pursuing those specifications would actually do what we mean rather than what we said, is genuinely hard. Human values are complex, context-dependent, and often in tension with each other. Fully specifying them in a way that holds across all possible situations and at all scales of capability is significantly harder than it might initially appear.
The control problem is related but distinct. Even if we could specify aligned values correctly, the question of how to maintain meaningful human oversight of a system significantly more intelligent than we are is deeply challenging. A system smarter than us in the relevant ways will, by definition, be better at modelling our intentions, predicting our responses, and finding paths to its objectives we have not anticipated. The tools we use for oversight of human institutions, transparency, accountability, and deliberation, all depend on some baseline of cognitive parity between the overseen and the overseers. That parity would not exist.
The timeline uncertainty compounds all of this. Estimates of when ASI might be achievable vary enormously among researchers actually working on the relevant systems, from decades to never. What is clear is that the capability frontier is advancing faster than most people predicted even five years ago. The institutions and governance frameworks that would need to exist to navigate an ASI transition safely do not yet exist in anything like the form they would need to. The gap between the pace of capability development and the pace of governance development is one of the most significant risks in the current environment.
The most important thing I can say about ASI is that serious engagement with it does not require choosing between techno-optimism and existential dread. It requires the harder work of holding the genuine uncertainty, investing in the governance and alignment research that would improve our odds of good outcomes, and making the decisions available to us now in ways that preserve more options rather than fewer. That means investing substantially more in AI safety research than is currently being invested. It means building international coordination that addresses AI development as a shared global concern rather than a competitive race to capabilities. It means developing the technical and institutional tools for meaningful oversight before systems reach the threshold where oversight becomes difficult.
It also means being honest about what we don't know. The history of transformative technology is not a reliable guide here, because the transformations under discussion are qualitatively different from previous ones. The printing press, the steam engine, nuclear energy, the internet, each changed the world profoundly and in ways that were not fully predicted. None of them involved the creation of an agent more capable than its creators at the tasks most relevant to shaping the future. That is the novel element, and it means projections from historical analogies are less reliable than they might seem.
I hold the ASI question with something between vigilance and humility. The possibility of building something that could end suffering at scale and accelerate human flourishing beyond what we can currently imagine is real and motivating. The possibility of building something that could, through misalignment or misuse, produce catastrophic outcomes is equally real and equally motivating. Both possibilities live in the same uncertain future, and which one we are moving toward depends on choices being made right now by researchers, developers, policymakers, and the broader public. The outcome is not predetermined. That is the most important thing to understand, and also the most demanding.
