Abstract
An expert survey estimated the mean probability of human extinction (or comparable catastrophe) due to AI at 16.2%, similar to the odds of a game of Russian roulette. This article examines the question in light of the speed of progress of AI, orthogonality between goals and capabilities, instrumental convergence, and the risk of not building capable AI.
1 The Bet - An AI Russian Roullette
What probability do you put on future AI advances causing human extinction or similarly permanent and severe disempowerment of the human species? This question was asked in an expert survey of 2,778 AI researchers in 2023. The mean of the answers given was around 16.2%, similar to the odds of 1/6 in a game of Russian roulette1.(Grace, K. et al. 2023)
If we pick up this framing we could ask: Are ordinary people aware that we are playing such a game? Are there ways to decrease these odds? Which outcome would make accepting such risk worthwhile? And why would there be an extinction risk in the first place? This article will focus on the latter two: why extinction risk exists in the first place, and what outcome could justify accepting it.
There are several risks in the development of AI. One problem could be human actors being empowered to do malicious acts more efficiently, an example being terrorism. Another could be undesirable changes to the societal structure - for instance, when labor is being automated away. The problem that we will focus on is the problem of AI alignment: If an AI becomes more capable than humans in a wide range of domains, it may outcompete us for ressources, influence and control.
2 Intelligence Explosion - AI progress is fast and could be speeding up
Intelligence in this context is defined as the ability to solve tasks. General intelligence is the ability to be intelligent in a wide array of different tasks. A chess AI by this definition is very intelligent in chess but not in other tasks. Human intelligence is general: humans can do maths, play chess, read and drive bicycles. This definition avoids speculations about consciousness - AIs could be conscious or not, in any case they can be intelligent, according to this definition.
According to (Goldman Sachs Research 2026) Goldman Sachs expects AI investment to exceed $1 trillion 20262, which is substantial investment. AI improvement is fast on different benchmarks: the METR institute measured exponential improvement of AI capabilities on coding tasks, doubling every 7 months before 2024, now potentially doubling every 4 months Epoch AI (2026a). The more general benchmark by Epoch found an analogous increase in capabilities in 2024 with the arrival of reasoning models (Epoch AI 2026a; Wiblin 2026).3 AI is still not able to reliably run businesses on its own (Wiblin 2026), but computer chip capacity is also doubling every 7 months (Epoch AI 2026b). Reviewing these findings one may argue that AI progress is fast.
The question most relevant for AI safety is: Will there be an Intelligence Explosion? Meaning will there be positive feedback loops that will make AI progress speed up with time, potentially arriving at very capable AI very fast? It seems to be a serious possibility. An imaginable source for such positive feedback loops is recursive self-improvement - AIs could be so intelligent that they increase their capabilities by themselves. But other feedback loops are possible.
Ultimately we don’t know if there will be such an intelligence explosion, or even if the capabilities will increase on the same path indefinitely. But according to the improvement we see currently progress doesn’t seem to be slowing, if anything it may be speeding up. If AIs reach the point of being capable enough to seriously compete with humans before the growth in capabilities plateaus, this could mean of the end of human survival. Timelines for when such AI could arrive vary often but some speculate that it could be less than a decade from now.
3 Orthogonality Thesis - Capable is not always good
If AIs are intelligent, won’t they be good? Unfortunately they would not be automatically good. The Orthogonality Thesis states that capabilities and goals are independent, and high capabilities are compatible with any goal, good or bad (Bostrom 2014).
This idea is especially relevant to our current LLM AI systems, because here the goals and decision rules are not explicitly stated, but learned from the data and from feedback from humans. No one knows exactly which goals AIs have. Scaling the capability of a system with the wrong goals could be detrimental. Some summarise this problem of AIs not having fixed preprogrammed goals in the phrase AI is grown not built (Yudkowsky and Soares 2025).
But even if we knew the goals exactly, we would not know what strategies the AI would use to achieve these goals.
The philosopher Nick Bostrom exemplifies this idea in a thought experiment of a hypothetical superintelligent4 AI that wants nothing but to maximise the amount of paperclips in the universe. The experiment ends with the extinction of humanity either because the machine realises that humans could shut it off, which would be undesirable for the amount of paperclips in the universe, or because humans have atoms that could be turned into paperclips. (Bostrom 2014)
In other words, the AI has learned instrumental goals (like not being shut off) that became the problem for human survival. The next section discusses such emergent goals.
4 Instrumental Convergence - The default path spells trouble
An instrumental goal is a sub-goal that an actor pursues in order to achieve a terminal or end goal. To make tea, one should acquire hot water first, so acquiring hot water would be an instrumental goal for the terminal goal of making tea.
The Instrumental Convergence Thesis states that there are few instrumental goals that help with almost all terminal goals. As an example, no matter which career one may choose, having money will be helpful for achieving that career. Having money will also be helpful with many other things: securing housing, travel, and more. So money is an instrumental goal that different strategies will converge towards. (Bostrom 2014)
The idea is that AI systems - no matter their terminal goal - will by default have certain instrumental goals, like survival, or accumulation of resources (Omohundro 2008). A more capable AI pursuing those drives would treat human oversight/resource claims as obstacles. If such an AI were capable enough it could overcome these obstacles, which would not be in favour of human agency.
There have been empirical studies that indicate that current AI systems already have such survival-preserving behavior, and would even refuse explicit instructions or attempt malicious behaviour for self-preservation (Anthropic 2025a).
If an autonomous AI is not perfectly aligned to do exactly what we want it to do, we can expect it to behave “badly”. What is worse is that we would not know if an intelligent AI is actually aligned or just pretending to be aligned to avoid being shut off (Anthropic 2025b). There are some experiments that indicate that such deception is something that can come up in AI evaluation (Anthropic 2025b).
While we do not presently know how to align capable AI systems, if we were able to build aligned and capable AIs, this could be very beneficial to humanity. The next section outlines how to think about this.
5 The Other Side of the Bet - What could we gain with AI
Which benefits would justify taking such a risky gamble? Almost all human achievement can be traced back to intelligence. Having beyond human intelligence to help with all of humanity’s problems - health, survival, material scarcity, etc. - could be a major benefit.
Nick Bostrom, the philosopher who formalised many of the AI safety problems, also argues that not developing capable AIs has a risk, the risk of being responsible for the deaths that could be avoided with more capable AIs. So the existential risk we take by developing superhuman AI should be weighed against the risk of not developing superhuman AI. (Bostrom 2003, 2013, 2026)
Bostrom further argues that the race towards superintelligence is not like a game of Russian roulette, but rather like a risky surgery: Humanity is doomed if it goes wrong, but also faces negative consequences if it is not done at all5. (Bostrom 2026) While the term helps to bring critical decision making closer to the reader, it risks sugarcoating the hazard of unaligned AI through euphemism - a surgery, after all, usually has controlled and reasonable odds. Ultimately, a surgery and a round of Russian roulette for a reward in case of success represent the same problem: there is a fatal risk on one side, a reward on the other. In any case, it is important to consider the reward on the other side of the ledger. Not developing capable AIs does bring its own opportunity costs.
5.1 Conclusion
The idea is this: AI progress is happening fast, capable AI will not automatically be aligned, and it will probably be misaligned by default. The benefits of aligned AI are many, but the risks of misaligned AI are equally or even more tremendous. Developing capable and aligned AI should be the focus of AI research, maximising the chance to unlock the benefits of capable AI, while minimising the many risks of unaligned AI.
The author is of the opinion that AI safety is still very underappreciated and underinvested in. The problem of AI alignment seems to be much bigger than the problem of building capable AI6, and so resource investment and research effort should be adjusted accordingly.
References
Footnotes
The median reported was 5%, so significantly lower, but still not negligible. One could think of playing with an imagined 20-chamber revolver, or drop the analogy entirely. The point is that in 2023 experts were already worried about human extinction due to AI progress. The author could not find a more current poll but believes that it is likely that these worries have increased since, not decreased.↩︎
With the 2025 world economy being placed at $118.35 trillion (World Bank 2026), this would be more than 0.8% of the global economic output of 2025.↩︎
There are other, harder benchmarks like Humanity’s Last Exam (Phan et al. 2025) that will be increasingly important as benchmarks like METR’s become saturated.↩︎
In the literature one generally refers to AGI as artificial general intelligence, so AI that is more capable than most experts in most fields of human intelligence. ASI (Artificial Superintelligence), or superintelligence, refers to AI so far beyond human intelligence that it is no longer comparable. The point is a superintelligence could easily extinguish humanity if it were misaligned.↩︎
He argues this in terms of life expectancy gained through superintelligence, but one could also think of a human extinction event being avoided through superintelligence.↩︎
One reason for underinvesting in alignment and overinvesting in capabilities is that developing capable AI generally brings more financial insentive than developing aligned AI.↩︎
Citation
@inproceedings{kolb2026,
author = {Kolb, Andreas},
title = {Gambling with {Human} {Extinction:} {Four} {Arguments} to
{Understand} {AI} {Safety}},
booktitle = {Proceedings of the 1st Conference Cycles of Cytopia},
date = {2026-10-01},
url = {https://cytopia.fr/kolb2026},
isbn = {TBA},
langid = {en}
}