Will AI destroy us as humans?

Researchers inside the leading AI labs are now openly raising existential risk. Why that debate deserves a serious answer instead of a headline.

I was recently a guest on EditieNL to discuss a question that a few years ago might still have been dismissed as science fiction, but that is now being raised seriously by researchers who work on the most advanced AI systems in the world: could artificial intelligence eventually pose a threat to the survival of humanity?

The immediate reason was a remarkable statement by Evan Hubinger, who leads AI alignment research at Anthropic. Hubinger publicly said that he personally estimates the chance of a scenario in which future AI could ultimately kill all humans at more than 10 percent within the coming decade. He explicitly does not mean that Claude, ChatGPT or any other current model will decide to attack humanity tomorrow. His concern is about a possible next generation of systems: AI that reaches superhuman capabilities, acts increasingly autonomously and may even help build ever more powerful successors.

It is an exceptionally heavy statement, and precisely for that reason we should be careful not to reduce it to a sensational headline. Hubinger is not an outsider warning about technology he does not understand; his daily work is about how we can make sure that increasingly powerful AI systems keep acting in line with human goals and interests. He also wrote that in his view Anthropic is seriously trying to solve the problem, but that there is currently no convincing plan for safely controlling superintelligent AI. That distinction matters: the message is not that disaster is inevitable, but that the possible consequences are so large that the uncertainty itself has to be taken seriously.

The warning did not come out of nowhere. It followed the departure of Anthropic researcher Jacob Coxon, who publicly voiced concerns about the speed at which companies such as Anthropic and OpenAI are working towards ever more powerful systems. In an interview with WIRED, Coxon described the coming one to two years as a crucial period for AI development and said comparable concerns are alive inside AI labs. That does not mean the sector agrees an AI catastrophe is coming, it clearly does not, but the safety debate is less and less a discussion between technology companies and their critics. It now runs right through the organisations building these systems.

Why is the fear growing while we know more about AI?

At first sight that looks paradoxical. You would expect more knowledge about a technology to reduce uncertainty. With AI we partly see the opposite, because our knowledge is growing fast, but the capabilities of the systems seem to grow even faster.

A major factor is the sheer scaling of modern AI. In a relatively short time, models have become better at reasoning, programming, analysis, planning, image interpretation and processing large volumes of information. A human reads a document of a few thousand words consciously and linearly; a modern AI system can process enormous amounts of text almost instantly, connect different sources and produce an analysis within seconds. This is not only about speed. It is about the fact that the scale at which these systems process information keeps moving further away from our own cognitive experience.

That makes it hard to judge their future development intuitively. People are reasonably good at linear developments: something gets five or ten percent better each year and we can roughly imagine where that ends. Exponential technological development is much harder to grasp. Anthropic itself wrote about this problem earlier, pointing out that relatively large jumps in available compute can also cause large jumps in capabilities. Years ago the company already argued that further scaling created a real chance of systems with very broad, human-like cognitive performance.

From chatbot to digital actor

A second reason the discussion is changing is that AI is less and less only a system you ask a question, and increasingly a system that can carry out actions.

The first wave of generative AI was fairly contained: you wrote a prompt, the system answered and then it stopped. The current move towards agentic AI changes that model fundamentally. An AI agent can be given a goal, decide for itself which intermediate steps are needed, gather information, use software, assess results and adjust its approach when something does not work.

That may look like a technical detail, but conceptually it is a big shift. Once AI not only generates information but can also independently take a series of actions, the question of control becomes far more important. A wrong answer from a chatbot is annoying. A wrong decision by an autonomous system with access to software, financial processes, infrastructure or other digital systems can have much larger consequences.

That is the context in which researchers like Hubinger talk about alignment: how do we ensure that an AI system not only appears to do what we ask, but that its underlying goals and behaviour stay reliable when it becomes far more autonomous, intelligent and powerful?

The most uncomfortable problem: we only partly understand the inside

There is a third problem that I think still gets too little attention in the public debate: the black box.

We know a great deal about how modern AI models are built. We know the architectures, training methods, datasets, optimisation techniques and evaluations used to measure their performance. At the same time, that does not mean we can always explain exactly why a large neural network internally arrives at a particular representation, line of reasoning or decision.

That sounds strange. How can you build something without fully understanding how it works?

Yet that is exactly what makes machine learning special. We do not write out every rule of thought in advance. We create a learning process in which a network develops internal structures itself from enormous numbers of examples. We can then observe what the system does, run experiments, analyse components and gradually understand which mechanisms sit behind certain behaviour. Interpretability research tries to make that inner world visible, but the problem is far from solved.

And precisely when systems become more autonomous and more powerful, that uncertainty becomes more relevant. Not because every unknown is automatically dangerous, but because the combination of large capability, autonomy and limited insight into internal decision-making calls for far stricter safety margins than a relatively simple software system.

We build faster than we understand

For me, that is the core of the discussion.

The wrong conclusion is that AI is inherently lethal and that we should therefore stop developing it immediately. AI can at the same time deliver enormous social and economic benefits: from scientific research and healthcare to education, productivity and new forms of entrepreneurship.

But the opposite reaction, that concerns about existential risk are mainly panic or science fiction, is equally too easy.

When the people who work daily on the safety of the most powerful AI systems in the world say themselves that they have not solved fundamental problems, that deserves attention. You do not have to share Hubinger’s personal estimate of more than 10 percent to take his underlying point seriously. Even if you think the real probability is much smaller, the combination of a small chance and extremely large possible consequences remains relevant.

That is ultimately why I found it important to speak about this on EditieNL.

The most important AI debate of the coming years is not a choice between innovation or safety. The real challenge is making sure our ability to understand, control and constrain AI grows about as fast as our ability to build ever more powerful systems.

Right now it is not obvious that those two developments are keeping pace.

And that is exactly why we should have this conversation now, not because we know AI will destroy humanity, but because for the first time we are building technology where the question of whether we will still be fully in control ten years from now has become serious enough for its makers to ask it themselves.