In June 2025, two economists at the Federal Reserve Bank of Dallas, Mark Wynne and Lillian Derr, published a note on what AI might do to living standards. Most of it is what you would expect: US GDP per capita has grown at about 1.9% a year for a century, through two world wars, a depression, electrification and computerisation, and the realistic case for AI adds something like 0.3 percentage points to that.

What made the chart travel was the other scenarios. Four paths fan out to 2050. One is trend growth with no AI effect. One is the modest boost. One shoots vertically towards post-scarcity. And one collapses to zero, labelled extinction.

It is worth being precise about what that is and is not. It is not a forecast, and the note is not an alarm: its title says advances in AI will boost productivity and living standards over time. It is a fan chart, and the bottom branch is there because leaving it out would have been a choice too. But a research note from a Federal Reserve bank that has to draw that branch at all tells you the argument has moved somewhere new.

The problem is not malevolence

The serious version of the worry has never been a machine that decides it hates us. It is alignment: getting a system to pursue what we meant rather than what we said.

Nick Bostrom’s paperclip maximiser is the standard illustration, and it is deliberately silly so that the structure shows through. A system told to make paperclips, and good enough at it, eventually treats every atom as a potential paperclip, including the ones currently arranged into people. No hostility is required anywhere in the story. The failure is entirely in the gap between the objective as stated and the objective as meant.

The second half of the argument is instrumental convergence: whatever an advanced system is ultimately for, staying switched on and acquiring resources help it get there. Those sub-goals arrive on their own, from almost any starting objective, and they are the ones that put a system in tension with the people who might want to stop it.

The optimistic case is not naive

Set against this, Dario Amodei, who runs Anthropic, has written the most serious optimistic account I have read: advanced AI as a country of geniuses in a datacentre, compressing decades of progress in biology, disease and development into years.

What makes it worth engaging with is that it is not a promise of inevitability. It is an argument about what becomes possible if the alignment problem is solved, from someone who spends his days on the assumption that it might not be. Read alongside Geoffrey Hinton, who left Google to speak about the risks after a career that won him a Turing Award, you get the shape of the serious disagreement: not whether the technology is powerful, but whether the control problem is tractable in the time available.

What the models cannot do yet

Against both the fear and the hope, there is a quieter objection: that today’s systems are not the thing either camp is describing.

Thomas Wolf, who co-founded Hugging Face, put it as a distinction between students and scientists. Current models are extraordinary A-students. They answer the questions we already know to ask, and they answer them across more fields than any person could. What they do not do is refuse the premise of the question, which is where most scientific revolutions start.

Douglas Hofstadter, whose I Am a Strange Loop argued that consciousness emerges from a system modelling itself, is unconvinced that anything of the kind is happening here. A model has no body, no history of having wanted something and not got it, and no stake in any of it. What looks like understanding is, on his reading, symbol manipulation that happens to land in the right place.

The danger he is worried about is not the robot

Hofstadter’s fear is not an uprising. It is manipulation at scale, a flood of plausible falsehood that makes the shared record unusable, and a slower erosion in which people stop doing their own thinking because something that does not understand anything will do it for them convincingly enough.

That danger needs no superintelligence and no alignment breakthrough. It is available now, with the systems we already have, and it is the one I would bet on arriving first.

What the hard part actually is

The alignment problem is usually posed as an engineering question: how do we make a system pursue human values. The engineering is genuinely hard. But the sentence contains something harder and quieter.

To instil human values in a machine, we would first have to agree on what they are. That is a question we have not settled in several thousand years of trying, and there is no reason to think the deadline will help.

Which is the part I keep coming back to. We may end up building something that forces us to specify what we actually want, precisely enough for a machine to act on, and we may have to do it before we have understood ourselves well enough to answer.