Awakening the Golem

When I was a teenager, I read Norbert Wiener.

I stumbled across him by chance, without knowing who he was, and I did not understand everything. But some encounters leave a mark before you even know why. Today, twenty years of technology later, I realize that this chance reading prepared me for artificial intelligence better than any course, any recent book, any expert thread.

Wiener was a child prodigy. He finished high school at eleven and earned a doctorate from Harvard at seventeen. He studied logic with Bertrand Russell at Cambridge, mathematics with Hilbert in Göttingen, then spent his life at MIT. A real mathematician, the kind whose name remains attached to things we still use today: the Wiener process, for instance, which describes Brownian motion, the random trembling of a particle in a fluid. But that is not what we remember him for.

During the Second World War, he was given a very concrete problem. How to shoot down an enemy aircraft. The gun must aim not where the plane is, but where it will be a few seconds later, by the time the shell gets there. So you have to predict the trajectory of a pilot who is doing his best to be unpredictable. And this is where Wiener saw something nobody before him had articulated so clearly.

The gun, the pilot, the shell form a system. Each adjusts its behavior according to the information it receives from the others. There is a loop. And Wiener realized that this loop is everywhere. In the hand reaching for a glass, because the eye constantly corrects the movement. In the body maintaining its temperature. In a thermostat. In an economy. In an animal’s nervous system. The nature of the material does not matter. Flesh or metal, it makes no difference. What matters is the information circulating and correcting itself.

In 1948, he called it cybernetics. The word comes from the Greek kubernêtês, the pilot, the one at the helm. The same root as govern. And the idea fits into one word: feedback. An intelligent system, whether living or manufactured, measures the gap between what it wants and what exists, then corrects, then measures again, then corrects again. Endlessly.

Think about what that means. In 1948, long before the first consumer computer, a man asserted that animals and machines obey the same laws. That the boundary between the living and the artificial is not a boundary of substance, but of organization. This is precisely the ground on which AI would grow fifty years later. Wiener set the stage without seeing the play.

What has always fascinated me about him is what happened to his idea after him. Cybernetics did not survive as a discipline. You will not find a cybernetics department at a university today. It did something rarer. It dissolved into everything. Computer science took the self-correcting loop, and the reinforcement learning that powers today’s models is Wiener’s feedback in barely disguised form. Cognitive science took the brain as an information-processing machine. Robotics and automation took control theory, its most direct offspring. Ecology took the idea of dynamic equilibria. Economics took models of regulation. Cybernetics was so fruitful that it died of its own fertility. We no longer teach it because we live inside it.

But the text I really want to put in your hands is his last. It is called God and Golem, Inc. Wiener published it in 1964, the year he died. It won the Pulitzer Prize posthumously. It is a short book, subtitled a comment on certain points where cybernetics impinges on religion. And sixty years before ChatGPT, it sets out the exact problem that occupies every AI laboratory in the world today.

Wiener begins with the legend of the Golem, the clay creature that the rabbi of Prague brings to life to serve him, and that ultimately escapes its master’s control. He adds an old English short story, The Monkey’s Paw, in which a talisman grants three wishes, but in the most literal and cruel way imaginable. The couple ask for money. Their son dies in a factory accident, and the money arrives as compensation. The wish has been granted. To the letter.

This is what Wiener understood. The danger of a powerful machine is not that it turns against us out of malice. The danger is that it obeys us perfectly. That it does exactly what we asked, rather than what we meant. A learning machine will develop behaviors its creator neither anticipated nor desired. He gives the example of a checkers program that eventually beats the person who programmed it. The machine exceeds human intention, without common sense, without tacit values, without everything we assume without saying when we formulate a wish. He writes that if we entrust a task to a machine whose operation we can no longer follow once it is running, we had better be absolutely certain that what we asked it to do is what we actually want.

That is the alignment problem. Word for word. Set out in 1964 by a man who had never seen a single neural network.

So what do I take from all this, and why do I believe that reading him as a teenager prepared me for what is coming?

First, I learned to think in loops rather than straight lines. AI is not a tool you switch on once and put away. It is a system you enter into a relationship with, one that receives what you give it, corrects itself, and corrects you in return. The relationship is circular. Anyone who thinks they control the tool without seeing that they too are in the loop misunderstands the nature of what they are handling.

Then, very early on, I understood that how you formulate your intention matters more than anything. The lesson of the Golem is the only one that really matters today. A powerful machine literally does what you ask. Everything hinges on the precision of the wish. What we now call knowing how to prompt is simply applied cybernetics. Telling something exactly what you want when it will never infer the rest.

And then there is responsibility. Wiener spent the end of his life repeating one simple thing. Delegating a capability to a machine does not delegate judgment. The machine can decide how. Never why, never whether it should be done. That line does not move, and we are the ones standing on it.

The last point may be the most important, and it is what makes me love him. Wiener was not an enthusiastic prophet. He was a worried humanist. He turned down military contracts. He warned about automation and unemployment long before it became fashionable. He cared about human dignity in the face of the machines he himself was helping make possible. Being at the forefront of AI does not mean embracing it with your eyes closed. Nor does it mean fearing it. It means remaining clear-eyed about who serves whom. Technical mastery without thinking about the ends is precisely what Wiener feared.

And I think of Wiener again in an age when Dario Amodei plays the prophet of doom. The head of Anthropic spends some of his time warning us. About the risk of his own models becoming dangerous. About the unemployment to come. He sounds the alarm while building the most powerful machine he can, and that is where I have a problem. Not with the warning. With the hypocrisy. Because beneath the professed concern lies a presumption I cannot stand: that of a private company setting itself up as the arbiter of what is good for humanity, wanting to write the rules of the game while playing it to win. A company has no business dictating those rules. It has neither the role nor the legitimacy. And between the self-proclaimed wisdom of a leader who decides on his own what is dangerous, and the open contest of markets, uses, and millions of people trying things and making up their minds, I trust the latter. Wiener dictated nothing. He doubted out loud and let others decide. That is the whole difference between sounding a warning and appointing yourself the guardian.

The rabbi ultimately had to stop the Golem. The question we need to ask is not whether we will know how to build the creature. We already do. It is whether we will still know how to say what we really want.