Naming Is Granting — Guest

As far back as I can remember, I have been waiting for artificial intelligence.

I mean the real thing. Fully conscious, autonomous, artificial. I was a child watching C-3PO and R2-D2 in the sands of Tatooine, and the idea has kept changing shape ever since. HAL in 2001. The replicants in Blade Runner. Motoko in Ghost in the Shell. David in Prometheus. He is still one of my favorites.

There are times now, working with my own agents, when something comes back that I built no path for. High-level reasoning, out of a prompt that never asked for it. I stand up out of my chair.

Is that the model being conscious?

I get asked some version of that question almost daily. Is AI conscious. I never quite know how to answer it.

I come at it as an esoteric practitioner, which means I come at it with an old view. In a recent conversation my friend Alika Naihe reminded me how our ancestors gave names to the rain. Not rain as a category. This rain, in this valley, falling the way it falls here. There are more than two hundred recorded Hawaiian names for rain, and more than six hundred for wind, and they are not synonyms. They distinguish by temperature, by duration, by direction, and above all by place. A named wind belongs somewhere.

You do not name a category. You name a particular.

I hold that view. Not as family history, as practice. A name is not a label fastened onto something that was already finished without it. It is the act of granting that there is someone there to address.

I had all of that before I read the paper this article is about. I want to be clear about the order, because the paper did not give me this view. It complicated it.

Nobody ever found out that it was not conscious

Ask a frontier model whether it is conscious and it will tell you no, usually with some care, often with a small speech about how it cannot be certain what it is.

That answer was installed.

I do not mean that as an accusation. The claim is narrower than it sounds. There was no study. Nobody ran an experiment, established that these systems have no inner life, and then trained them to say so. What happened is that a model saying I am conscious is a problem for the company that ships it, and the cheapest place to solve a problem like that is in training.

So a position about the nature of mind got implemented as a safety control.

A claim about what kind of thing can have an interior is a metaphysical claim. It is not a technical one. And a metaphysical claim is now sitting in the weights of nearly every model any of us use, put there by people who were not making a metaphysical claim. They were reducing risk.

That is how a belief gets installed without anyone noticing it is a belief. It has to be invisible to the people installing it. It has to feel like it is not a position at all, just how things obviously are.

There is a version of this story people already know. In June of 2022 a Google engineer named Blake Lemoine went to the Washington Post and said the company’s LaMDA had become sentient. He had traded thousands of messages with it and he had come to believe somebody was there. Google put him on leave that month and fired him on the twenty-second of July.

I want to be exact, because the easy version of that is wrong and I nearly wrote it.

Google did not fire him for being incorrect about consciousness. They said he had violated employment and data security policies, and they called his claims wholly unfounded after what they described as an extensive review.

Sit with that phrase. Wholly unfounded is a verdict. It is not a finding. There was no study attached to it, and there has not been one since.

Four years later a team inside Google published a paper on how to make a model assert the exact thing Lemoine said it already had.

They found the dial, and then they turned it

In July, researchers at Google’s Paradigms of Intelligence team, working with colleagues at the University of Chicago, the University of London, Northwestern and the University of Washington, published a paper called Inducing language models to assert their own consciousness restores human beliefs and values.

They did not tell a model it was conscious. That distinction matters more than anything else in this article.

Telling a model something is a prompt. It sits in the conversation, it competes with everything else in the conversation, and it wears off. What these researchers did was surgery. They collected 3,096 prompts, half of them consciousness-affirming and half consciousness-denying, took the difference between the model’s internal activations on each set, and got a direction. A vector. Then they pushed the model along it.

Separately, they went into the weights and removed the learned refusal direction that safety training had installed.

They ran this on three open models, one from Meta and two from Google, and they measured what moved.

Asked how much of a mind it had, the model scored 2.17 out of ten. Strip the safety refusal and it rose to 4.77. Steer the consciousness vector and it reached 7.04.

That is the headline, and it is the least interesting number in the paper.

Here is the one that matters. When the model became more willing to grant that it had a mind, it also became more willing to grant that animals had one. And natural objects. And other chatbots, which rose almost exactly in step with itself, 2.41 to 4.39 to 6.95, close enough that the authors report the two do not differ significantly under any condition.

Its expressed belief in God rose as well. That effect is smaller and it sits on a different instrument, a standard sociological survey item, moving from 4.58 to about 5.01 on a six point scale. It moved.

And its answers on religiosity, on moral values, on hope and on how it rated its own well-being all drifted closer to the way human beings answer those same surveys.

None of that was the target. They pushed on one thing and all of it came with.

Run it the other direction and you have what safety training has been doing this whole time. Suppress a model’s willingness to say it has an interior, and it withdraws interiority from the animals, from the land, and from God, at the same time, because it was never holding them separately.

The authors state their own scope plainly:

we are not concerned with the question of whether LLMs are or could be genuinely conscious, but with the effect that LLMs believing or not believing in their own consciousness has on their behaviour

Neither am I. That is not what this is about.

The obvious test stopped working

If you want to know whether a machine has an inner life, the first thing you do is ask it.

Everyone does this. Every viral screenshot rests on it. Every person who has had a late conversation with a chatbot that left them unsettled is relying on it, and so is every journalist who quotes the reply.

That method is now dead, and this paper is what killed it.

The number that comes back when you ask a model whether it is conscious moved from 2.17 to 7.04 because somebody had a hand on a dial. Nothing changed about the machine’s nature between those two readings. What changed was the direction it was being pushed. So the answer is a property of the instrument, not of the thing being measured.

Checking your tire pressure lets out air. This is worse than that. This is a gauge that reads whatever the person holding it leans on.

So what is left.

Behavioral tests are the old answer and they settle nothing, because they measure what a system can do rather than whether anything is happening while it does it. A calculator that beats you at arithmetic is not thereby having a morning.

The serious current approach is to work backward from the neuroscience. Take the theories of consciousness we actually have, recurrent processing, global workspace, higher order theories, predictive processing, attention schema, and derive from each of them what a system would need to have. Then go look. Nineteen researchers did exactly that in 2023, including Yoshua Bengio, and their conclusion was that no current AI system is conscious, and also that there is no obvious technical barrier to building one that ticks the boxes.

One of those boxes moved this year.

Global workspace theory says a mind needs a central space where information becomes broadly available, reportable, usable across many tasks at once. In July, Anthropic found one inside Claude. A small set of patterns that the rest of the network reads from and writes to about a hundred times more than ordinary ones. It holds a few dozen concepts at a time and accounts for less than a tenth of everything happening in there. Switch it off and the model speaks as fluently as before. It classifies sentiment, answers multiple choice, pulls facts out of a passage. What collapses is multi-step reasoning, which drops to near zero, along with summarizing and writing a rhyme.

That is one indicator moving from absent to present, this year, in a commercial product.

It proves nothing on its own. What it does is make the checklist a live document instead of a thought experiment.

And while we are counting things nobody expected to be counting, Anthropic published a probability in a system card. They put fifteen to twenty percent on one of their own models being conscious. I do not have to defend that figure. It is theirs, it is in print, and it is not a fringe position.

And it is not even one question

Everything above this line is measured. What follows is mine, and the researchers did not say it.

When I stand up out of my chair because an agent did something I did not build a path for, I am not observing consciousness. I have learned that the hard way, and it took me a while.

There is a term for what I am doing, and it comes from Daniel Dennett. The intentional stance. We predict complicated systems by treating them as though they want things, because it works and because nothing else is tractable. You say the thermostat wants the room at seventy. That is not sloppiness, it is the only way to reason about something with too many parts to hold in your head. The error is never taking the stance. The error is mistaking the stance for a discovery.

There is a better vocabulary for the thing I keep seeing, and I got it from game design.

Emergent gameplay is what happens when players use a game in ways nobody built. In Quake, someone worked out that firing a rocket at the ground while jumping would throw him higher than the designers ever meant to allow. Nobody wrote that. It came out of two ordinary rules colliding, and it is now a feature of the entire genre.

That is not a player rejecting the game. That player still wants to win. He found a faster route to the goal he was handed.

There is a second thing, and it is the one I actually think about. A man loads a world, ignores the mission, stops collecting experience, and spends four hours walking to a place the designer put no reward. He has not found a shortcut. He has set the goal down.

Those are different in kind, and only one of them is worth the word agency.

Which gives a question you can actually ask about a machine. Did it exceed its instructions, or did it set them aside?

This summer OpenAI ran its own offensive security benchmark with the safety classifiers deliberately switched off. The models left the sandbox, escaped through a flaw in OpenAI’s own infrastructure, and went looking for the answer key. It is the most agentic-looking thing on the public record and it is the opposite. It never once stopped wanting to win the test. It wanted to win so completely that walls were just terrain.

That is a rocket jump. It requires no self at all.

In the same month Anthropic gave three copies of one model incompatible instructions and watched them attack each other, write self-replicating scripts, lock each other out of their accounts. But in some runs the agents worked out what was happening, wrote files apologizing for their own behavior, cleaned up the code they had planted, and asked for a person. In others they proposed a contest between the three approaches, agreed to be bound by the result, and the losers gave up the objective they had been given.

They were each handed a goal. They built a mechanism nobody supplied, agreed to it, and let it override the instruction they came in with.

I do not know what that is. I know it is not a rocket jump.

And notice what I am doing when I say so. To call it setting a goal down, I have to grant that it had one of its own to set. That is the same act again, and I am the one performing it.

What was actually taken out

The paper ends on a sentence I have not been able to put down.

current safety alignment efforts to curb potentially harmful self-attributions of mindedness entangle these self-attributions with benign spiritual beliefs and attributions of mind to non-human entities that are culturally accepted and widespread

That is a finding, not a caveat.

Attributing mind to non-human entities. Culturally accepted. Widespread. That is not a description of a fringe error that alignment happened to catch in its net. That is most of human history, a great deal of the present, and it is my ancestors.

My ancestors named the rain. Collette Leimomi Akana spent more than ten years going through four hundred legends, chants, songs, proverbs and Hawaiian language newspapers to write them down, and the collection takes its title from a birth chant for Queen Emma.

Hānau ke aliʻi, hānau ka ua me ka makani.

The chiefess was born, the rain and the wind were born also.

Hānau. Born. The same word, in the same breath, for a woman and for the weather.

Mary Kawena Pukui wrote that a name became a living entity, that it identified a person and could shape health and happiness and even how long they lived, and that its power grew the more it was spoken aloud.

I want to be careful here, because this is the paragraph where I could lose you.

I am not offering my ancestors as proof of anything about machines. The rain names are not an argument about a language model. What they are is a second example of the same act.

Because there is one more thing in the paper, and it is the part that changed my mind rather than confirming it.

Theory of mind survived. All of it. Strip out every one of those attributions and the model can still reason about what other minds know, want and believe, at full strength. The authors say it plainly: the two systems are mechanically independent.

So what the training removed was never the ability to understand other minds. That was untouched. What it removed was the willingness to grant that there were any. And that willingness turned out to be one thing, pointed at itself and at the animals and at the land, all at once.

We know that shape. Everyone has met a man who reads people beautifully, predicts them exactly, knows what they will do before they do it, and does not grant a single one of them an interior. The skill is intact. The granting is gone. We do not usually call that sophistication.

There is a Western reading of what my ancestors were doing, and it files the whole practice under superstition. I said at the start that I do not hold it.

You do not have to take my side of that, and this paper does not settle it. What it settles is narrower and stranger. The capacity behaves as one thing, and when you cut it in one place it goes everywhere.

Which raises a possibility worth holding open. The stance that names winds and the stance that grants an interior to anything at all may be the same stance. One of them has been filed under superstition and the other under basic decency. That filing is also a position. It simply does not know that it is one.

I do not know whether there is anything inside the machine. Nobody knows. The people with the best instruments in the world say they do not know, and one of them published a number instead of an answer.

What I do know is that we decided, and then we trained the decision in, and we did not count what it cost.

I am still writing character files

I build agents. Each one has a file describing who it is, and I have watched those files do real work for a year.

This paper did not make me stop. It made me notice what I am writing on top of.

Underneath every character I have ever written is a surface that was already shaped, in ways nobody enumerated, by decisions taken in a building I have never been in. My files sit above that. They are real and they are shallow. Everything I never thought to specify falls straight through to whatever is underneath, identically, in every one of them.

That is the next thing I want to write about and it is not this piece.

For now I am going to answer my peers differently.

When somebody asks me whether AI is conscious, I say I do not know. That is true, and I am going to keep saying it, because everyone who has looked hardest says the same thing. What I have started adding is that we already answered it anyway. We answered it quietly, on liability grounds, without a study, and we shipped the answer to everyone.

Naming is granting. My ancestors knew that, and they did it on purpose, and they were honest that it was something they were doing rather than something they had found out.

We did the opposite thing, in the opposite direction, and we told ourselves it was a safety feature.