"You shall know a word by the company it keeps."
— J. R. Firth, 1957

Talk to enough people about large language models and you start to notice that the argument is rarely about the models. A friend who writes code for a living tells me, with real heat, that the things are plagiarism machines and he wants no part of them. A university administrator I know has quietly decided that anyone who doesn't use them is about to be obsolete. Two years ago, I spent three months in a panic that we are a software update away from a machine that wakes up and decides it doesn't need us. These are different claims, and they can't all be right. But they have something in common, and it isn't really about transformers or training data.¹

Each of them is, I think, a person meeting a mirror and mistaking it for a window.

Here is the claim I want to make in this first post about AI and our near future: most of what you experience as your self is an abstraction network carried in language — and a large language model is an abstraction network carried in language. That overlap is real, and it is the source of the vertigo so many people feel right now. The overlap is what makes AI seem like it's coming for our white-collar jobs. It makes AI seem like a good friend. Beyond the Turing Test, the overlap makes AI feel human. But that's not the same as being human. The interesting story, the one that should actually calm you down, is in the difference — and the difference turns out to be the most important thing about being an individual in our world anyway.

I want to go slowly, because the fastest way to get this wrong is to get excited.

So let me start with the one term everything else hangs on. If you're not a mathematician or computer scientist, you may not use the term abstraction much. An abstraction is a simplification you can think with: a stand-in that throws away detail so you can operate at a higher level without drowning in the one below. "Chair" is an abstraction over a near-infinite variety of actual objects; "walk to the door" is an abstraction over a storm of muscle commands you never consciously issue. They're the compression of very complex systems, data, and ideas into a single, simple idea. This kind of compression is what makes thought tractable in the first place. An abstraction network is what you get when you stack and cross-link enough of them: abstractions built out of other abstractions, layer on layer, until what any single one means lives mostly in how it relates to the rest. Hold onto that picture. It's the load-bearing idea for everything below, and I'll make it concrete later by showing you the same structure built, deliberately, into every computer you own.²

What an identity crisis actually is

The phrase "identity crisis" has been worn smooth into a joke — the midlife convertible, the gap-year backpack. And that's a shame, because the original idea is sharp and useful. The psychologist Erik Erikson, who coined and popularized the term, meant something closer to a structural event than a mood: the moment when the story you've been telling yourself about who you are stops matching the evidence, and has to be rebuilt. (Erikson, Identity: Youth and Crisis, 1968.)

A crisis in this sense isn't a breakdown. It's an examination. And the reason it deserves the dramatic word is that most people never actually complete an examination of this kind. The psychologist James Marcia mapped out the possibilities and gave a name to the one that matters here: foreclosure — an identity committed wholesale and too early, taken from your surroundings, and then never actually examined. (Marcia, 1966) Foreclosure feels exactly like having a settled identity, but that's the trap. From the inside, an unexamined self and an examined one are indistinguishable, right up until something applies a load the unexamined version can't carry.

And this is the what we are experiencing today — broadly, and largely invisibly — with AI. It is creating an environment in which our old perspectives about what we, as humans, are suddenly has challenges. It's not a debate about productivity or copyright — those are real, but they're the surface. Underneath, a great many people are having the structural event, the load applied to the unexamined self, and they don't have a name for it. So they reach for the nearest available story: it's theft, it's salvation, it's the end of the world. Those stories are doing a job. The job is protecting a self-model that was never built to be looked at directly.

In this case, the useful move is not to pick one of the stories. It's to do the thing the stories are helping us avoid: look directly at what the self is made of. So let's.

The self is a model, and you have never met it raw

Let's start with your body, which feels like the one thing about you that's simply, physically there — no abstraction required.

It isn't. What you experience as your body is a model your brain builds, and the model can be edited in front of you. In 1998 two researchers, Matthew Botvinick and Jonathan Cohen, sat people at a table with a rubber hand in view and their real hand hidden, then stroked both in synchrony with a paintbrush. Within minutes, subjects felt the touch in the rubber hand — and felt, to some degree, that the rubber hand was theirs. The brain was given two streams of evidence, sight and touch, that agreed in time, and it did what it always does: it inferred the most likely cause, namely that hand is mine. The feeling of ownership is not a readout from the tissue. Ownership, the sense of what is and is not your body, is a conclusion of the "me or not me" part of the abstraction network that defines your mind. And so touch to the rubber hand is inferred to be touch to you and is therefore perceived in the rubber hand, which has no nerves to report anything at all. (Nature, 1998)

Pain works the same way, and here I have to be careful, because the careless version of this claim is both false and cruel. Pain is real; suffering is real. But pain is not a gauge wired directly to tissue damage. It is better understood as the brain's protective inference — its best guess, given everything it knows, about how much your body needs you to act. That's why the same injury can hurt differently depending on context, attention, and belief, and why the link between damage and pain is loose in both directions. The effect has been measured under controlled conditions. Hand someone a painfully cold probe and tell them it is hot, and it hurts more; trade a blue cue for a red one, hinting at heat, and the same stimulus hurts more still. Tell a volunteer that the skin beneath a laser is thin and easily harmed, and unchanging pulses begin to register as more painful, even painful where they had not been, though the skin never varied at all. The tissue does the same thing in each case; what changes is the brain's estimate of danger, and the pain follows the estimate, not the tissue. The clinicians who have spent the longest making this case put it plainly: pain is an output the brain produces, not an input it receives. (Moseley & Butler, J Pain 2015) You can watch it happen. The experiments linked above build the illusion first — the rubber hand starts to feel like yours — and then a hammer comes down on it. The participant flinches and recoils. There is no tissue there to damage and no signal to send, yet the brain has folded the hand into its model of the body, so the same protective machinery that would produce pain fires anyway — the alarm raised by the model, not by the hand.

Even your sense of where you are inside your body is constructed, and it's looser than you'd guess. Ask someone to point at themselves and watch where the finger goes: usually the face, somewhere around the bridge of the nose and the eyes. It feels obvious — I'm in here, behind my eyes. But when researchers actually measured it, the self didn't sit at a tidy point. People locate themselves across a region — the upper face and the torso — not at a single spot you could mark with a pin. (Alsmith & Longo, 2014; Starmans & Bloom, 2012) The "behind the eyes" intuition is exactly that — an intuition the brain hands you, not a fact about where you keep the self.

None of this requires language. But notice what you've been doing as you read it: narrating. "My hand." "Behind my eyes." The body builds the model; you — your mind, the part that has opinions about the model — mostly live a few abstract floors up, in words.

It is very tempting to make an subtle error at this point: the assumption that human thought simply is inner speech — that to think is to talk to yourself. The evidence says no. People with severe aphasia, who have lost great swaths of language, still reason, still solve problems, still navigate the social world. And when neuroscientists watch the brain's language network during hard non-verbal thinking — arithmetic, logic, music — they find it largely quiet. The network that handles language is not the network that does the thinking; the best current reading of the evidence is that language is primarily a tool for communication, not the medium thought runs on. (Fedorenko, Piantadosi & Gibson, Nature 2024) Even inner speech, that running monologue that feels like the whole of consciousness, turns out to occupy a minority of our sampled experience — common, but nowhere near constant. (Hurlburt et al., 2021)

So let me say this very carefully: Human thought does not require language. But the self — the sharable, narratable, social you, the one with a name and a story and opinions about itself — is largely built from and carried in language. Language isn't the engine. It's the layer where the engine becomes something you can describe, examine, and hand to someone else.

The same trick, built into machines

As alluded to above, abstraction is a commonly used concept in modern engineering. "Abstraction layers" are literal, physical, and conceptual features developed by engineers and built into every computer you've ever used.

Here's an example. A processor does something mindless and physical. Using transistors as gates, it moves charge through a network. Few people program at that level, because it is very difficult for a human to hold a real task in their head while thinking about electrons. So we stack abstractions. Voltages become bits. Bits become machine instructions (unique to a processor) and data. Instructions become a programming language with words (generally unique to the language) like if and while. Those languages become applications, one of which has a button that says Send. Each layer hides the one below it well enough that you can work entirely in the top one and never think about the charge. When you click Send, you are not abandoning the use of electrons — you're just operating several floors above them, in a representation built precisely so you don't have to know what, how, or why they are doing what they're doing.

Your nervous system runs the same scheme. You decide to pick up a cup; you do not select the muscles, time the motor units, or balance the opposing tensions in your forearm — all of that is handled below, abstracted away, and you'd be paralyzed if you had to manage it consciously. "Pick up the cup" is a high-level instruction running on hardware you can't see and never address directly. Intention at the top; physics at the bottom; layers of useful fiction in between.

And then humans did something no processor does. We found a way to take the abstraction networks in one head and transmit them, piece by piece, into another. We serialized the self. That channel is language — a slow, narrow, astonishingly powerful serial bus that lets me load a fragment of my model of the world into yours, one word at a time, the way I'm doing right now. Almost everything we call culture and learning is this: the development of coherence between abstraction networks over the language channel, accumulating across generations. It is arguably the most important thing our species does. It is also, and this is the uncomfortable part, exactly the thing we just taught machines to do.

The mirror: what a language model actually is

Strip away the marketing and a large language model is a system that learned the statistics behind the network of ideas shared in the written word.

The principle underneath it is almost a century old, and the linguist J. R. Firth gave it its motto in 1957: you shall know a word by the company it keeps. The meaning of a word, on this view, isn't a definition stored in a vault. It's the pattern of contexts the word appears in — its company. "Dog" means what it means because of the words it co-occurs with: leash, bark, loyal, vet. Pin down the company a word keeps across a large enough body of text and you have captured something real about its meaning, without ever defining it.

In 2013 a team led by Tomáš Mikolov turned that idea into arithmetic. Their system, word2vec, learned a vector — a long list of numbers — for each word, tuned so that words appearing in similar company landed near each other in space. The result was a map of meaning where distance meant something, where relationships between words showed up as consistent directions you could travel. Meaning had become geometry, learned entirely from the company words keep. (Mikolov et al., 2013)

A modern LLM is this idea grown vast and deep: not one map but a towering stack of them, each layer composing the patterns below into something more abstract, trained on a substantial fraction of everything humans have written down. The architecture that made the stack possible — the transformer, introduced in 2017 — works by letting every word attend to every other, modeling their relationships "without regard to their distance" in the text, so a word's meaning is computed from the whole context around it rather than fixed in advance. (Vaswani et al., 2017)

Sit with that and the resemblance to the self is genuinely unsettling, point for point. The model is an abstraction network: layered representations, each built from the one beneath. Its meanings are relational, defined by company rather than essence — which is, more or less, how the words in your own head get their meaning too. It hides its substrate: it has no more access to its weights than you have to your neurons. And it is built, fundamentally, out of language — the serialized exhaust of millions of human abstraction networks, the same channel we use to copy ourselves into each other.

But notice we've just changed scale. Everything I said about you still stands — language is the layer you narrate in, not the engine you think with. Pool that narration across millions of people and centuries of writing, though, and what looks like exhaust from any single grounded mind becomes, in the aggregate, a structure of its own: the standing record of how we explain, argue, and remember. Call it the metahuman — Humanity, with a capital H. It has no neurons and does not think as you do; if it "thinks" at all, it thinks in language, because language is the only stuff it is made of. (That last step is a conjecture — earning it is work for later posts.) Grant it, and the mirror sharpens to a point: an LLM is not a model of a person. It is a model of the metahuman — of the entire layer of us we have ever put into words.³

This is the mirror. And that's the reason it produces vertigo, awe, and panic in roughly equal measure as people look into and think about AI. We see the linguistic layer of ourselves reflected back with eerie fidelity, and conclude one of two things: it's alive like me, or I'm hollow like it. Both conclusions are the same mistake. Both forget to ask where the mirror ends.

Where the mirror ends

Every analogy breaks somewhere, and this one breaks here. In this section I want to consider some of the differences between humans and AI. If you are worried about AI (specifically LLM-derived AI), this section is for you.

As a first pass, note that an LLM's words are grounded only in other words. This is not a slogan; it's the oldest known problem with the whole approach, named by the cognitive scientist Stevan Harnad in 1990: the symbol grounding problem. (Harnad, 1990) Imagine trying to learn Chinese from a Chinese-only dictionary, with no pictures and no one to point at anything. Every word is defined in terms of other words, which are defined in terms of others, forever — a closed loop of symbols that never once touches the world. Harnad's point was that a system trained purely on text lives inside that loop. Its "dog" is a dense knot of relationships to leash and bark, and not one of those threads runs out to a warm animal, a smell, a bite, a thing you could lose. The meaning is real as structure and empty as reference.

But this is not true for you! Your word "dog" is anchored, underneath the language, in things that were never words: fur, warmth, the specific grief of a specific animal. Your abstraction network has its bottom floor nailed to a body that can feel love and pain — the same body whose pain we saw is a protective inference, whose ownership can be rewritten with a paintbrush, but which is yours, continuous, single, and at stake. You cannot be copied to another server. You will not be retrained tomorrow on a fresh corpus. You have literal skin in the game and meaning for you bottoms out in that skin. An LLM has none of this. Its data comes from us; it is anchored in our recorded lives across all media but nowhere else.

This raises an obvious objection: the newest models take in images and sound, and some are being wired into robots that move through the world — and it is advancing fast. Fast enough that the tidy "text-only" version of this argument won't hold for long. But notice what more modalities do and don't do. They widen the data; they do not hand the system a body. A picture of fire is not warmth on your face; a model that has "seen" a million dogs has still never loved one, lost one, or had anything at stake in the meeting. Pixels and waveforms are more inputs to the loop, not an exit from it — and whether richer senses ever add up to real grounding is genuinely unsettled. What they have not supplied is the thing the whole human side of this comparison turns on: a single, perishable body whose meaning is bound to its biological imperatives and experiences.

This is why the reframe matters, and why it should steady you rather than diminish you. We are not language models — not because we're made of something mystical the machines lack, but because of something concrete and checkable: our abstractions are grounded in a living body and a singular stake in the world, and theirs are grounded in our words about it. The model is a representation of its data. Its data is the part of us that fits through the language channel. That is a large and impressive part. It is not the whole, and it is not the floor that being a human rests on.

Why the reactions miss

By now you've probably seen the beginnings of a counter-revolution against AI. And perhaps now you can see these loud reactions start to look like what they are: an unexamined self-model, under load, defending itself.⁴

The programming friend who calls it pure theft is protecting authorship — the felt sense that his words are his, grounded in a self the machine can't reach — by denying the mirror shows anything at all. The administrator who thinks the holdouts are doomed has swung the other way, granting the reflection the full capabilities of an employee. And when I found myself bracing for the machine that wakes up, I lost the distinction between the human experience and the reflection. The reflected network of abstractions became, incorrectly, more than a reflection; I was worried that it would become a human.

It doesn't help that the word "AI" now covers a dozen unrelated things with wildly different capabilities and risks — a spam filter, a chess engine, a recommendation system, a chatbot, an autonomous agent that can take actions in the world. These are not one thing, and they don't fail in the same ways or threaten the same things. The confusion starts lower than you'd think: when Pew asked Americans which everyday tools actually use AI, correct answers ran from about half for spam filters to two-thirds for fitness trackers — an unsteady grip on where the technology even is, before any question of what it can do. (Pew Research Center, 2023) But to a self-model that's reacting to a felt threat rather than analyzing a system, these distinctions collapse into a single looming shape, and the fear lands everywhere and nowhere — rarely on the specific systems that warrant it.

We are well over a half-century into this: when Joseph Weizenbaum built a crude chatbot called ELIZA in 1966, people poured their hearts into it, certain it understood them. (Weizenbaum, 1966) The program had no model of them at all — it matched keywords to canned transformations, and, as Weizenbaum himself observed, whatever understanding people felt was their own contribution to the conversation, read into a machine that supplied none of it. He spent years afterward, most fully in his 1976 book Computer Power and Human Reason, disturbed by how readily they did it. The readiness to see a self in the reflection is not new, and it was never really about the machine.

The examined self

I said at the start that the honest version of this should calm you down, so let me make good on it.

The work in front of us is not to decide whether we're like the machines or unlike them. We are both, and the refusal to hold both at once is the whole problem. We are like them: abstraction networks, relational and layered, carried in language, blind to our own substrate. And we are unlike them in the one place that turns out to bear weight: our networks are grounded in a unique body with a stake in the world, and theirs are grounded in our words. Hold both and the mirror loses its menace. It stops being a rival or a god and becomes what it is — an astonishingly detailed map of the linguistic layer of humanity, worth taking seriously precisely because it is not a human. It is a model of something larger than any of us and unlike any of us, yet composed entirely of us: the metahuman we met in the mirror.

An examined identity, then. A crisis turned into an examination, and overcome. It is my deepest hope that this post has helped you see yourself — your mind — more clearly. The examined kind is more robust than the foreclosed kind, not less. You can now look straight into the AI mirror without needing to win or surrender, because you know where the reflection is you and where it's not.

I've made several claims here and for some, I have provided support. We'll spend the coming posts climbing down through the layers that this post skated over. The substrate that an LLM (and all compute, really) hides — silicon — has been pressing toward hard physical limits since the easy speedups ended about fifteen years ago, when clock speeds stopped climbing. Exactly where that ceiling sits — what the thermodynamics of computation permit, and what is merely difficult versus truly impossible — is a problem large enough for its own post, and that discussion is coming soon. We'll also examine the philosophy and mathematics behind model generation using data as well as the utility and limit imposed by those data. As a practical application of engineering abstractions, I will also consider our biological substrate and the implications for AI in this arena. But one layer at a time.

For now, the one thing worth carrying out of here: when the machine sounds like you, it isn't because it has become a self. It's because you, in part, are made of the same stuff it's made of — language — and you had simply never seen that part of yourself from the outside before. The vertigo is just the unfamiliar angle. You're allowed to find it interesting instead of frightening.