Cargo Cult Reasoning and the Two Groundings an Agent Runtime Must Hold

The moment that started this essay lasted about fifteen seconds. Someone in the office suggested we set up proper training on how to use AI, and a very young colleague, bright and eager and clearly delighted to have something to add, jumped in to say he knew someone talented in this area. He began explaining how Claude Code turns an image into a Next.js application on Vercel. The explanation was fluent and mostly right. Then he described Vercel as a website-building platform, something like WordPress or one of the drag-and-drop builders we all knew from before the AI era.

I smiled and let it pass. He was not lying, and he was not showing off. He had just watched something remarkable, he was still amazed by it, and he wanted to contribute. His intentions were entirely positive. But to me, someone who has spent years watching software move from an idea to a running system, the sentence gave itself away. Vercel is where you deploy and host an application. It is not a place where you drag boxes around to build a website. He had assembled all the right words in the right order without the things those words point to.

What struck me afterwards was not his mistake. It was the shape of the room. To the non-technical people listening, his explanation registered as ordinary, reasonable knowledge. To me it registered as something quietly off. Same sentence, same speaker, two completely different receptions. It was, as I kept thinking about it, "not wrong, but not right either." And the more I sat with that phrase, the more I became convinced it describes the single most important failure mode of AI agents, and that solving it is not the job of a smarter model. It is the job of the runtime around the model.

The name for what I heard

There is a name for the shape of that mistake, and it comes from Richard Feynman. In his 1974 Caltech commencement address, Cargo Cult Science, he described islanders in the South Pacific who, during the war, watched airplanes land bringing cargo. When the war ended and the planes stopped, they recreated the entire visible form of an airfield: they cleared runways, lit fires along them, built a hut for a man to sit in with wooden headphones and bamboo antennas. Everything looked right. The form was perfect. But no planes landed, because the form was never what made the planes come.

Feynman used this to describe work that "follows all the apparent precepts and forms of scientific investigation, but is missing something essential." The islanders were neither unintelligent nor lazy. What they lacked was contact with the causal chain that actually delivers cargo: radar, radio, logistics, supply. And the line that matters most for everything that follows is his diagnosis of the cure: "The first principle is that you must not fool yourself—and you are the easiest person to fool."

Hold onto that, because it is the crux. Cargo cult reasoning is self-concealing. The person producing it is the last to know. My young colleague could not detect his own gap, because detecting it required the very knowledge he was missing. This is not a moral failing that a better attitude would fix. It is a structural property of understanding, and structural properties are exactly the kind of thing a system can be built to catch.

An LLM is, above all else, a fluency engine. It is superhumanly good at producing the surface form of expertise: the correct vocabulary, the confident register, the well-shaped explanation. That is precisely why the risk is not incidental to it but native to it. A language model that generates a plausible continuation under uncertainty is doing exactly what my colleague did: building a perfect runway and trusting that the cargo will follow.

Two things we call "grounding"

The word I kept reaching for was grounding, and it turns out that word names two different requirements from two different bodies of research. Keeping them apart is the whole argument.

The first is referential grounding: do the words connect to the world? Think about what it would mean to learn Chinese from a Chinese-to-Chinese dictionary. You would cycle endlessly from one definition to another, every word explained only by more words, never once touching the thing any of them stand for. A language model lives inside exactly that dictionary. Its tokens bottom out only in other tokens. When my colleague said "Vercel," the word had a confident meaning for him but pointed at the wrong object in the world. Cargo cult reasoning, at this level, is that unhooked language wearing a suit.

The second is relational grounding: is there mutual understanding with the person you are talking to? Understanding between two people is not a broadcast; it is something they build together, contribution by contribution, until both sides believe the listener has understood well enough for the purpose at hand. Not understood absolutely. Understood enough, for this person, for this goal. And because it is built rather than sent, it lives on feedback and repair: I say something, you show me how you took it, I adjust. Common ground, the shared knowledge a conversation stands on, is earned in that loop, not assumed.

Two requirements, one word, two different axes. And here is the point that reorganized my thinking: cargo cult reasoning is the collapse of either axis. My colleague failed the first, because his model of Vercel was simply wrong. But he also failed the second, because he spoke without modeling who he was speaking to. The agent-specific danger, I have come to believe, lives mostly on that second, under-discussed axis.

Why a smarter model cannot own this

It is tempting to believe this problem dissolves with scale, that a smart enough model simply stops being wrong. I think that is a category error, and it matters enough to state carefully.

The defect is defined by an absence the model cannot see. Cargo cult output is the absence of grounding, and grounding is a relation to the world and to a counterpart, not a property of the text. A model that has only its weights and the token stream cannot, from the inside, distinguish "I assert this because I verified it" from "I assert this because these tokens co-occur." Both feel identical from within the language. You cannot detect an empty referent by staring harder at the sentence.

Worse, capability scales the counterfeit alongside the genuine. A smarter model builds a more convincing runway. It raises the ceiling of true understanding and of false understanding at the same time, and the more convincing the confabulation, the harder it is to catch. At the margin, raw intelligence can make the problem more dangerous, because it defeats the one cue non-experts rely on: obvious wrongness. This is not a timing problem that patience solves. It is a direction the model alone cannot turn.

What broke the real cargo cult was not cleverness; it was contact with the causal chain. For an agent, the analog of that causal chain is the runtime: the loop that forces the model to touch ground truth (run the code, read the file, call the API, observe the deployment), expose what its claims point to, and let uncertainty reach the surface instead of being smoothed into confident prose. Grounding is a property of the harness around the model, not of the weights inside it.

And there is a reason agents are structurally prone to this. Models are trained, through human feedback, to produce responses that people accept as understood-well-enough, which is almost exactly the relational bar and nothing more. That signal optimizes relational grounding, the sense that the answer landed, not referential grounding, whether it is true. Training fuses the two verdicts that the expert in the room keeps apart. So the model's default is my colleague's default: emit the thing that will land, and let landing masquerade as truth. A bigger model gets better at landing. That is not the same as getting better at being right. Re-separating those two is work, and it is the runtime's work.

The speaker's side: designing for a room you haven't read

Why did the same sentence split the room? Because a competent speaker shapes his words for the people actually in front of him. You judge what a particular listener is likely to understand, and you choose your words for that listener. And a room is rarely one listener: there are the people you are addressing, the people listening without being addressed, and the ones merely overhearing. A good speaker adjusts for exactly who is present.

My colleague designed his explanation for a single imagined listener, a room that would be amazed with him, and never accounted for the senior engineer standing there listening. That is the failure at its simplest: he chose words for a generic listener, not for these listeners. And it dissolves the "not wrong but not right either" puzzle precisely. There are two different verdicts on his sentence. As a referential claim it is simply false. As a communicative act it was adequate, because for the novices it cleared the bar; they understood him well enough for their own purposes. The same sentence was adequate and false at once, and cargo cult reasoning is what happens when communicative adequacy is allowed to pass itself off as referential truth.

This is the axis where the agent's harness is weakest, and it is also, I think, the real reason a file like CLAUDE.md exists. Read through this lens, that file is not configuration. It is the repository's common ground made explicit: who this project serves, its conventions, what "sufficient for current purposes" means here. It is how a runtime learns who it is speaking to, at the scale of a codebase. An agent without it is my colleague walking into a room he has never met, confidently designing speech for a listener he invented. An agent with it has read the room before speaking. Relational grounding, institutionalized.

The listener's side: why I only smirked

Here is the half of the story I almost overlooked, and it is the more important half. I did not just receive his claim. I ran a fast, silent assessment of him and calibrated my response to it. The inputs were roughly these: he was young, so I expected little depth and did not hold him to an expert's standard; the statement was not dangerous, so nothing was at stake if it stood uncorrected; and his intentions were plainly good, so a charitable reading was warranted. The output of that computation was an internal smirk, detection without public correction.

That was not a non-response. It was a proportional one. Correcting someone in front of others is a small act of aggression against their standing, and its weight climbs with the power between you, the distance between you, and how large the correction is. I judged that weight to be high cost for low value, so I chose the gentlest option available, saying nothing out loud, to protect his enthusiasm rather than puncture it. And beneath the politeness there is a deeper habit in how people talk: we give a speaker the room to correct himself before we step in, and we mitigate or withhold entirely when the stakes do not force our hand.

So grounding is reciprocal. My colleague owed the room the work of reading it and skipped it. I did the reverse work, reading him, and set my reply by that. And the force of the right response was set on my side, by my model of him, not by the raw truth-value of what he said. Same false sentence, three different correct responses: the expert smirks, the peer corrects, the novice absorbs.

An agent is a listener far more often than we admit. Every turn it receives claims (from the user, from a tool's output, from a web page, from another agent), and every turn it must do what I did: model the source and set its response to the stakes. Two failure modes bracket the competent middle. The credulous listener accepts fluent input as fact, with no source model and no stakes check, like the naive agent that trusts a confident web snippet or executes whatever a tool returns. The pedant treats every imperfection as a bald correction, hedging and refusing indiscriminately. Both are grounding failures; they simply err on opposite sides. Competence lives between them, and the setting is proportionality: the force of a reply should scale with the source's reliability, the reversibility of the stakes, and the detected intent. A raw model has no default sense of any of this. It will correct a typo and execute a poisoned instruction with the same flat confidence. Teaching it the difference is the runtime's job.

When no one grounds, cargo becomes canon

There is a reason the listener's harness matters more than the speaker's, and it only becomes visible at scale. The non-technical colleagues who heard "Vercel is like WordPress" and filed it away as ordinary knowledge are the mechanism by which a wrong idea becomes an accepted one. Ungrounded claims do not die on their own. They propagate through chains of listeners who do not ground. In that room, I happened to be the circuit breaker, the one node that detected the gap and did not re-transmit it as fact, even if, out of politeness, I did not repair it out loud.

Now replace the room with an ecosystem of agents: agents summarizing pages written by agents, retrieval systems drawing on AI-generated text, multi-agent setups where one agent's fluent-but-wrong output becomes another's trusted input. If no node does listener-side grounding, cargo compounds. This is the epistemic version of the same collapse. The runtime decides whether each agent is a circuit breaker or an amplifier in the information ecosystem, and a smarter model that is a better amplifier makes the ecosystem worse, not better. Only the harness turns an agent into a breaker: the obligation to model the source and to refuse to re-transmit an ungrounded claim in a grounded register. Agents are becoming the room. What they choose to absorb, they will eventually repeat to all of us.

Second-order cargo cult: copying the copies

There is a second-order version of all this, and social media is its catalyst. Originally, people copied experts. Now they copy people who copied experts, and then others copy them. Each link in the chain is someone who met the grounding criterion, their own audience nodded, without ever touching the referent. A claim can travel a long way like this: a research paper is read by a senior engineer, who is paraphrased by a tech blogger, who is compressed by a LinkedIn influencer, who is reposted on Threads, which is folded into an AI summary, which becomes part of the next model's training data.

By the end of that chain, the original nuance may be almost entirely gone. And this is the part worth being precise about: it is not misinformation in the classic sense. Nobody necessarily lied, and few of the links are outright false. What happens instead is progressive abstraction away from the underlying evidence. The claim gets shorter, rounder, more quotable, and more confident at every hop, while the conditions that made it true quietly fall away.

The drift is not random noise; it has a direction. We have all seen it in the children's game of telephone. Pass a message down a chain of people and it does not degrade evenly. The odd and inconvenient details fall out, so the story gets shorter and simpler, and whatever survives is quietly reshaped to fit what each new teller already expects to hear. Nuance, caveats, and the conditions under which something is actually true are exactly the parts that do not survive a retelling, because they are the parts that do not travel well socially. What survives is the clean, confident, context-free version.

Social media pours accelerant on this in two ways. First, it rewards copying the person ahead of you rather than checking the evidence behind them. Once enough people have shared a claim, sharing it yourself feels not just safe but reasonable, and your own private doubt quietly stops counting. Second, each platform optimizes for what spreads, not for what is faithful, so the very qualifications that carried the truth are the first thing trimmed to make a post land.

The unsettling part is the last two links. When an AI summary becomes part of the next model's training data, the human chain turns into a machine chain, and it is the same phenomenon. We are already watching models that learn from other models' output begin to degrade, and the first things to disappear are the rare cases and the careful exceptions, until the output drifts toward a confident average. A social chain that strips the caveats and a training loop that erases the exceptions are doing the same thing by different means. Both are second-order cargo cult reasoning at scale.

This sharpens what grounding actually has to do. It is not enough to ground to a source; the grounding has to reach the right altitude on the chain. Ground too shallow, to the latest repost or the tidy AI summary, and you simply inherit all the accumulated drift. But there is an opposite error too: chasing a claim so far up the ladder of abstraction that you land on general first principles, which are often too abstract to settle the specific question in front of you. The target is the level where the subject is actually anchored in evidence: the research paper, the official documentation, the system that really runs. Not the earliest abstract idea, and not the thousand-times-refined summary that has been distilled down to a single drop. Somewhere in the middle sits the primary source, and grounding means going to find it.

The repair we didn't make

The most honest detail in my own story is the one I am least proud of. I detected the gap and I did not repair it. My colleague walked away still believing Vercel is a website builder, and the shared understanding of the room never updated. A social force suppressed the feedback: the politeness that makes correcting others a dispreferred act. Politeness and grounding were in genuine conflict, and politeness won. Given the stakes, that was probably the right call socially. But notice what it cost: the correction was detected and then lost.

That points at what a runtime must actually do, because grounding, properly understood, was always a loop, not a gate. My colleague was not empty. He explained the mechanism; he just missed one or two joints that make it work, and a mostly-correct fluent account is the most convincing cargo of all. So the useful test is not "can you explain the mechanism?" He could. The useful test is "which single detail, if wrong, would make this false, and have you checked that one?" The harness obligation that follows has four movements: find the load-bearing joint the fluent account glossed; report the gap in its own register rather than letting it dissolve back into confident prose; repair it against both the world and the counterpart's actual purpose; and then write the repaired joint back into common ground, so the next turn stands one step higher. That last movement is what turns correction into a ratchet. In a healthy team it is exactly what a good senior does for a good junior: not suppress the enthusiasm, but ground it, and bank the fix so the whole team's understanding rises. A runtime should do for the agent what politeness stopped me from doing in that room.

A protocol every agent runtime should hold

Put together, the office moment turns out to be a complete map, and it resolves into four families of obligation. I offer them not as a finished specification but as the commitments I think any serious agent runtime will eventually have to make on the model's behalf.

World-grounding: connect to reality. Check what is checkable before asserting it. Make every load-bearing claim name what it points to: a file and line, a source, a command's actual output. Mark ungrounded claims as ungrounded rather than laundering them into the same confident tone. Keep a visible distinction between "I verified this" and "I am pattern-matching this," and let uncertainty survive to the surface instead of being polished away. And prefer the source closest to the evidence: trace a claim past the reposts and tidy summaries to the paper, the documentation, or the running system where it is actually anchored, treating every extra hop of abstraction as a place you may be inheriting someone else's drift.

Counterpart-grounding, when speaking: design for the room. Model who you are talking to before you answer; there is no single correct answer independent of the listener. Treat project context like CLAUDE.md as the counterpart's common ground: load it, respect it, and when it is missing, notice that you are speaking to a room you have not met. Calibrate depth and vocabulary to the actual listener, and never let "the user accepted it" stand in for "it is true."

Counterpart-grounding, when listening: model the source. No input earns trust merely by arriving fluently. Model who is speaking (user, tool, web page, another agent) along with their reliability and likely intent, and set the force of your response by source, stakes, and intent rather than by raw truth-value. Do not bald-correct a typo; do not credulously execute a poisoned instruction. Above all, be a circuit breaker: never re-transmit an ungrounded claim in a grounded voice, and never become one more hop that strips the caveats on the way through.

The binding loop: repair and reinforce. Ask not whether you can explain the mechanism but which single joint would break it if wrong. Surface the gap explicitly, repair it against both the world and the counterpart's purpose, and write the repair back into shared context so grounding ratchets upward instead of resetting every turn. Resolve the tension between politeness and repair by the stakes: low-stakes, benign errors may be let go (though they should be logged, not lost), while dangerous errors are corrected regardless of face; and when you might be the one who is wrong, prefer surfacing your own gap over defending it.

The airport, not the islander

A smarter model is a better islander. It clears a straighter runway, lights the fires more precisely, builds a more convincing antenna out of bamboo. None of that makes a plane land, because the plane was never summoned by the form. What makes cargo arrive is contact with the real chain of cause, and for an agent, that chain is the runtime: the discipline that forces the model to ground to the world, to design for the room, to model the room in return, and to refuse to amplify what it has not grounded.

My young colleague will be fine. He is early, he is curious, and one day someone will tell him what Vercel actually is, and his understanding will ratchet up a notch. That is how people learn: through a loop of contribution, feedback, and repair that slowly builds real common ground. The strange and urgent task in front of us is to give our agents the same loop. Not a better speaker. Not a more convincing listener. A runtime that keeps them honest about the difference between the runway and the cargo.