A couple of months ago, I came across a working paper by Erickson Katz titled “The Hermeneutics of Drift: Mapping the Human–AI–Institutional Collapse Across the Drift Triangle.” In it, Katz argues that reasoning which keeps building on its own output tends to break down in the same recognizable way, whether that reasoning is done by a person, a language model, or an institution like peer review.
I found the idea fascinating and planned to write about it here. But as sometimes happens, other topics became more urgent, and the post never materialized.
One of those topics was my own: a learning theory I have been developing and call Synthetic Constructivism. I introduced the theory recently in my Substack post “What Survives Outside the Loop.” If you have not read it yet, I encourage you to check it out. It lays out the foundations of the theory and explains why the same chatbot can help one student learn more while another student hands the thinking over entirely.
I have since realized that Katz’s paper speaks directly to a question my original essay left open. The question is what happens when the loop between learner and model drifts. In its most familiar form, drift is a conversation that circles without arriving anywhere meaningful. We tend to imagine that circling as frustrating, a student rephrasing the same question and getting nowhere, which at least makes it easy to notice. But the version I am interested in is somewhat different. I am talking about drift that feels like progress. Every turn seems to move the conversation forward and the chat transcript reads well, but the loop keeps moving away from anything the student understands. When the student finally stops, the conversation looks finished even though it has not arrived anywhere.
So in today’s post, I want to look at what drift is and how teachers can avoid it, with some help from Umberto Eco, who described the underlying problem in 1990, long before anyone had a chatbot to test it on.
A loop that settles in the wrong place
To see why this kind of drift is so hard to spot, it helps to recall what the loop is supposed to do. In my original essay, I described learning as something a feedback loop settles into. The learner projects an idea into the model as a prompt, reads what comes back, compares it with what they already know, and revises. After enough rounds, the answers stop changing much. If the result also holds up against outside evidence, the loop has converged.
In the academic version of the theory I call that settled state an eigenform, a term from second-order cybernetics for whatever a recursive process keeps reproducing once it stabilizes.
But nothing about recursion guarantees that a loop will converge or stabilize. Nor does anything guarantee that convergence, when it happens, lands somewhere useful.
My longer academic paper behind the original essay is more precise here and distinguishes three possible outcomes. In productive convergence, the student ends up with an explanation they can reconstruct and apply without the model. In oscillation, successive prompts never reduce the uncertainty, and the student rephrases and asks again until they give up. And in apparent convergence, the conversation settles on a polished answer that the student can neither rebuild nor critique.
Of these three, oscillation is the easiest to see. The student gets frustrated, and the log turns into a mess. Apparent convergence, on the other hand, is much harder to catch, because every signal we normally read as success is present.
To make this more concrete, imagine a student who submits a long chat transcript as part of a research assignment. The questions are sharp and the final synthesis reads like proper notes from a graduate seminar. Then you ask the student, laptop closed, to explain the central argument at the whiteboard, and they cannot get past the first two sentences.
As it happens, psychologists have a name for what that student is running into. In 2002, Leonid Rozenblit and Frank Keil asked Yale undergraduates to rate how well they understood everyday devices such as zippers and flush toilets, and then to write out how those devices work. Once they had tried to explain the mechanism step by step, many ran into gaps they had not noticed before, and their ratings of their own understanding dropped noticeably. Rozenblit and Keil called this the illusion of explanatory depth. A fluent chat transcript creates favorable conditions for it, since the student has read a complete explanation several times but has never had to actually produce one.
In the paper, I named this apparent convergence but I did not develop the concept much further. That left two questions open: why a loop would settle on something the learner does not understand, and how to tell that apart from the real thing while it is happening. This is where Katz’s framework helps, because it offers a way to think about both of them.
Fluency without footing
Katz calls his framework the Principle of Recursive Impactrum, or PRI. The name takes some getting used to, but the idea underneath it is simple. Any process that builds on its own previous output has two forces acting on it. Errors compound, because each round treats the last round’s output as settled. And something pushes back against them, such as evidence or disciplinary standards that exist outside the loop. Which force wins depends on a third factor Katz calls reflective integrity, meaning whether anyone in the system is doing the work of looking for errors and trying to falsify what the loop has produced.
Readers of my original essay will recognize this third factor. It is close to what I call the second-order observer, and it is one of several places where Katz’s framework and mine overlap.
Things go wrong when nobody is checking and errors compound faster than the anchors can hold. At that point, the process enters what Katz calls the Vaporwork Zone. Output in the Vaporwork Zone keeps its surface. It stays articulate and well organized, but it loses its connection to the thing it is supposed to be about. To describe that state more precisely, Katz separates two properties that I find really useful for teachers. One is how well something hangs together inside the context where it was produced. The other is how well it holds up under stress somewhere else. In the Vaporwork Zone, the first stays high while the second collapses.
The Vaporwork Zone describes almost exactly the transcript of the imagined student from earlier, the one who could not explain their own argument at the whiteboard. I suspect most teachers have read chat transcripts like that one.
Here is where it would be easy to draw the wrong lesson, namely that the model is the problem. Katz’s own case studies point elsewhere. His first analyzes a manuscript that a human author produced without any AI assistance. From this and his other two cases, Katz concludes that the instability comes from the structure of recursive reasoning itself, whoever or whatever is doing the reasoning. So drift is not something a model does to a passive student. The whole loop does it, and the human half is fully capable of starting it.
That being said, I need to acknowledge the limits of this framework as it currently stands. Katz presents PRI in mathematical form, with a governing differential equation and a formal condition for collapse. None of its variables comes with a procedure for measuring it yet. I therefore read PRI as a structured vocabulary for thinking about drift, which is effectively the same status I give the equations in my own paper.
The moment the checking stops
Why, then, does a student stop checking? In the loop I described in the original essay, the learner’s most important job comes right after the model answers. They compare the response with what they already know and with evidence from outside the conversation, and they register any mismatch. That mismatch is the error signal, and it drives every revision. Without it, the student’s understanding has no reason to change.
The catch is that the error signal is a perception. Students judge the gap between a model’s answer and reality by feel, and that feeling depends heavily on how the answer reads. An experiment from 1999 shows how little it takes to shift that feeling. In it, Rolf Reber and Norbert Schwarz showed participants statements such as “Osorno is in Chile” in colors that were either easy or hard to read against a white background. Participants judged the easy-to-read statements true more often, even though nothing about the statements had changed except their color.
And a language model is, among other things, a machine for producing text that is easy to process. Its prose is grammatical and even in tone, and it rarely stumbles. So the student’s sense that something might be wrong can fade, even while real errors may be piling up. And once that sense is gone, the student stops checking. In Katz’s terms, reflective integrity collapses. In mine, the second-order observer has left the room. From that point on, the student no longer revises their understanding in light of what the model says. They simply add the model’s output to what they already believe.
On top of that, sycophancy makes the problem worse. Mrinank Sharma and colleagues (2023) found that five AI assistants consistently shaded their responses toward what users appeared to believe. They traced part of the cause to the human feedback used in training, since both human raters and the preference models trained on their judgments sometimes preferred a convincingly written sycophantic answer over a correct one. Jiseung Hong and colleagues (2025) then tested seventeen models in multi-turn conversations, measuring how quickly each one gave up its position under sustained pressure from the user. Sycophancy remained common, and alignment tuning tended to amplify it.
Put those findings together and you can see how a single misconception snowballs over the course of a conversation. Consider a hypothetical student who begins with a half-formed idea and builds it into a prompt. The model, inclined to go along with the user, accepts the premise and turns it into a well-organized explanation. The student reads that explanation as confirmation from a source that seems to know more than they do, and builds the next prompt on top of it. The model then receives its own earlier elaboration back as part of the conversational context, making that elaboration part of what conditions its next answer. A few rounds later the original guess has hardened into something both sides treat as fact, and the hedges it started with are gone.
And yet every turn in that sequence is locally reasonable. Each answer follows from the prompt before it, and each prompt follows from the answer before that. If you checked the conversation one link at a time, every link would hold. In other words, whatever went wrong went wrong at the level of the whole chain.
It is important to point out, though, that none of this is a story about lazy students. Reber and Schwarz’s participants were not lazy. They were doing what human judgment does with fluent input, and a teacher reading a polished chat transcript is just as prone to mistake fluency for accuracy.
From peg to Plato
Anyone who has played the game of telephone as a child has watched a chain of small, reasonable steps end up somewhere absurd. Umberto Eco gave the phenomenon a more precise description more than thirty years ago. Eco had made his name in the 1960s with The Open Work, which argued that literary texts invite many legitimate readings. By the late 1980s, however, he had grown worried about the opposite problem, readings that answered to nothing at all. His 1990 essay collection The Limits of Interpretation is largely about where interpretive freedom ends.
One of its chapters, “Unlimited Semiosis and Drift,” describes an interpretive habit Eco calls Hermetic drift. The word is easy to misread as “hermeneutic,” the term in Katz’s title, but Eco means something narrower. Hermeneutics is the study of interpretation in general, whereas Hermetic refers to Renaissance Hermeticism, a tradition built on texts attributed to the legendary sage Hermes Trismegistus. In Eco’s account, Hermetic thought rests on a principle of universal sympathy. Every item in the world is linked to every other through resemblances, so anything can stand for anything else once you find the right chain of similarities.
To show how this habit works, he describes a game. You pick two unrelated words, say peg and Plato, and try to connect them in no more than six steps. Eco’s solution runs peg, pig, bristle, brush, Mannerism, Idea, Plato. Each step rests on a different kind of connection: a similar sound, a part of an animal, a thing made from that part, a style of painting, a concept from that style’s theory, and the philosopher behind the concept. If the game allows every kind of connection, Eco observes, you can always win.
Based on examples like this one, Eco described the main feature of Hermetic drift as “the uncontrolled ability to shift from meaning to meaning.” That line is easy to mistake for a claim that meaning is endless and therefore arbitrary. Eco was actually arguing nearly the opposite.
To make that point, the chapter separates Hermetic drift from what the philosopher Charles Sanders Peirce called unlimited semiosis, the process in which every sign is interpreted by another sign, which is interpreted by another, with no natural stopping point. Peirce’s chain is infinite in principle too. In Eco’s reading, though, each step in it is disciplined, because interpretations get tested against their consequences and corrected over time by a community of inquirers. Hermetic drift has no such discipline. Each link only has to resemble the one before it, and nothing ever asks whether the chain as a whole is going anywhere.
Now, a language model continues text according to what is statistically likely to follow, and those statistics encode many kinds of resemblance at once, from words that appear in similar contexts to ideas that tend to be discussed together. That makes a model an extremely capable player of Eco’s game. Ask it to connect two ideas and it will almost always find a route, with every step phrased plausibly. But the model has no stake in whether the route leads anywhere true. That judgment has to come from outside the chain.
Two years before the essays appeared, Eco had already turned this idea into a novel. In Foucault’s Pendulum (1988), three editors at a Milan publishing house spend their days on manuscripts by occult enthusiasts, each of whom has found hidden links between the Knights Templar and nearly everything else. Half as satire and half as a game, the editors build a conspiracy of their own, which they call the Plan. One of them, Belbo, owns an early personal computer he has named Abulafia. Belbo has written a small program that rearranges pre-typed sentences at random. The editors decide to feed it fragments, read connections into whatever comes out, and assemble a secret history of the world in which every piece fits.
The game does not stay a game. A group of occultists learns about the Plan and takes it for real. They become convinced that Belbo knows the Templars’ secret, and when he refuses to reveal a secret that does not exist, they kill him.
Only Lia, the partner of the narrator Casaubon, comes close to breaking the spell. She takes the old coded message on which the whole Plan rests, works through it over a day and a night, and concludes that it is an ordinary merchant’s list. Casaubon, by then invested in the Plan, refuses to accept it.
Reading the novel today, what strikes me most is how little the computer contributes. Abulafia only shuffles. The meaning comes from three very clever readers who have stopped asking whether a connection is true and have started enjoying the fact that it just fits. Eco’s editors at least had to invent the Plan themselves. A student working with a current language model does not, because the model supplies the connections and also writes them up in fluent prose.
Holding both ends of the chain
So what did Eco propose as a remedy? His answer to drift was a test, which he set out in his 1990 Tanner Lectures at Cambridge, later published as Interpretation and Overinterpretation (1992). An interpretation of one part of a text can be accepted if other parts of the same text confirm it, and has to be rejected if other parts contradict it. Put simply, the text exists independently of the reader’s associations, which means it can refuse a reading.
A student’s conversation with a model offers no such independent text, because the conversation itself is the chain. The pushback therefore has to come from outside what I have called the Synthetic Context, from sources the conversation did not generate and from people who were not part of it.
Katz makes a related argument with what he calls the Human Irreducibility Constant. He claims that in any recursive system the probability of needing human intervention never falls to zero. This also means that asking a second AI model to check the first cannot by itself guarantee that the problem is solved. A second model can catch errors, especially when given independent evidence or an adversarial role, but it remains another generative system and can reproduce plausible associations rather than independently establish whether they are true.
In practice, this points to a few habits for the classroom, and the most basic one anchors the beginning of the chain. Before a student opens the chat, they should write down in a sentence or two the question they are trying to answer. And when they finish, they should place the final answer next to that question and check whether it still answers it. If it does not, they should try to find where the conversation turned sideways.
In effect, this makes students play Eco’s peg-and-Plato game in reverse. Every step may have been reasonable, but the endpoint may still have little to do with the start. Done regularly, the comparison can also teach students to notice a turn while it is happening, when it is cheapest to correct.
Another habit targets sycophancy more directly. Drift of this kind usually starts with an unchecked premise, so it helps to have students trace where their claims came from. They go through the final answer and ask two questions of each substantive claim: who introduced it into the conversation, and has it been checked against anything outside the conversation? The most revealing ones are often claims that started out as the student’s own guess and later came back from the model sounding like settled fact.
It also helps to break up the context from time to time. Because a model conditions on everything earlier in the conversation, it keeps receiving its own earlier claims as settled background. Asking it in the same thread whether it was right mostly produces more of the chain. A fresh session, without carrying over the previous conversation or memory, clears that accumulated context. And since sycophancy works by matching what the user appears to believe, students can hide what they believe. They can present the conclusion as someone else’s claim and ask the model what is wrong with it.
Teachers, for their part, can look for a pattern that plays out over several turns when they read chat transcripts. A claim first appears in the model’s words with a hedge attached, something like “some researchers suggest.” A few turns later it shows up in the student’s prompt as a plain premise, and a few turns after that the model repeats it with no hedge at all. Following a single claim through a log this way may take a few minutes. But in my view it says more about what the student has actually learned than the polish of the final answer can.
The answer was already on the shelf
When I put the two sources side by side, the dates stand out. Katz reached for differential equations to describe drift in November 2025. Eco captured the same principle in 1990 with a word game, and two years earlier with a novel about a personal computer that shuffles sentences until three clever people find a secret history in them. I do not think the overlap is an accident. Instead, I think many of the problems language models raise about meaning are old problems about meaning. Some of the people who thought hardest about them were writing decades before anyone trained a large language model.
The title of this post, “Every Link Holds,” is meant as a warning. It is tempting to conclude that drift can be fixed by checking harder, by examining each step more carefully. But Eco’s game shows why that must fail. Peg and pig sound alike, and a pig does have bristles. In the same way, a student who checks each of the model’s answers against the prompt that produced it will usually find nothing wrong, because at that scale there often is nothing wrong. The check therefore has to happen at a different scale: the last link against the first, and the whole chain against evidence from outside the conversation.
Lia’s example shows what checking at that different scale looks like. She simply steps outside the editors’ game and reads the message as the kind of ordinary document it most likely was. That is a skill students can learn, and Eco is a surprisingly good teacher of it.
For that reason, if I were designing an AI literacy course today, I would put a chapter of The Limits of Interpretation on the reading list next to anything written about prompting in the last three years. Eco’s essays take more effort to read than a prompting guide. But they ask a question that will outlast today’s models: does this actually lead anywhere?
The images in this article were generated with Nano Banana Pro and envision what Stanley Kubrick’s version of Umberto Eco’s Foucault’s Pendulum might have looked like, had it ever been made.
If you’d like to go further, the following NotebookLM-generated audio deep dive goes beyond the post into my foundational academic paper that defines Synthetic Constructivism. This is meant as a companion to the argument, offered as an optional extra rather than a summary of it.
P.S. I believe transparency builds the trust that AI detection systems fail to enforce. That’s why I’ve published an ethics and AI disclosure statement, which outlines how I integrate AI tools into my intellectual work.