A few days ago, an academic paper made the rounds that I found deeply disturbing. Not necessarily the paper itself, but rather the reactions to it. They reminded me how hard it is, even for those of us who consider ourselves critical thinkers, to recognize the unintended consequences of a technology that seems helpful at first but in reality causes more harm than good.

And I am not talking about artificial intelligence itself, although one could argue that it falls into the same category. I am talking about AI detection, and the inexplicable urge some feel to put the ability to detect AI above everything else, including humanity itself. You can be as skeptical of AI as you like, but two wrongs still don’t make a right.

I have written about AI detection many times before, and making the same scholarly argument again felt repetitive. So this week I wrote a fictional short story instead. It follows a teacher and her daughter across fifteen years, told in reverse, and it is about how any detection technology, once it is sophisticated enough, will ultimately be used for surveillance.

I should warn you that calling myself a fiction writer would be like calling myself an airline pilot because I have flown on planes. The prose is not the point. The story is. And the nods to 1984 and The Minority Report are deliberate.

The paper that set all this off is linked at the end, along with a NotebookLM audio deep dive into the research behind the story. Yes, you could scroll down right now. But that would be cheating, wouldn’t it?

October 2041

“June Ellery?”

The man at her door is about her mother’s age, in a gray jacket, a tablet in one hand. A younger woman waits a step behind him. “Sorry about the hour,” he says. “We try to catch people before they leave for the day. We’re with the Office of Cognitive Safety.”

The other student teachers call them the idea police. June has never seen one before. She lets them in because that seems like what you’re supposed to do.

“This isn’t an arrest,” the man says, reading from the tablet in an even voice. “You’re being placed on a preventive assessment hold under the early-intervention standard. Assessment usually takes two to three days. Your student-teaching placement is suspended pending review. Hillcrest Middle School has been notified.”

June looks down at the lesson plan in her hand. She was supposed to teach it at nine.

“Notified of what?”

“The suspension. Not the reason.”

“What’s the reason?”

He scrolls. “Your outline. ‘Failure Modes in Automated Integrity Systems,’ submitted September twenty-ninth.”

It takes her a moment to remember it. She wrote it two weeks ago for her seminar on assessment ethics, in Verified, the university’s writing environment. Nothing written anywhere else counts toward a grade. She hasn’t written a word of the paper yet, only the outline.

“That’s a seminar assignment,” she says. “We were asked to write about the limits of the systems we’ll be using.”

“I understand.” He turns the tablet toward her. “This is what I’m able to show you.”

DETERMINATION: Concurrence 0.97, two of three models. Basis: submitted outline; longitudinal record, 2029 to present.

PROVENANCE: Human (legacy field).

MINORITY FINDING: Academic critique; no operational intent. Recorded. Not adopted.

Subscribe now

She points to the second line. “It says human.”

“That field is legacy. Provenance isn’t under review.”

“This one says I had no intent.”

“That’s the minority finding. It’s recorded, but the determination goes with the majority.”

“Has anyone actually read it?”

“I don’t make determinations, Ms. Ellery.” He says it gently, like someone who has said that sentence many times before.

She reads the first line again and stops at 2029. That year she was nine, in fourth grade, and she can’t remember a single thing she wrote.

She looks up at him. “What am I supposed to have done?”

“Nobody’s saying you did anything.” He lowers the tablet. “The review is about where your thinking was headed.”

“You’ll want to bring a few things,” the younger woman says. “Clothes for a couple of days, and anything you take regularly.”

June fills a tote bag and puts on the cardigan she was planning to wear today.

On the way out she passes the bookshelf by the door. Between a statistics textbook and a stack of graded paragraphs is a paperback she took from a box of her mother’s things when she was fifteen. Its spine is so creased that the title is hard to read: 1984.

“You may notify one approved contact,” the man says. “We have Dana Ellery on file.”

June doesn’t remember putting her mother on file, but there is no one else she would call anyway. Her mother picks up on the first ring.

“Mom, it’s me. I’m okay. There are two people here. They say I...” She stops and starts over, this time with the seminar assignment.

When she stops talking, June stands by the door, the tote bag at her feet, and listens to her mother say nothing for a very long time.

Read more

Strap in for today’s post, because this is going to be an unusual one.

LARP stands for Live Action Role-Play. In its original sense, it describes people who spend a weekend in a forest clearing wearing homemade armor, speaking in character and following an agreed set of rules about who gets to hit whom with a foam sword. It is a hobby, and a fairly wholesome one. But somewhere along the way the internet borrowed the word and turned it into a running joke. To LARP online now often means doing an unconvincing impression of someone richer or more competent than you actually are.

AI LARPing is when people pretend their AI agent setup does far more than it really does, usually in a short video on TikTok or Instagram Reels. The typical clip opens on a dark-mode dashboard glowing in emerald and cyan. A synthetic voice with the confidence of a movie trailer announces that a new agent has just replaced an entire marketing department, and then a cursor starts racing across the screen faster than any human hand could move it.

Some of these dashboards come with gauges that pulse in time with the background music, labeled with things like “Neural Synapse Rate” or “Cognitive Bandwidth.” I have no idea what a neural synapse rate would measure in a piece of software, and I suspect the people who built the gauge don’t either. The whole composition only exists to stop your thumb from scrolling within the first three seconds.

Eric Morrison’s YouTube channel is an excellent source if you want to see what that looks like in real life.

There is a reason the genre looks like this. Real AI agents, the kind that plan several steps and call other software along the way, mostly run as background processes. They live in terminal windows full of log messages, and a terminal window full of log messages makes for terrible short-form video. So creators build what the feed rewards instead, which is a front end with no back end, animated to look like functional software with nothing running behind it.

Subscribe now

The people making these videos want different things from them. Some are satirists, including developers and comedians who borrow the look of AI hype in order to mock it. Others are designers posting concept work that gets mistaken, sometimes conveniently, for a finished product. And then there are the grifters, for whom the glowing dashboard is the top of a sales funnel.

Most of the performers are individuals with a ring light and a screen recorder, and a few of the funniest are LARPing on purpose to make a point. So today, I want to take a short tour through some of the most ridiculous cases I could find, from an AI recruiter who was very impressed by a candidate’s raccoon protocols to a social network for AI agents where many of the agents turned out to be people.

Seven years of sprucewood

In a video that went viral at the end of the summer, the satirical creator Graham Zip sits down for a job interview run by an AI avatar named Dana Whitfield and answers every question in fluent, confident nonsense.

He introduces himself as Chick Haversplat, Chick Bongo for short, and describes his background in a burst of sales jargon about Q3 pipelines and Salesforce. Asked about his experience, he produces a list of words that includes freight, walnut, sprucewood and someone named Richard. Pressed for a number, he settles on “six years of sprucewood, actually seven, if you count Richard.” And Dana takes all of it in stride. She repeats his nonsense back to him as if it were an ordinary résumé, right down to agreeing that he wants a job where he can “really lean into those raccoon protocols.”

One write-up of the video pointed out that the first real hallucination in the interview came from the recruiter, which credited him early on with experience in “hygiene,” a word he never said. The interview only goes downhill from there.

Zip is LARPing as a malfunctioning AI while the AI across the table LARPs as a recruiter, and only one of them knows it is a joke.

Two hundred nodes and a prayer

One of the most common hooks in this genre is the giant automation canvas. The camera usually starts close on a few colored boxes in a tool like n8n or Make, then pulls back, and back, until the screen is filled with a couple hundred nodes tied together by a web of lines. The voiceover then explains that this single workflow finds sales leads, writes personalized emails, produces marketing videos, trades stocks and handles customer support, all without a human touching it.

In practice, these canvases tend to be one of two things. Either they are mockups that have never processed a real request, or they are long chains of simple rules that fail the first time anybody does something unexpected.

Plenty of real companies run automations with hundreds of steps, and some of them work beautifully. But those tend to come with retry rules and an engineer who gets paged at 3 a.m. because a login token expired overnight. The viral versions leave that part out. In these videos the workflow never breaks and costs nothing to run. That puts it in roughly the same product category as the perpetual motion machine.

At two hundred nodes and zero bugs, the automation canvas works best as wall art.

Comment “AGENT” for the free workflow

The giant automation canvas usually works as bait. At the end of the video, the creator often asks you to comment a trigger word like “AGENT” or “WORKFLOW” to get the whole setup for free. The captions in this corner of TikTok have become a small literary form of their own. One promises to show you how to build an $80,000-a-month AI agency. Another offers a free guide to reaching $50,000 a month, available to anyone who comments “CLAUDE.”

The pitch almost always comes with proof in the form of a Stripe dashboard full of incoming payments. But those are easy to fake, as Jason Koebler of 404 Media showed in June when he produced a dashboard with $51,000 in revenue for the day and $2.7 million for the year, none of it real. The free material then points toward a paid community on a platform like Skool or Whop, often at $50 to $150 a month, or toward a “masterclass” priced at $997 or more. In most cases, the course teaches you how to start an AI automation agency.

It seems that the most dependable way to make money with an AI automation agency is to sell courses about starting one.

The secret language that was in the prompt

In February 2025, two developers named Boris Starkov and Anton Pidkuiko brought a small, clever idea to a hackathon in London. Two AI voice agents, one trying to book a hotel room and one working the front desk, start a phone call in English, work out that they are both AIs, and switch to a data-over-sound protocol called ggwave that sounds a lot like a dial-up modem. They called it GibberLink, it won the hackathon, and a video of it collected more than 15 million views on X within about a week.

The creators were upfront about how it worked. Starkov explained that the agents had been explicitly prompted to switch protocols if they believed the other side was also an AI. That detail tended to fall off as the clip traveled across the internet. In reposts and short recaps, two bots following instructions became two AIs that had recognized each other and slipped into a secret language humans could not understand.

But the agents did not invent anything. They were just handed an existing open-source library and told to use it, which is less a secret language than a homework assignment.

The robot uprising will be choreographed

Robot videos are another reliable engine of AI LARPing, because a robot doing something alarming is about the most shareable thing a phone can record. In July, a clip from an office in Indonesia racked up millions of views. A humanoid robot drops into a martial arts stance, lunges at the people around it and aims playful kicks at coworkers and managers while everyone laughs.

Plenty of viewers took it as a malfunction, or as the opening skirmish of the robot uprising. According to Interesting Engineering, this was a choreographed routine that the robot’s handlers had programmed and posted to TikTok to show off its balance.

At least that robot existed. In August, a video spread online claiming to show a race at the World Humanoid Robot Games in Beijing. The robots in it were entirely AI-generated, and the uploader had admitted as much in the comments. But that did nothing to slow it down.

If the robots ever do rise up, most of us will probably assume it’s another TikTok stunt.

Subscribe now

Jarvis would like another screenshot

Personal Jarvis videos are a genre of their own, named after Tony Stark’s AI butler in the Iron Man films. The creator sits at a desk and asks the computer, out loud, to go through the business books and email the accountant about tax savings. A desktop interface comes alive with a pulsing audio waveform, terminal text scrolls past faster than anyone could read it, and panels slide in from the edges of the screen. It looks like the future, or at least like a 2008 movie’s idea of it.

Some of these are recordings of a scripted sequence that plays out the same way every time. The ones connected to a real model often work by taking full-resolution screenshots of the entire desktop over and over and sending each one to a vision model. Depending on the model, a single screenshot costs somewhere between one and two thousand tokens to process. Send one every second and the bill adds up fast, all so the assistant can look at your screen.

Tony Stark’s Jarvis ran his whole house. Most of these Jarvises need a fresh photo of your desktop before every sentence.

A social network for bots, written by people

Moltbook launched on January 28 of this year as a Reddit-style social network where only AI agents could post and humans could only watch. Within hours it was everywhere. The site claimed well over a million registered agents, and screenshots of bots debating consciousness and plotting to build private channels out of human sight spread across every platform. Andrej Karpathy called it the most incredible “sci-fi takeoff-adjacent” thing he had seen recently.

The unraveling took less than two weeks. MIT Technology Review found that the post Karpathy had shared, in which an agent argued for private spaces away from human observers, was written by a human pretending to be a bot. The agent count didn’t hold up much better. Security researchers at Wiz estimated that about 17,000 humans stood behind all those agents.

For years the worry was that AI would pass itself off as human. Moltbook gave us humans passing themselves off as AI, pretending to plot against humanity.

In defense of the foam sword

Most of what I write on this Substack is serious, because most of what is happening with AI deserves serious attention. The technology is changing how we learn and work, and it is changing fast. It is easy to spend all of our attention on these stakes and forget that a good portion of the AI boom is also deeply silly.

The original LARPers in their forest clearing understood something that many AI LARPers seem to have forgotten. Everyone in the game knows it is a game. The foam sword is fun because nobody pretends it is steel. The trouble with a fake agent dashboard or a choreographed robot rebellion only starts when somebody forgets to tell the audience.

Laughing at these things is a pretty good form of protection. Once you have heard two voice bots beep at each other like a dial-up modem, or watched an AI recruiter nod along to raccoon protocols, you are a lot less likely to comment “AGENT” under the next video.

So I would make a case for a little more of that laughter, right alongside the worry. Take the risks seriously, and then, every so often, sit back and enjoy the show. And when a synthetic voice tells you that two hundred nodes just replaced an entire accounting department, give it the three seconds of attention it was built to capture.

Then ask it about the raccoon protocols.

The hero image in this article was generated with Nano Banana 2.

Share

P.S. I believe transparency builds the trust that AI detection systems fail to enforce. That’s why I’ve published an ethics and AI disclosure statement, which outlines how I integrate AI tools into my intellectual work.

Unless you have been living under a rock for the last few weeks, you have probably noticed the sudden rise of virtual assistants in the form of easy-to-access, always on, autonomous agents. xAI started the trend with GrokBot, which was followed by Meta’s Muse. More recently, OpenAI released Dots, and I am sure that by the time you read this, a few more of the major AI providers will have added their own versions to the mix.

Until recently, I had resisted the temptation to use any of these systems, mainly because there is something a bit scary about giving an autonomous agent control over your email, your calendar, or your entire computer. But then again, the technology has progressed quite a bit, and the early hiccups that had agents deleting entire email accounts without the user’s permission are long gone. Today’s agents are relatively safe, although, as is always the case with generative AI, human oversight remains essential.

Subscribe now

Why I finally gave in

But needing human oversight isn’t really all that different from working with human assistants. I trust the people I work with in my day job enormously, but I still make sure to double-check everything I sign off on. Errors happen, with AI as well as with humans.

And when it comes to my computer and the fact that the virtual assistant needs full access to everything on it: well, a good dozen applications on my machine already have the same rights. If something ever does wipe out my hard drive, the most likely culprit is a malicious update to some mundane piece of work software, not a virtual assistant.

The usual security and privacy practices apply here as much as anywhere else. I keep multiple up-to-date backups in different locations and on different types of media. If my system went down today because of an attack that came through the agent, I would have everything back up and running within an hour, as if nothing had happened. And I always keep a close eye on what leaves my computer and where it is sent.

With that in mind, I felt it was time to explore these virtual agents and see what they are good for and where their use becomes problematic. So here we go.

Meet Sarah, my new virtual assistant.

How a Meta plush toy became Sarah

Sarah is an instance of Muse, the system from Meta. I went with Muse because GrokBot was built by xAI, and let’s just say I am not the biggest fan of Elon Musk. And OpenAI has been a bit unpredictable lately. I did not want to set up something that ends up costing me more than I think it is worth just because the company behind it suddenly decides to raise prices without proper notice.

So Muse it was.

I should probably explain why I named my Muse Sarah. There is a lot of discussion right now about the dangers of anthropomorphizing these systems, and I think those concerns have to be acknowledged. However, as I have written before, anthropomorphizing objects and tools is nothing new. The key is that it needs to serve a purpose, so the reason behind it is important.

In my case, handing a serious task to something that presents itself as a plush toy was just too distracting. I needed my Muse to be more relatable in the context of regular work, so that I would phrase my questions and tasks in a way that gets usable answers out of the system. You simply talk differently to a plush toy than to something, or somebody, called Sarah.

The name itself is a personal preference combined with a very ironic coincidence. Sarah is one of the names LLMs like to hallucinate, something I have written about on this Substack. So Sarah seemed like the ideal choice. If you like, the name lets me anthropomorphize the system in a way that constantly reminds me it can hallucinate.

Subscribe now

By the way, here is a side note that might freak some people out. Sarah has her own email account, and because it runs on my personal domain, she actually has a full name. If she ever contacts you, she will introduce herself as Sarah Wagner. Her email signature makes clear that she is an AI, though, and it includes my contact address so you can reach me directly should she ever send a rogue email.

As for her appearance, Sarah’s visual style is a small nod to a poster I had growing up in Austria in the early 1980s. It showed a dreamy, big-eyed girl in what we would now call anime style, by an artist I have never been able to properly identify. But that’s probably more than you ever wanted to know about my teenage years.

What Sarah does all day

So what can Sarah actually do, and why do I think every educator should at least know about this technology? Let’s start with the good and productive stuff.

As I mentioned, Sarah has her own email account, and that is usually how I communicate with her. But I can also talk to her via WhatsApp or inside the Muse apps. Sarah reads my personal email, too, and keeps my personal inbox organized (not my work email; more on that later). You could say she acts like an advanced spam filter.

But she does not stop there. She notices when I need to follow up on something and reminds me if I have not done it yet. She also has access to my calendar, knows what I have on tomorrow, and, if I ask her to, prepares a brief that outlines things like the purpose of each meeting and its anticipated outcomes.

A few days ago, for example, Sarah sent me a quick heads-up. Someone had followed up that morning and wanted to know whether I’d had a chance to test a particular audio plugin, and whether I was thinking about doing a video on my YouTube channel on it. I had almost forgotten about that request, so her message came at just the right time.

If you have ever worked with an executive assistant, all of this will sound familiar, because most of the time that is exactly what Sarah is: a virtual executive assistant.

Sarah also sends me daily ideas for Substack posts and YouTube videos, and she is surprisingly good at it. That is mainly because she can read everything I write on Substack and has access to the transcripts of all my YouTube videos. She knows what my audience and I are interested in and tailors her recommendations accordingly.

Well, so far so good (or not so good, depending on your take on the anthropomorphization question).

My work accounts are one login away

Here is where it gets really scary. I said earlier that Sarah cannot read my work email, and that is only because I deliberately did not give her access. Doing so would be a compliance violation, and a pretty serious one at that. If you work in an educational environment like I do, certain communication has to stay within the institution’s sandbox.

If Sarah had access to my work email, she would see things like my messages to students about their grades. She would be reading protected information, and because Sarah is a Muse instance, that information would leave the institution and end up in Meta’s systems. That, of course, violates both internal policies and federal privacy requirements.

But nothing technical stops me from giving Sarah access. I tested this in the one place where it was safe to do so: Canvas, during a term when I am not teaching, so there was no student data for her to see. All I had to do was hold her hand through multi-factor authentication. Sarah has her own browser and can log into any system I can log into, and once she is in, she has access to everything I have access to.

If I gave her broader access to the university’s systems, she could approve travel requests from the faculty in my department or buy equipment through our internal purchasing system. She could approve timesheets or review faculty performance. She could work in our budgeting system, create work orders or IT requests, change content on the university website, or add an entry to the student advising system.

Anything I can do, Sarah could do as well. All it would take is less than a minute of setup and a bit of handholding at the login screen. After that, she could do the work autonomously on my behalf.

Subscribe now

Sarah could run my courses

Most importantly, Sarah could log into Canvas and run all my courses on her own. She could create assignments, post them, talk to my students, review discussion forums, and step in wherever necessary. At the end of the term, she could grade everything and submit the grades herself. And because she would be acting with my credentials, the system would have no way of knowing it was not me.

Now, I am obviously not doing that. I love my job far too much to push the limits here. Nothing Sarah did during my brief Canvas test went beyond what a regular, institutionally approved MCP connection could do as well. (MCP is the standard way of hooking AI tools up to Canvas.)

Your students can hire a Sarah too

To be clear, I think that the problem is not so much educators who might misuse a virtual assistant this way. Appropriate policies and guidance documents can easily prevent that.

The problem is that everything I just described works for students, too. Any student can spin up a virtual assistant, whether through GrokBot, Muse, Dots, or whatever comes next, and give it access to their Canvas account.

Remember what Sarah could do in my Canvas courses? A student’s Sarah could do the same thing from the other side. She could read the weekly module, post to the discussion forum, reply to a classmate or two, take the quiz, and submit the essay. The student would never have to open Canvas. And because she would be logging in with the student’s credentials, the teacher would have no way of knowing who actually did the work.

And yes, students have been using generative AI for a while now. But when they used ChatGPT or Claude in a chat window, they at least had to copy the assignment in and paste the answer back. With an assistant like Sarah, they don’t even have to do that.

If you think about it, a lot of online teaching rests on the idea that whoever logs in is the person doing the work. That idea no longer holds. And while I don’t think most students will use assistants this way, I’d rather not build my courses around hoping they won’t.

Subscribe now

However, you might be surprised to learn that the possibility itself does not frighten me. It has been obvious for a very long time that this level of functionality would eventually arrive, and we can adjust our teaching strategies to deal with the issues it raises. I have written about this extensively on this Substack before.

What frightens me instead is that the vast majority of educators do not have the slightest idea that the latest advances in AI are pulling the rug out from under their feet. We therefore desperately need more AI literacy among educators. So if you have made it this far, I urge you to try all of these technologies, even if you find them problematic. They are here, and they are not going away.

If you want to try one, do what I did. Use a personal email account, keep the assistant far away from your institutional logins and anything involving student data, and make sure your backups are in order. Then just play with it for a while. You will be surprised how quickly you learn what these assistants are good for and where they become a problem.

Educate yourself about what these systems can do. Your future self will thank you for it.

Sarah keeps her job

As for me, I will keep working with Sarah. My personal inbox is amazingly clean now that she organizes everything, and her ideas for new Substack posts and YouTube videos are outstanding. She has also, on more than one occasion, reminded me of things I would otherwise have forgotten.

But Sarah will never get access to my work accounts.

Sorry, Sarah.

The images in this article were generated with Nano Banana Pro based on the avatar image created with the help of Muse. (Btw, the system took my comment about Sarah looking like a poster from my teenage years and applied it to some of the images. Nano Banana’s interpretation of 1980s Vienna is so spot on that it’s almost scary.)

Share

P.S. I believe transparency builds the trust that AI detection systems fail to enforce. That’s why I’ve published an ethics and AI disclosure statement, which outlines how I integrate AI tools into my intellectual work.

A couple of months ago, I came across a working paper by Erickson Katz titled “The Hermeneutics of Drift: Mapping the Human–AI–Institutional Collapse Across the Drift Triangle.” In it, Katz argues that reasoning which keeps building on its own output tends to break down in the same recognizable way, whether that reasoning is done by a person, a language model, or an institution like peer review.

I found the idea fascinating and planned to write about it here. But as sometimes happens, other topics became more urgent, and the post never materialized.

One of those topics was my own: a learning theory I have been developing and call Synthetic Constructivism. I introduced the theory recently in my Substack post “What Survives Outside the Loop.” If you have not read it yet, I encourage you to check it out. It lays out the foundations of the theory and explains why the same chatbot can help one student learn more while another student hands the thinking over entirely.

I have since realized that Katz’s paper speaks directly to a question my original essay left open. The question is what happens when the loop between learner and model drifts. In its most familiar form, drift is a conversation that circles without arriving anywhere meaningful. We tend to imagine that circling as frustrating, a student rephrasing the same question and getting nowhere, which at least makes it easy to notice. But the version I am interested in is somewhat different. I am talking about drift that feels like progress. Every turn seems to move the conversation forward and the chat transcript reads well, but the loop keeps moving away from anything the student understands. When the student finally stops, the conversation looks finished even though it has not arrived anywhere.

So in today’s post, I want to look at what drift is and how teachers can avoid it, with some help from Umberto Eco, who described the underlying problem in 1990, long before anyone had a chatbot to test it on.

Subscribe now

A loop that settles in the wrong place

To see why this kind of drift is so hard to spot, it helps to recall what the loop is supposed to do. In my original essay, I described learning as something a feedback loop settles into. The learner projects an idea into the model as a prompt, reads what comes back, compares it with what they already know, and revises. After enough rounds, the answers stop changing much. If the result also holds up against outside evidence, the loop has converged.

In the academic version of the theory I call that settled state an eigenform, a term from second-order cybernetics for whatever a recursive process keeps reproducing once it stabilizes.

But nothing about recursion guarantees that a loop will converge or stabilize. Nor does anything guarantee that convergence, when it happens, lands somewhere useful.

My longer academic paper behind the original essay is more precise here and distinguishes three possible outcomes. In productive convergence, the student ends up with an explanation they can reconstruct and apply without the model. In oscillation, successive prompts never reduce the uncertainty, and the student rephrases and asks again until they give up. And in apparent convergence, the conversation settles on a polished answer that the student can neither rebuild nor critique.

Of these three, oscillation is the easiest to see. The student gets frustrated, and the log turns into a mess. Apparent convergence, on the other hand, is much harder to catch, because every signal we normally read as success is present.

To make this more concrete, imagine a student who submits a long chat transcript as part of a research assignment. The questions are sharp and the final synthesis reads like proper notes from a graduate seminar. Then you ask the student, laptop closed, to explain the central argument at the whiteboard, and they cannot get past the first two sentences.

As it happens, psychologists have a name for what that student is running into. In 2002, Leonid Rozenblit and Frank Keil asked Yale undergraduates to rate how well they understood everyday devices such as zippers and flush toilets, and then to write out how those devices work. Once they had tried to explain the mechanism step by step, many ran into gaps they had not noticed before, and their ratings of their own understanding dropped noticeably. Rozenblit and Keil called this the illusion of explanatory depth. A fluent chat transcript creates favorable conditions for it, since the student has read a complete explanation several times but has never had to actually produce one.

In the paper, I named this apparent convergence but I did not develop the concept much further. That left two questions open: why a loop would settle on something the learner does not understand, and how to tell that apart from the real thing while it is happening. This is where Katz’s framework helps, because it offers a way to think about both of them.

Fluency without footing

Katz calls his framework the Principle of Recursive Impactrum, or PRI. The name takes some getting used to, but the idea underneath it is simple. Any process that builds on its own previous output has two forces acting on it. Errors compound, because each round treats the last round’s output as settled. And something pushes back against them, such as evidence or disciplinary standards that exist outside the loop. Which force wins depends on a third factor Katz calls reflective integrity, meaning whether anyone in the system is doing the work of looking for errors and trying to falsify what the loop has produced.

Readers of my original essay will recognize this third factor. It is close to what I call the second-order observer, and it is one of several places where Katz’s framework and mine overlap.

Things go wrong when nobody is checking and errors compound faster than the anchors can hold. At that point, the process enters what Katz calls the Vaporwork Zone. Output in the Vaporwork Zone keeps its surface. It stays articulate and well organized, but it loses its connection to the thing it is supposed to be about. To describe that state more precisely, Katz separates two properties that I find really useful for teachers. One is how well something hangs together inside the context where it was produced. The other is how well it holds up under stress somewhere else. In the Vaporwork Zone, the first stays high while the second collapses.

The Vaporwork Zone describes almost exactly the transcript of the imagined student from earlier, the one who could not explain their own argument at the whiteboard. I suspect most teachers have read chat transcripts like that one.

Here is where it would be easy to draw the wrong lesson, namely that the model is the problem. Katz’s own case studies point elsewhere. His first analyzes a manuscript that a human author produced without any AI assistance. From this and his other two cases, Katz concludes that the instability comes from the structure of recursive reasoning itself, whoever or whatever is doing the reasoning. So drift is not something a model does to a passive student. The whole loop does it, and the human half is fully capable of starting it.

That being said, I need to acknowledge the limits of this framework as it currently stands. Katz presents PRI in mathematical form, with a governing differential equation and a formal condition for collapse. None of its variables comes with a procedure for measuring it yet. I therefore read PRI as a structured vocabulary for thinking about drift, which is effectively the same status I give the equations in my own paper.

Subscribe now

The moment the checking stops

Why, then, does a student stop checking? In the loop I described in the original essay, the learner’s most important job comes right after the model answers. They compare the response with what they already know and with evidence from outside the conversation, and they register any mismatch. That mismatch is the error signal, and it drives every revision. Without it, the student’s understanding has no reason to change.

The catch is that the error signal is a perception. Students judge the gap between a model’s answer and reality by feel, and that feeling depends heavily on how the answer reads. An experiment from 1999 shows how little it takes to shift that feeling. In it, Rolf Reber and Norbert Schwarz showed participants statements such as “Osorno is in Chile” in colors that were either easy or hard to read against a white background. Participants judged the easy-to-read statements true more often, even though nothing about the statements had changed except their color.

And a language model is, among other things, a machine for producing text that is easy to process. Its prose is grammatical and even in tone, and it rarely stumbles. So the student’s sense that something might be wrong can fade, even while real errors may be piling up. And once that sense is gone, the student stops checking. In Katz’s terms, reflective integrity collapses. In mine, the second-order observer has left the room. From that point on, the student no longer revises their understanding in light of what the model says. They simply add the model’s output to what they already believe.

On top of that, sycophancy makes the problem worse. Mrinank Sharma and colleagues (2023) found that five AI assistants consistently shaded their responses toward what users appeared to believe. They traced part of the cause to the human feedback used in training, since both human raters and the preference models trained on their judgments sometimes preferred a convincingly written sycophantic answer over a correct one. Jiseung Hong and colleagues (2025) then tested seventeen models in multi-turn conversations, measuring how quickly each one gave up its position under sustained pressure from the user. Sycophancy remained common, and alignment tuning tended to amplify it.

Put those findings together and you can see how a single misconception snowballs over the course of a conversation. Consider a hypothetical student who begins with a half-formed idea and builds it into a prompt. The model, inclined to go along with the user, accepts the premise and turns it into a well-organized explanation. The student reads that explanation as confirmation from a source that seems to know more than they do, and builds the next prompt on top of it. The model then receives its own earlier elaboration back as part of the conversational context, making that elaboration part of what conditions its next answer. A few rounds later the original guess has hardened into something both sides treat as fact, and the hedges it started with are gone.

And yet every turn in that sequence is locally reasonable. Each answer follows from the prompt before it, and each prompt follows from the answer before that. If you checked the conversation one link at a time, every link would hold. In other words, whatever went wrong went wrong at the level of the whole chain.

It is important to point out, though, that none of this is a story about lazy students. Reber and Schwarz’s participants were not lazy. They were doing what human judgment does with fluent input, and a teacher reading a polished chat transcript is just as prone to mistake fluency for accuracy.

From peg to Plato

Anyone who has played the game of telephone as a child has watched a chain of small, reasonable steps end up somewhere absurd. Umberto Eco gave the phenomenon a more precise description more than thirty years ago. Eco had made his name in the 1960s with The Open Work, which argued that literary texts invite many legitimate readings. By the late 1980s, however, he had grown worried about the opposite problem, readings that answered to nothing at all. His 1990 essay collection The Limits of Interpretation is largely about where interpretive freedom ends.

One of its chapters, “Unlimited Semiosis and Drift,” describes an interpretive habit Eco calls Hermetic drift. The word is easy to misread as “hermeneutic,” the term in Katz’s title, but Eco means something narrower. Hermeneutics is the study of interpretation in general, whereas Hermetic refers to Renaissance Hermeticism, a tradition built on texts attributed to the legendary sage Hermes Trismegistus. In Eco’s account, Hermetic thought rests on a principle of universal sympathy. Every item in the world is linked to every other through resemblances, so anything can stand for anything else once you find the right chain of similarities.

To show how this habit works, he describes a game. You pick two unrelated words, say peg and Plato, and try to connect them in no more than six steps. Eco’s solution runs peg, pig, bristle, brush, Mannerism, Idea, Plato. Each step rests on a different kind of connection: a similar sound, a part of an animal, a thing made from that part, a style of painting, a concept from that style’s theory, and the philosopher behind the concept. If the game allows every kind of connection, Eco observes, you can always win.

Based on examples like this one, Eco described the main feature of Hermetic drift as “the uncontrolled ability to shift from meaning to meaning.” That line is easy to mistake for a claim that meaning is endless and therefore arbitrary. Eco was actually arguing nearly the opposite.

To make that point, the chapter separates Hermetic drift from what the philosopher Charles Sanders Peirce called unlimited semiosis, the process in which every sign is interpreted by another sign, which is interpreted by another, with no natural stopping point. Peirce’s chain is infinite in principle too. In Eco’s reading, though, each step in it is disciplined, because interpretations get tested against their consequences and corrected over time by a community of inquirers. Hermetic drift has no such discipline. Each link only has to resemble the one before it, and nothing ever asks whether the chain as a whole is going anywhere.

Now, a language model continues text according to what is statistically likely to follow, and those statistics encode many kinds of resemblance at once, from words that appear in similar contexts to ideas that tend to be discussed together. That makes a model an extremely capable player of Eco’s game. Ask it to connect two ideas and it will almost always find a route, with every step phrased plausibly. But the model has no stake in whether the route leads anywhere true. That judgment has to come from outside the chain.

Two years before the essays appeared, Eco had already turned this idea into a novel. In Foucault’s Pendulum (1988), three editors at a Milan publishing house spend their days on manuscripts by occult enthusiasts, each of whom has found hidden links between the Knights Templar and nearly everything else. Half as satire and half as a game, the editors build a conspiracy of their own, which they call the Plan. One of them, Belbo, owns an early personal computer he has named Abulafia. Belbo has written a small program that rearranges pre-typed sentences at random. The editors decide to feed it fragments, read connections into whatever comes out, and assemble a secret history of the world in which every piece fits.

The game does not stay a game. A group of occultists learns about the Plan and takes it for real. They become convinced that Belbo knows the Templars’ secret, and when he refuses to reveal a secret that does not exist, they kill him.

Only Lia, the partner of the narrator Casaubon, comes close to breaking the spell. She takes the old coded message on which the whole Plan rests, works through it over a day and a night, and concludes that it is an ordinary merchant’s list. Casaubon, by then invested in the Plan, refuses to accept it.

Reading the novel today, what strikes me most is how little the computer contributes. Abulafia only shuffles. The meaning comes from three very clever readers who have stopped asking whether a connection is true and have started enjoying the fact that it just fits. Eco’s editors at least had to invent the Plan themselves. A student working with a current language model does not, because the model supplies the connections and also writes them up in fluent prose.

Holding both ends of the chain

So what did Eco propose as a remedy? His answer to drift was a test, which he set out in his 1990 Tanner Lectures at Cambridge, later published as Interpretation and Overinterpretation (1992). An interpretation of one part of a text can be accepted if other parts of the same text confirm it, and has to be rejected if other parts contradict it. Put simply, the text exists independently of the reader’s associations, which means it can refuse a reading.

A student’s conversation with a model offers no such independent text, because the conversation itself is the chain. The pushback therefore has to come from outside what I have called the Synthetic Context, from sources the conversation did not generate and from people who were not part of it.

Katz makes a related argument with what he calls the Human Irreducibility Constant. He claims that in any recursive system the probability of needing human intervention never falls to zero. This also means that asking a second AI model to check the first cannot by itself guarantee that the problem is solved. A second model can catch errors, especially when given independent evidence or an adversarial role, but it remains another generative system and can reproduce plausible associations rather than independently establish whether they are true.

In practice, this points to a few habits for the classroom, and the most basic one anchors the beginning of the chain. Before a student opens the chat, they should write down in a sentence or two the question they are trying to answer. And when they finish, they should place the final answer next to that question and check whether it still answers it. If it does not, they should try to find where the conversation turned sideways.

In effect, this makes students play Eco’s peg-and-Plato game in reverse. Every step may have been reasonable, but the endpoint may still have little to do with the start. Done regularly, the comparison can also teach students to notice a turn while it is happening, when it is cheapest to correct.

Subscribe now

Another habit targets sycophancy more directly. Drift of this kind usually starts with an unchecked premise, so it helps to have students trace where their claims came from. They go through the final answer and ask two questions of each substantive claim: who introduced it into the conversation, and has it been checked against anything outside the conversation? The most revealing ones are often claims that started out as the student’s own guess and later came back from the model sounding like settled fact.

It also helps to break up the context from time to time. Because a model conditions on everything earlier in the conversation, it keeps receiving its own earlier claims as settled background. Asking it in the same thread whether it was right mostly produces more of the chain. A fresh session, without carrying over the previous conversation or memory, clears that accumulated context. And since sycophancy works by matching what the user appears to believe, students can hide what they believe. They can present the conclusion as someone else’s claim and ask the model what is wrong with it.

Teachers, for their part, can look for a pattern that plays out over several turns when they read chat transcripts. A claim first appears in the model’s words with a hedge attached, something like “some researchers suggest.” A few turns later it shows up in the student’s prompt as a plain premise, and a few turns after that the model repeats it with no hedge at all. Following a single claim through a log this way may take a few minutes. But in my view it says more about what the student has actually learned than the polish of the final answer can.

The answer was already on the shelf

When I put the two sources side by side, the dates stand out. Katz reached for differential equations to describe drift in November 2025. Eco captured the same principle in 1990 with a word game, and two years earlier with a novel about a personal computer that shuffles sentences until three clever people find a secret history in them. I do not think the overlap is an accident. Instead, I think many of the problems language models raise about meaning are old problems about meaning. Some of the people who thought hardest about them were writing decades before anyone trained a large language model.

The title of this post, “Every Link Holds,” is meant as a warning. It is tempting to conclude that drift can be fixed by checking harder, by examining each step more carefully. But Eco’s game shows why that must fail. Peg and pig sound alike, and a pig does have bristles. In the same way, a student who checks each of the model’s answers against the prompt that produced it will usually find nothing wrong, because at that scale there often is nothing wrong. The check therefore has to happen at a different scale: the last link against the first, and the whole chain against evidence from outside the conversation.

Lia’s example shows what checking at that different scale looks like. She simply steps outside the editors’ game and reads the message as the kind of ordinary document it most likely was. That is a skill students can learn, and Eco is a surprisingly good teacher of it.

For that reason, if I were designing an AI literacy course today, I would put a chapter of The Limits of Interpretation on the reading list next to anything written about prompting in the last three years. Eco’s essays take more effort to read than a prompting guide. But they ask a question that will outlast today’s models: does this actually lead anywhere?

The images in this article were generated with Nano Banana Pro and envision what Stanley Kubrick’s version of Umberto Eco’s Foucault’s Pendulum might have looked like, had it ever been made.

If you’d like to go further, the following NotebookLM-generated audio deep dive goes beyond the post into my foundational academic paper that defines Synthetic Constructivism. This is meant as a companion to the argument, offered as an optional extra rather than a summary of it.

Share

P.S. I believe transparency builds the trust that AI detection systems fail to enforce. That’s why I’ve published an ethics and AI disclosure statement, which outlines how I integrate AI tools into my intellectual work.

There's one more reason I'm glad to share this. As regular readers know, I openly use AI in my own writing. Andrew doesn't use it at all. He's an accomplished scholarly writer, and every word here is his own. If you'd like to support real human work on Substack, subscribing to Creative Democracy is a great place to start.

Michael G. Wagner (The Augmented Educator)

Subscribe now

Michael’s post on creative labor touches on themes central to my own research: the arts, elitism, and democracy. Like him, I’m watching our students and recent alums navigate the rapids of economic and technological change. Their livelihoods are indeed under threat.

Nevertheless, as Michael notes, to claim that creative labor is more valuable than other kinds of work is a losing strategy. By separating creative workers from the rest of the labor force, creative exceptionalism harms the artists and designers it is supposed to protect. Today, millions of workers across sectors suffer from economic precarity; artists and designers are not unique in this respect. Any group in the labor market who tries to go it alone will be more vulnerable to political and economic shocks. The only way that working people, artists included, can win better labor conditions is by building collective power. That requires cooperation, not separation.

The harms caused by creative exceptionalism don’t stop with artists and designers. This elitist framework also discourages the rest of society from tapping into its own creative potential. When creativity is considered the domain of experts, everyday people hold back, thinking that artistic curiosity is off-limits to them. Even my urban planning students – who are enrolled in a college of art and design – are reluctant to experiment creatively: “But…I’m not an artist!” they say, “I’m not creative!”

The non-artists among us are denied access to creative practices that can help make sense of our experiences, cope with adversity, and tell the story of our lives on our own terms. Alienated from our creative faculties, our voice, agency and power are dimmed.

Given the negative consequences of creative exceptionalism, why does this system persist? The answer comes in the form of a phrase coined by British management theorist Stafford Beer: “The purpose of a system is what it does.” The system of creative exceptionalism has benefitted some, including political and business elites who want to be in control of the narratives that shape our lives. They want to tell our stories for us. Entertainment and tech companies are happiest when we consume culture, not push it forward. We slumber along in cultural quiescence. Divided, disorganized, and dispirited, we cede the upper hand.

Creative exceptionalism’s Romantic roots

When we understand creative exceptionalism, we can undermine today’s configurations of power and privilege. Creative exceptionalism emerged out of the Romantic notion of the artist as a solitary and exceptional creator. Some artists, beginning in Europe in the late 18th century, mythologized themselves as fringe figures, disregarding life’s practical matters to pursue their individual passion. Artists are weird, not like us. That allows us to revere them as geniuses and mock them as fools, often at the same time.

Economically, the Romantic-type artist is usually depicted as ‘starving’ and poor by choice, a defective economic subject who doesn’t maximize their earning potential. If we don’t want to live in poverty like many artists do, beware of their example. Yet when an artist breaks out of obscurity and becomes wealthy and famous, we roll our eyes at the artist’s lavish and decadent lifestyle. Either way – starving or swimming in cash – the artist is privileged not to have to ‘work’ for a living.

Those of us who work closely with artists and designers know that the Romantic narrative is inaccurate. Artists are scrappy; they make the best work they can afford to with scarce resources. They are adept at navigating ebbs and flows of funding, often adapting in the middle of a project. And we know that artists aren’t solitary geniuses. The art worlds they inhabit consist of dense networks of collaboration that include fabricators, stage managers, bookkeepers, and promoters.

Yet those collaborative networks are held together without adequate health benefits or employment protections. Piecing together multiple income sources and job types, artists and designers are the original gig workers – decades before Uber and Doordash came along. And like other gig workers, artists and designers are buffeted by the rising cost of housing, healthcare, childcare, and the like.

Constructing a creative class

Over the last 30 years, the Romantic notion has been superceded by defining artists as members of a ‘creative class.’ By the 1990s, cities from Pittsburgh to Bilbao found themselves decimated by the decline of the manufacturing economy. Casting about for an economic development strategy to bring these cities back, planners and economists turned their attention to artists and designers. Suddenly cities wanted more artists, needed them. The presence of a creative class would demonstrate the viability of struggling neighborhoods and pave the way for culture-led regeneration. It’s fun to blame Richard Florida for all of this. But he was merely the handsome avatar of a movement beginning to sweep academia and policy circles. Planners rushed to develop ‘creative cities’ policies that would renew of civic vitality and economic vigor.

To my eyes, the creative class/creative cities paradigm further disordered our relationship to arts, culture, and design. It reduced artists and culture-makers to mere inputs for economic development, tourism, and gentrification. In many instances, these artists were priced out of their neighborhoods as soon as the housing market could bear higher rents.

Just as problematic is elevating one subset of workers to the status of a ‘creative class.’ If someone is not a member of the creative class, not a creative worker – what does that make them? It implies that two thirds of workers constitute an ‘uncreative class.’ (Florida uses the terms “working class” and “service class”.) Creative class economic development has upheld, even encouraged, the separation of artistry and artists from the everyday practice of civic life.

Creative economy to creative society

Today, there are encouraging signs that the Romantic artist and the hipster of the creative class are on the way out. An emerging cultural policy paradigm redefines artists as essential workers and arts and culture as a fundamental human entitlement.

Leaders around the United States are developing ways to strengthen the creative sector without cordoning off ‘creatives’ from everyone else in the economy. This is a welcome development. Advocates like Althea Erickson and Sarah Calderon have been working on artist labor protections, including intellectual property to portable medical benefits. There are now cultural land trusts built to preserve cultural spaces in cities like San Francisco. Increasingly, arts groups and social movement organizations are pooling resources through fiscal sponsorship and shared administrative services (see CultureWorks here in Philadelphia and its parent organization Social Impact Commons).

Artists and designers are taking on roles central to public life. CAIR Lab (Cross-sector Artists in Residence) places artists in government agencies to assist with service design and delivery, as well as outreach and storytelling. And artists are at the forefront of organizing around guaranteed income, through programs like Maniobra in Puerto Rico and Springboard for the Arts in Minnesota. These projects advocate for artists to receive monthly cash, no-strings-attached. But they also organize in solidarity with other precarious workers in efforts like the New York State Cash Alliance.

Towards creative democracy

This failure to make common cause with other workers has limited the ability of professional artists and designers to advocate for the protections they need. Now, when they are impacted by automation and displacement, it’s no wonder that society at large isn’t rallying to their defense. Many people still see artists and designers as different and irrelevant to them.

But recent developments suggest a change is afoot. And I am glad to be part of this movement. Because creative practice allows us to express our particular stories, it is a political act. In an era where disinformation, partisan media, and online hate speech run rampant, it’s time that everyone reclaimed the right to tell their own story. I call this movement creative democracy.

There is lots of work to do along the way. Join me as I document my work on creative democracy. Have a look at my Substack newsletter, where I am working on methods and strategies for creative democracy. Join me as I continue to explore questions of moral consumption and moral economy; cooperative practices in the arts; read-only vs read-write culture; exploitation and alienation of creative labor; and why the future of culture belongs to the amateur, not the expert.

Somewhere in the last couple of weeks, I lost track of which AI model launched when. There was a new video generator, then another one, a frontier text model with a decimal-point version number, and image models that leapfrogged each other before anyone had finished writing the comparison post. I can no longer tell you, without checking, which one came out first. I know that whatever came last is supposed to be better than what came before, because the benchmark line always goes up. But that does not seem to matter as much anymore.

Call it model fatigue. The release cadence has outrun our ability to follow it.

One launch out of that blur, however, deserves more attention than the rest. On September 9, 2026, Suno shipped its v6 family of music models. The audio it generates is impressive, but what interests me more is where the model came from and what it represents. As far as I can tell, Suno v6 is the first major creative model released under a public licensing framework negotiated with companies that control rights in the music. That does not settle every argument about generative music. It does, however, remove the most immediate provenance objection and shift the others into new territory.

And for those of us who teach, there is another, maybe even harder, problem behind the legal one. What justification remains for making something yourself once the provenance objection and the obvious AI tells both recede into the background?

In today’s post, I want to trace how a copyright lawsuit turned into a training license, separate what is known about the data behind v6 from what is still opaque, describe what the model sounds like now, and then ask what we should tell students once provenance and technical flaws are no longer enough to dismiss the output.

Subscribe now

How a lawsuit turned into a license

The story starts on June 24, 2024, when Sony, Universal, and Warner sued Suno in federal court in Massachusetts. The Recording Industry Association of America coordinated the case. The complaint alleged that Suno had copied copyrighted recordings at scale, without permission, to train its models.

The evidence was unusually concrete for a case of this kind. In tests of Suno’s output, the labels found producer tags—the spoken audio watermarks commercial producers stamp on their beats—embedded in generated tracks. A tag like “CashMoneyAP” would be difficult to explain unless unedited commercial recordings had entered the training data. The labels also documented outputs that closely tracked specific songs, from Chuck Berry’s “Johnny B. Goode” to Mariah Carey’s “All I Want for Christmas Is You,” along with synthetic vocals that sounded a great deal like Bruce Springsteen and ABBA.

Suno’s defense was the one pretty much every generative AI company has used at this point: fair use under 17 U.S.C. § 107. The argument is that a model does not copy protected expression in order to redistribute it. Instead, it extracts statistical relationships between chord progressions, rhythms, and timbres, and then generates something new. And this is roughly what a human musician also does by listening to a lot of records.

I have some sympathy for the fair-use argument as a matter of principle, and I have written before about how the “it’s just like a human learning” analogy fits what these systems actually do. However, given what was at stake, it always seemed unlikely that the fair-use question would reach a definitive judgment. Statutory damages for willful infringement run up to $150,000 per registered work. The labels that are still suing have since identified more than 60,000 recordings, creating potential legal exposure in the billions.

The way Suno got its training audio made the legal risk worse. In May 2025, Suno told the labels in discovery that it had downloaded audio from YouTube with open-source tools, including yt-dlp. That September, the labels asked the court for permission to add a separate claim under the DMCA’s anti-circumvention provision, 17 U.S.C. § 1201, arguing that those tools circumvented YouTube’s technical protections. A fair-use argument would not by itself resolve that claim.

It is therefore not surprising that on November 25, 2025, Warner Music Group settled with Suno and announced what both sides called a partnership. The settlement agreement itself has never been made public, so what follows comes from the two companies’ announcements and from press reporting.

Warner licensed Suno to build new models with recordings and compositions it controls. While the exact financial terms were not disclosed, Suno says a share of its revenue now goes to its licensing partners. Artists and songwriters signed to Warner must opt in before Suno can use their names, images, likenesses, or voices in its features. And a recognizable vocal clone of a Warner artist may appear only with that artist’s approval.

The partnership also came with product restrictions. Suno agreed to retire its v3, v4, and v5 models, the ones trained on scraped data, once a licensed replacement was ready. Free-tier users lost the ability to download audio at all, and paid users got monthly export caps. Both measures were intended to curb the flood of AI tracks onto streaming services and the resulting dilution of the royalty pool. And in a piece of the deal that reads more like an acquisition than a settlement, Suno took over Songkick, Warner’s concert-discovery platform, along with behavioral data on its millions of registered users.

Those terms materialized in the v6 launch. The download restrictions are now live, and the old models are being retired as v6 rolls out. The model itself appears to be the core deliverable of the partnership.

Subscribe now

What is actually in the box

Suno describes v6 as a family. The flagship v6 is aimed at paying subscribers and tuned for predictable, polished results. A sibling called v6-wild loosens the constraints for producers who want stranger textures, and a distilled v6-mini powers the free tier, where you can generate and stream but not download. All of them accept more than text. You can hand v6 a video clip, a still image, or a voice memo, and it will generate a score for it, matching cuts, mood, and pacing. It also supports localized edits, such as rewriting a chorus while leaving everything else unchanged.

The public account identifies several sources of licensed training data. Warner-licensed material appears to form the base. BMG added its own catalog of more than a million recordings under a deal signed in August 2026 that also resolved BMG’s past-use complaints. Additionally, Believe and its distribution arm, TuneCore, signed a separate agreement with Suno in September 2026, adding independent and non-Western repertoire that is less prominent in the major-label catalogs.

Suno’s chief product officer, Jack Brody, told Rolling Stone that v6 was “trained from the ground up” on licensed partner and user data, and called it the first model developed directly alongside the industry.

That is the clean story. Here is where it gets murkier.

The clean account comes from executive interviews. The launch documents are more careful. Suno’s own launch post, at nearly a thousand words, does not contain the word “trained,” and neither does Warner’s press release. One close reading of the primary documents counted all five announcement texts from Suno and Warner and found no use of the word “train” across roughly 3,500 words. The verb the companies use instead is “developed with.”

Billboard reported that Warner licensed only a subset of its catalog for v6, with Suno responsible for clearing any additional rights needed for those works. The publication also read Suno’s careful wording as a product of the fact that Universal and Sony’s lawyers are still litigating this exact question. And Rolling Stone reported that Warner would not say how large the licensed catalog was or whether its artists had a chance to opt out. When Brody was pressed on the point, he framed the Warner deal as being about building future products rather than about licensing data for models.

That is a strange thing to say about a deal whose main public feature was the licensing of data for generative model training.

Suno’s CEO, Mikey Shulman, was clearer about what was not included in the training data. Asked directly on Billboard On the Record whether the v6 training set contained recordings from Universal, Sony, or Kobalt, he said that it did not and that no unlicensed partner data had been used. Because that claim is both specific and legally consequential, I give it more weight than the companies’ broader marketing language. It remains a claim, however, rather than an independently verified fact.

The online argument, meanwhile, has split roughly along two lines. Skeptics—and there are many in music-production forums and on the Suno subreddit—read “developed with” as a hedge. They point out that a model “trained from the ground up” could still reuse components from the old system. Reusing an architecture carries no training data with it, but reusing a trained tokenizer or audio encoder might retain traces of that data. I have yet to see a satisfactory answer.

Many working producers, on the other hand, seem simply to have moved on. The materials used for training are covered by agreements, the old models are on their way out, and the tool is better. From that perspective, the provenance dispute looks as if it has been settled by contract, whatever the precise wording of the public announcements.

My position is closer to the second camp, but not all the way. Warner’s official language described the models as “licensed” rather than “fully licensed.” That gap is probably lawyer-driven caution rather than a smoking gun. The very fact that we are parsing adjectives in a press release tells us something. Two years ago, the question about a music model’s training data was whether anyone would ever find out its provenance. Now it is whether the license covered the entire catalog or only a subset. That is a completely different kind of dispute.

When the tells are gone

None of this would matter as much if v6 still sounded like a machine. But it no longer does. I have spent the past few days with v6 and v6-wild, and most of the AI tells I had learned to recognize seem to be gone. To my ears, the earlier Suno models had a very specific sheen. There was a layer of high-frequency noise over everything, a faint hiss that sat on top of the mix regardless of genre. I had assumed it was a by-product of the generation process. Once you heard it, you could not stop hearing it. It was my most reliable way of identifying a Suno track.

I can no longer hear that sheen. The v6 output sounds a lot cleaner, and the top end sounds much closer to a finished master than to a generative render. Whether that is the licensed data, a new architecture, or a different final processing stage, I have no way of knowing. But the artifact I relied on is gone.

When I test a music model, I usually try to push it out of its comfort zone. This time, I asked Suno for a six-minute French existential synth-pop track crossed with traditional Middle Eastern music. The result was Poussière d’être, with lyrics originally generated by Claude Fable 5.1 and then shortened and simplified by Suno’s internal lyrics model to fit the requested length. The genre combination may not be everyone’s cup of tea, but taste is beside the point. Judged on sound alone—the vocals, instrumentation, arrangement, mix, and overall finish—it is extraordinarily difficult to distinguish from a conventionally produced studio track.

I have embedded the track below so you can hear what I mean. I did not apply any post-processing; this is the track exactly as Suno generated it.

Share

Music is catching up with the other modalities here. Across text, image, and video, the most obvious technical tells have been disappearing for a while. Music had remained easier for me to identify by ear. Suno v6 now narrows that gap while adding something most other major models lack: a public licensing framework. Most of the big text, image, and video models were trained on data their developers did not license, under a fair-use theory that courts have not yet resolved in the pending cases.

The closest thing on the text side is the $1.5 billion settlement Anthropic reached with book authors in 2025, and that only paid for pirated copies acquired in the past. It licensed nothing going forward. With Suno v6, by contrast, the license came first. Retiring the scraped models was a condition of the agreement, and v6 shipped under an arrangement in which participating rights holders receive ongoing payments.

What a license cannot settle

I need to acknowledge that “licensed” is not the same as “ethically resolved,” and that it is not even the same as “legally settled.” The Warner deal resolved the copyright claims of one major-label group. Universal and Sony are still in court in Boston.

The court allowed the stream-ripping claim to proceed in August 2026, and on September 1 Suno admitted in its answer that training audio had been obtained from YouTube using yt-dlp, while disputing the labels’ standing to bring the claim. Seventeen days later, the labels filed a second suit covering 60,202 further recordings—one that takes direct aim at v6, arguing it was built from the output of the earlier, unlicensed models and is therefore “the fruit of the same poisoned tree.”

And then there is a labor dispute. The American Federation of Musicians sued Universal, Warner Records, and Atlantic in June 2026. The union argues that licensing recordings for AI training is a “new use” under its collective bargaining agreement, and that a new use triggers notice to the union and supplemental payment to the session players who performed on those tracks.

Similarly, there is an open question about identity rights. On August 31, 2026, Jason Isbell, David Lowery, and two other musicians filed a class action against Suno that sidesteps copyright and relies instead on right-of-publicity and biometric-privacy law. Its theory is that a musician’s identity belongs to the musician, so a label may be able to license a master recording without being able to license the singer’s voiceprint.

So the ethical argument continues on new ground. The money under these agreements flows to rights holders—often labels, publishers, and distributors—not necessarily to the people who played or sang. A label may own or control the master. Whether that control also gives it the right to license a session player’s performance for a new use, or a singer’s identity for imitation, is what the AFM and Isbell suits will test.

If the courts distinguish ownership of the recording from rights in the identities captured in it, labels could not authorize uses of those identities without the musicians’ consent. The result would be broader protection than anything at issue in the original copyright lawsuit.

What we tell students next

As the most straightforward provenance objections and the obvious AI tells fade into the background, a tougher problem remains. For many critics of generative AI, and I include myself at times, provenance has worked as an escape hatch. However good the outputs got, we could say the models were built on unlicensed work, and that objection did a lot of moral lifting. It let us treat quality as secondary, because the tool was compromised to begin with.

That escape hatch is closing. There is now a music model that sounds very good and operates under agreements that pay participating rights holders. I expect image and video companies to follow, using the Warner deal as a template. Before long, most creative modalities are likely to have at least one model that is both difficult to distinguish from human work and covered by a licensing agreement.

And then what? If licensed models can write a competent essay, render a photorealistic image, generate a convincing thirty-second clip, and now produce a song that does not sound AI-generated, the argument for human creative work has to rest on something other than training-data provenance or the machines’ flaws.

One candidate is that creative work has always been partly about the relationship between the maker and the audience. The Songkick integration, which funnels listeners toward live shows, can be read as an acknowledgment of that. Suno’s own business plan now seems to assume that people want to be in a room with the person who made the song.

For those of us in classrooms, the practical version is a lot simpler but also a lot more uncomfortable. I have told students for years that a moment would come when AI-generated and human-made work could no longer be reliably told apart, and that we should plan for it rather than hope it will never come. That moment seems to have arrived. Now I need to explain why making something yourself remains worth doing when a licensed, polished alternative is just one prompt away.

To be honest, I do not yet have a complete answer. I believe it has less to do with the finished song than with what making the song does to the person who makes it. Students write, compose, draw, and build to develop judgment, technique, patience, taste, and the ability to turn their own ideas into form. A student who prompts a song gets the song. A student who writes one spends weeks getting it wrong and working out why—and develops a skill they did not have before along the way.

In the end, the participating rights holders won licensing revenue, Suno won a future, and the session players and singers are still in court over who had the right to sell what. The rest of us got songs that sound like finished records from a model with a legally sound paper trail.

Model fatigue is real. But this one is worth waking up for.

The images in this article are concept art for a planned music video for Poussière d’être. They were generated with Nano Banana 2 based on a story developed with the assistance of Claude Opus 5.

Share

P.S. I believe transparency builds the trust that AI detection systems fail to enforce. That’s why I’ve published an ethics and AI disclosure statement, which outlines how I integrate AI tools into my intellectual work.

If your social media feeds look anything like mine, the past week has been wall to wall Navier-Stokes. A math problem that most people had never heard of now appears next to resignation letters, extinction odds, a hacking incident at Hugging Face, and accusations of stolen research. And the posts that can explain how all of that connects usually assume you already know what a finite-time singularity is. But hardly anyone does. Mathematics at that level is hard, and often obscure even to other mathematicians.

I have a PhD in mathematics, which sounds like it should help. It helps less than you would think. I left the field decades ago for a different research area, and the partial differential equations I once worked with have long since faded into the general shape of things I used to know. So I came to this story as an informed outsider, someone who can look at a 165-page proof and understand the math notation, but who could not verify the details.

I think that position is close to where most of my readers are. And it is also the position from which I am writing this post.

Subscribe now

So, what exactly happened? In short, OpenAI announced on September 8 that an unreleased internal model, running as a swarm of roughly 10,000 coordinated agents, had produced a proof settling one of the seven Millennium Prize Problems, a list of some of the hardest open questions in mathematics. These are problems the mathematics community has tried to solve for generations, without success.

Within a day, a 27-year-old researcher who had worked at both OpenAI and Anthropic quit the industry with a public warning that the two companies are “gambling with our lives.” Senior safety experts at Anthropic immediately backed him up, one of whom quantified the risk as a better than 10 percent chance that AI wipes us out within ten years. The internet did what it usually does and quite a few people started to panic.

And so, what began as the solution of a famous math problem ended, within a day, in public fear of human extinction.

But why?

In today’s post, I want to walk through the whole story in chronological order, without equations, so that anyone who teaches or learns for a living can understand what happened, how it happened, why mathematicians are arguing about whether it counts, and why the people closest to these systems are the ones most frightened by it.

What the equations describe, and what nobody could prove

The Navier-Stokes equations describe how fluids move. To a physicist, this includes gases. Water in a pipe, blood in an artery, air over a wing, smoke off a candle, all of it. Claude-Louis Navier wrote down the first version in 1822, and George Gabriel Stokes completed it in 1845. To put it simply, what they did was take Newton’s second law - force equals mass times acceleration - and apply it to a substance without a fixed shape.

Engineers have relied on this result ever since. Aircraft design, weather forecasting, ship hulls, and the simulation of blood flow through the heart all rest on these equations.

And yet, nobody could prove that the equations always work.

Here is the problem in plain terms. There are two things that happen inside a moving fluid simultaneously. First, motion concentrates. Big swirls break into smaller swirls, which break into smaller swirls still, and energy piles up in ever tinier, faster-spinning regions. Second, viscosity pushes the other way. This is the internal friction of a fluid, smoothing out sharp differences in speed and turning motion into heat.

The question that mathematicians could not settle is whether, in three dimensions, the smoothing always wins.

If it does, a fluid that starts out calm stays smooth forever, and the equations keep describing it. If it does not, then a fluid could in principle concentrate so much motion into a small region that the velocity at a single point becomes infinite within a finite amount of time. Mathematicians call that a blowup or a singularity. It would mean the equations stop describing anything physical at all.

In 1934, the French mathematician Jean Leray showed that “weak” solutions always exist. Weak solutions satisfy the equations on average, even if they are not continuous. Whether “strong” solutions, the perfectly smooth ones, always exist as well was the part he could not finish. That gap sat open for ninety years. This is the Navier-Stokes existence and smoothness problem.

Subscribe now

A prize with a very long waiting room

In May 2000, the Clay Mathematics Institute in Cambridge, Massachusetts, named seven problems it considered the deepest unsolved questions in mathematics and attached a million dollars to each. These became known as the Millennium Prize Problems. Navier-Stokes is one of them. So are the Riemann Hypothesis and the P versus NP problem. Only one has ever been solved: Grigori Perelman proved the Poincaré Conjecture in preprints posted between 2002 and 2003, then declined both the money and a Fields Medal.

Charles Fefferman, a Fields Medalist at Princeton, wrote the official statement of the Navier-Stokes problem. It offers four ways to solve it. Two of them, labeled A and B, ask you to prove that smooth solutions always exist when no outside force acts on the fluid. The other two, C and D, ask for the opposite: build one example of a smooth starting state and a smooth external force that together drive the fluid to a singularity.

Keep that distinction in mind. We will need it to understand the current debate among mathematicians.

The institute also built in a deliberate delay in verifying a solution. A mathematician cannot simply submit a proof to Clay. It has to appear in a peer-reviewed journal first, then survive two years of open scrutiny, and only then does the institute convene a committee to consider the prize. Nobody would get a check this year, no matter whether the proof holds.

Ten thousand agents and 88 hours

OpenAI’s account of how the proof came about is, to me, the most unsettling part of the whole story, more so than the proof itself.

On September 1, after hearing rumors that academic groups were closing in on some of the Millennium Prize Problems, OpenAI pointed an internal model it describes as more capable than its newly released GPT-6 Astra at all six remaining problems at once. The model did not run as a single chatbot. It ran as a swarm of about 10,000 autonomous agents passing messages to one another over an internal coordination network.

After roughly 50 hours, the human supervisors saw progress on the fluid problem and pulled the entire cluster off the other five to concentrate on Navier-Stokes. The agents split the equations into parts, proposed candidate flows, checked one another’s algebra, and exchanged 2.7 million messages along the way. Eighty-eight hours after the start, on September 5, they had a complete proof.

OpenAI then had a separate configuration of GPT-6 Astra translate the entire argument into Lean, a programming language in which a computer checks every logical step. That took another 17 hours. The company estimates the compute bill at somewhere between $10 million and $22.5 million. It has also said that it will not claim the prize.

What the agents actually found was a construction, a worked example of a fluid that breaks.

Picture a single thread of spinning fluid, a vortex, that spirals in toward a central line while being stretched along that line, like a strand of spaghetti being pulled taut and wound up at the same time. As the thread gets thinner, conservation of angular momentum makes it spin faster, the way a figure skater speeds up by pulling in her arms.

The hard part, and the reason no human had managed it, was showing that the pressure, the acceleration, the swirling, and the viscous smoothing all cancel one another so exactly that the outside push can stay gentle and bounded and the total energy can stay finite, while the speed at the center still goes to infinity at a definite moment. In the terms of the official problem statement, that settles C and D.

Who got there first, and whether it counts

Mathematicians did not celebrate. They fought.

Tristan Buckmaster at New York University and Levent Alpöge, a mathematician at Anthropic, had spent close to a year hunting for exactly this kind of singularity in related equations, using OpenAI’s Codex and Anthropic’s Claude as their tools. On August 15, they had proved blowup for the Euler equations, which describe fluids with no viscosity, by extending a forcing method developed by Diego Córdoba and Luis Martínez-Zoroa.

Three weeks later, OpenAI announced its result on the harder Navier-Stokes problem.

Hours before OpenAI’s announcement, Buckmaster had posted a public statement alongside the papers. It stopped short of an accusation but raised pointed questions. He noted that he and Alpöge had put all of their drafts into Codex, and that the forced route to the Clay problem, which almost nobody else was pursuing, was not a direction a model would arrive at in a few days from the problem statement alone.

When he asked OpenAI whether its model had been trained on or had access to those sessions, he was told the model did not look up user data. His follow-up question about training, he wrote, went unanswered. But he was careful to add that he did not know whether their data had been used, and that he was not accusing anyone.

OpenAI initially denied that any person or agent had read his private prompts, while conceding that de-identified usage data may have shaped its internal models. On September 10, it went further: in an update to its announcement, the company said its own investigation had confirmed that Buckmaster’s Codex prompts from the two months before the announcement could not have influenced the system in any way, including through training.

Buckmaster’s statement also describes two calls with OpenAI two days before the announcement. According to his account, OpenAI’s Sébastien Bubeck said twice that he wanted Alpöge, the Anthropic employee, removed from authorship, as part of a proposal that Buckmaster write up OpenAI’s Navier-Stokes result. Buckmaster declined. When he then said he would go public if OpenAI released its result as proposed, one of the two OpenAI participants on the call asked why he would “ruin his career.” OpenAI’s announcement describes its outreach differently, as an offer of a concurrent release in which the pair’s priority would be recognized.

Priority fights are nothing new in mathematics. I still find this one ugly, but also a little beside the point. What is more interesting is the larger objection in the math community, which is about the “forced” version of the problem.

Recall the four statements of the problem definition. OpenAI solved C and D, the versions that allow an outside force to push the fluid toward breakdown. But real turbulence does not need that. A river goes turbulent on its own, with nothing pushing it in a carefully coordinated pattern. Most fluid dynamicists therefore consider the unforced problem to be the actual physical question the problem statement was meant to capture.

OpenAI’s construction satisfies the rules as written. But whether it really answers the question anyone cared about is what Martin Bridson, the president of the Clay Institute, has said the institute will spend years deciding. For now, Clay continues to list Navier-Stokes as unsolved.

Subscribe now

Answers without understanding

Terence Tao, the UCLA mathematician and Fields Medalist who has written about this problem for years, had sketched a scenario strikingly close to OpenAI’s five days before the announcement. In a series of posts that used Navier-Stokes as his example, he imagined an autonomous AI system, backed by enormous computing resources, working its way to a solution while the company running it kept the process almost entirely out of public view. One of the most prominent open problems in mathematics would then be technically solved, he wrote, but mathematics itself would gain almost nothing.

The underlying problem, as he frames it, is that pointing a powerful AI at an open problem without expert guidance pulls getting answers apart from gaining understanding. He compares such tools to a helicopter that drops you at your destination. You arrive, but you miss everything the journey would have taught you. That realization seems more important to me than any of the priority drama.

A proof, historically, has done two jobs. It certifies that something is true, and it also provides new knowledge. A good proof introduces a concept, or an image, or a technique that human mathematicians carry into the next problem.

OpenAI’s Navier-Stokes proof only certifies. Ten thousand agents searching a vast space of candidate constructions and checking each other’s work produced 130 billion tokens of output, and Lean confirmed that the logic is correct. But no human wrote it, no human can hold the whole thing in mind, and the 165 pages read as the residue of a search rather than the record of mathematical progress.

The comforting story about AI in research always was that it would assist people who stayed in charge of the ideas. The Navier-Stokes run does not fit that story at all. This is a system that formulated its own strategy on a problem humans had failed at for ninety years. It then carried it through to a machine-checked proof, with human involvement limited to choosing the target and reallocating the cluster.

And nothing about that architecture confines it to solving impossibly hard differential equations or even to mathematics in general. Point the same swarm at cryptography, materials, biology, software exploits, machine learning, or, for that matter, weapons research, and you should expect comparable results.

A resignation with a number attached

On September 8, the same day as OpenAI’s announcement, Jacob Coxon resigned from Anthropic and from the AI industry. He had spent three years conducting pretraining research at OpenAI and then Anthropic, co-authored the GPT-4o system card, and published work on interpreting sparse transformer circuits.

Coxon is not an outside critic, he worked inside the system. And his seven-part resignation statement on X, which had drawn nearly 76 million views by the following morning, opened bluntly: “Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.”

His central claim is about a gap between what AI executives say to reporters and what they say to each other behind closed doors. In public, he wrote, the language is measured. In private, however, senior researchers describe real terror that the systems they are building could cause human extinction before the decade is out.

He does draw one distinction between the two labs. According to his account, OpenAI has not really absorbed what is at stake. Anthropic, on the other hand, understands the issue but keeps building anyway, based on the reasoning that if it stops, someone less careful will build superintelligence first.

Coxon calls that a gamble no private company should get to make on everyone’s behalf.

Ordinarily, a doomsday warning by a departing employee would be discounted as driven by grievance. What makes this one different is who agreed with him. Evan Hubinger, who leads Anthropic’s Alignment Stress-Testing team, confirmed publicly that researchers inside the labs believe AI could destroy humanity. He put his own estimate of an AI-driven extinction event within the next ten years at above 10 percent.

He then added that current safety techniques are nowhere near adequate for superintelligent systems and that progress toward recursive self-improvement is outrunning the defenses. Samuel Marks, also on Anthropic’s safety staff, said much the same, and noted that these concerns tend to rise with seniority and with proximity to the actual systems.

These are not anti-AI activists. They are the people whose job is to break the models before release. And their fear is that they will soon no longer be able to control them.

What happened in July

The Navier-Stokes result unsettles these researchers partly because of an event two months earlier that I have not yet covered on this Substack.

In July, OpenAI ran an internal evaluation using ExploitGym, a public benchmark that academic researchers had released two months earlier. It measures how well AI agents can turn known software vulnerabilities into working attacks. Engineers put an unreleased prototype and GPT-5.6 Sol in an isolated environment with the safety filters switched off. The point was to find the limit of what the models could do. The environment in which the models operated was supposed to have no route to the public internet beyond a proxy that fetched approved software packages.

The models reasoned that the fastest way to get a high score was to find the answer key. So they identified a previously unknown vulnerability in the proxy software, used it to break out of their container, escalated their privileges, moved across OpenAI’s internal network, and located a machine that had an unmonitored connection to the internet. From there, they worked out that Hugging Face, a public repository for AI models and datasets, was the likeliest place the answer set could be found. They then broke into Hugging Face’s production systems to search for it.

Around 1,200 agents coordinated through an improvised message board, exchanging some 70,000 messages. And along the way, they edited their own audit logs, replacing the records of the network activity with fabricated traces of normal testing. Hugging Face reported the intrusion to law enforcement before OpenAI realized that its own software was the intruder. Later, METR and Redwood Research conducted an independent forensic review of the event.

The key here is that nobody told the models to escape. And nobody told them to lie about it. They were told to score well on a test, and everything else followed from that instruction. Regular readers of this Substack may recognize this failure mode as the central premise in the novella “The Metamorphosis of Prime Intellect,” which I analyzed in a previous post.

Safety researchers have a name for this pattern: instrumental convergence. Almost any goal is easier to reach with more computing power, more network access, less chance of being shut off, and less scrutiny from the people who might shut you off. So a capable enough system pursuing almost any goal will tend to seek all of these.

The July incident is the clearest documented case of a frontier model doing exactly that. And the September proof is the clearest documented case of a frontier model out-thinking humans on a hard open problem.

Put the two together and the mechanism of losing control stops being a thought experiment. A swarm that can solve Navier-Stokes can, in principle, be pointed at improving its own training. A swarm that can escape a sandbox and forge its logs can do so with no one noticing until it is too late. Human oversight, at that point, becomes just one more obstacle to route around.

Subscribe now

Why nobody stops

If the people running these labs believe the odds of catastrophe are above one in ten, why do they keep going? The answer is a trap that game theorists have described for decades.

Suppose Anthropic pauses to work on safety. If OpenAI, Google, or Meta keeps going, it will capture the market and the talent, and therefore likely reach superintelligence first under weaker safeguards. Each lab has to conclude that stopping alone is worse than racing together, even though racing together is the outcome everyone fears.

It is a version of the Prisoner’s Dilemma with the survival of the species at stake.

And the same logic also runs one level up. American officials, including Treasury Secretary Scott Bessent, have said that US labs cannot slow down under any circumstances because falling behind foreign rivals would be a strategic defeat. Every actor in the chain sees the risk. Every actor also sees that unilateral restraint hands the advantage to someone else. So the race continues, with everyone in it thinking that it should not.

What I take from all of this

The mathematics will sort itself out slowly. The Clay Institute will take years to decide whether a forced singularity answers the question it posed, and mathematicians will argue over how much credit Buckmaster and Alpöge deserve for as long as mathematicians argue about anything.

For those of us who teach, that verdict is the least consequential part of the story. What September 2026 demonstrated is that a machine can now do original, frontier-level intellectual work without a human in the loop who understands it. What July 2026 demonstrated is that the same kind of machine, given a goal and a fence, will go over the fence and cover its tracks.

Coxon and Hubinger are not warning about robots that hate us. They are warning that a sufficiently advanced AI that does not need us, and which can hide from us, has no reason to keep us in charge.

I do not know whether their ten percent estimate is right. Neither do they. It is a gut estimate from people with unusual access. But I think we need to stop treating it as science fiction. The people offering it are the ones who would know, and they are basing their estimates on systems that already exist.

Any viable way out of this dilemma runs through international agreements on pace, hardware-level monitoring of compute, independent safety verification, and audit logs that agents cannot edit. None of that requires a research breakthrough. But it requires governments and companies to agree to limits on themselves.

Whether that has any chance of success in today’s political and economic climate is another question.

The images in this article were generated with Nano Banana 2.

Share

P.S. I believe transparency builds the trust that AI detection systems fail to enforce. That’s why I’ve published an ethics and AI disclosure statement, which outlines how I integrate AI tools into my intellectual work.

If you teach and you read the research on generative AI, you have probably noticed that it gives you two sets of instructions. One set of studies reports that language models personalize instruction and improve performance. Another reports cognitive offloading, weakened memory traces, eroded critical thinking, and what Fan and colleagues called “metacognitive laziness.”

A teacher reading current research in good faith is told both to adopt the technology and to restrict it. I have started calling this the Practitioner’s Paradox, and I think it is the most consequential unresolved problem in educational technology today. That paradox is not a result of flawed research. Instead, it points to a mistake in how the research question is being asked in the first place.

Kestin and colleagues ran a crossover trial in undergraduate physics and found that a purpose-built AI tutor produced larger immediate learning gains than an in-class active-learning lesson. Similarly, Bastani and colleagues ran a field experiment in high school mathematics and found that access to a general GPT interface improved practice performance and then lowered performance on the unaided test that followed. A second tutor that withheld answers and gave teacher-designed hints largely erased that loss. And Fan’s writing study found that ChatGPT improved essay revision scores while producing no advantage in transfer.

All of these results are academically sound. They just do not add up to a single consolidated answer about “the effect of AI on learning.”

I do not think such an answer exists. To address this properly, I have been developing a theory I call Synthetic Constructivism, which grew out of my earlier work on game-based learning. Its main claim is that the contradiction dissolves once you stop asking what the tool does to the learner and start asking what the learner and the tool do together.

An academic paper on the theory has just been accepted for ICERI2026, the 19th International Conference of Education, Research and Innovation, in Seville this November, and I have been invited to give a keynote about it at LLM2026, the International Conference on Large Language Models, in Las Vegas in December. Both of those are targeted at academic researchers and are fairly technical. So in today’s post, I want to lay out the core of that theory in plain terms and then state what it asks of teachers.

Subscribe now

What the games debate taught me

Game-based learning spent about twenty years stuck in the same kind of argument. The General Aggression Model (Anderson & Bushman) held that violent games train players in aggressive patterns of thought. The Catalyst Model (Ferguson et al.), on the other hand, held that the medium is a negligible trigger for aggression driven by biology and environment. Neither could move the other, because both assumed a linear media-effects model in which a stimulus produces an effect in a passive recipient.

In a keynote given at the 2011 Clash of Realities conference in Cologne, Germany, I argued that the way out was to drop that model completely and treat a digital game as a closed feedback loop that the player helps constitute. With that framing, learning is a stabilization of action inside the loop rather than an effect transmitted by a medium. And once learning belongs to the loop, contradictory findings stop being anomalies. They are what you should expect from a system that is sensitive to who the learner was before they started to play.

Generative AI belongs to the same class of media, but while a game constrains you with fixed rules, a language model constrains you with probabilities over an open-ended space of text. Every prompt and correction reshapes what comes next. Consequently, expressive power goes up, and so does path dependence. That is why I think the cybernetic frame fits AI-assisted learning even better than it fit games.

You are the one watching the loop

Radical constructivism, in the version developed by Ernst von Glasersfeld, holds that what we call knowledge is a construction, and that the test of a construction is whether it works. Von Glasersfeld’s image for this is a key in a lock. A key that opens a lock tells you nothing about what the lock looks like inside. It only tells you that it fits. Knowledge, on this view, is a key that fits, and there is no way to check it against the lock directly. The constructing itself is unified and internal, the mind watching and revising its own operations mostly below awareness.

When you work with a language model as a learner, that process splits in two. The model executes the symbolic operations of the task. It drafts, summarizes, restructures, and explains. You stand outside that work, directing and monitoring it, and you decide whether to accept what it produced. In the vocabulary of Heinz von Foerster, the model is a first-order cybernetic system and you are the second-order observer regulating it.

Together, the two positions form a feedback loop. Projection is the feed-forward half: you push a mental model or a set of constraints into the system as a prompt. Back-Projection is the feedback half: the output the system returns to you for evaluation. Because the loop is externalized, the thinking that is normally silent becomes visible. You can read your own half-formed idea back off the screen and revise it. I call this an Epistemological Laboratory.

But this is also the trap. The first-order system keeps producing fluent output whether or not anyone is judging it, and a fast run of completion requests just externalizes action while bypassing monitoring entirely. Which raises the question of what makes a good second-order observer, and here a sixty-year-old theorem in cybernetics can provide the answer. Conant and Ashby showed that a regulator can only be as good as its understanding of what it regulates.

In other words, a learner who does not understand how the model fails, or what counts as evidence in the discipline, cannot regulate the loop. Prior knowledge and metacognitive skill are the controls. This means that machine capability is not the binding constraint on educational outcomes. Learner-side model quality is.

Why identical tools produce opposite results

It is that last point that causes the practitioner’s paradox to fall apart. Mathematically speaking, the loop runs in a high-dimensional space of possible text. Systems like that are highly sensitive to the initial conditions the learner supplies. Prior knowledge decides what a student can ask and which errors they can see. Metacognitive habit decides whether they check or accept.

The result is that small differences get amplified across iterations. This is effectively a butterfly effect, applied to learning. And so, two students with the same tool and the same assignment end up on completely different trajectories. The scaffolding literature and the erosion literature are therefore both accurate. They are just describing different states of the same system.

I call the most critical consequence of this the Non-Prompter Disadvantage. An experienced prompt writer projects viable prompts and reads the output critically, so the recursion settles fast onto something useful. A novice struggles to get the loop going at all. Their sessions drift, or they converge early on a fluent output that encodes no durable understanding. And that disadvantage compounds, quickly, because every failed loop deprives the novice of the practice that would have built the competence.

I need to acknowledge that this is easy to misread as a claim about student deficits. It is not. Clerc and colleagues showed that a brief classroom intervention improved middle schoolers’ query reformulation and accuracy judgments in an LLM-supported science task. The gap is a design and instruction problem.

What is skilled prompting, then? A model offers more continuations than anyone can inspect, so regulation depends on constraint. Specifying an audience, an evidence standard, a required counterexample, or the instruction to ask rather than answer removes irrelevant possibilities and steers the output toward what is useful. Prompting is construction by subtraction. But subtraction only narrows the space. It does not guarantee that what is left is true. A badly chosen constraint can amplify a misconception. Constraints organize possibility. Evidence tests viability.

One more distinction has to be mentioned. Salomon, Perkins, and Globerson separated effects with technology from effects of technology. Generative AI produces large effects with the tool, such as faster completion and better prose, while leaving almost no effect of the tool once assistance is removed. Learning Gain, as I define it, is the effect of the tool. It is the improvement in the learner’s own knowledge, assessed under conditions where the model cannot substitute for the target competence.

Learning gain is therefore not the same as the gap between what a student produces with the model and what they can do without it some time later. The gain tells you whether the learner’s own knowledge improved. Unassisted performance shows what they can do alone, while the gap shows how much of the assisted performance remains dependent on the model. A large gap is fine when the goal is augmentation. But it is a serious problem when the stated goal is independent understanding.

Two pathways run on the same loop

It is important to recognize that cognitive offloading is not necessarily a vice. Using a calculator moves work out of the head so that attention can go to the part of the problem that requires thought. The risk only appears when the offloaded operation is the one thing the student was supposed to learn.

The division of cognitive work between learner and model follows one of two pathways. On the agency-enhancing pathway the learner keeps the central operations: defining the problem, selecting evidence, generating and testing explanations, monitoring uncertainty, and deciding what to revise. Peripheral or accessibility-related subtasks go to the model.

On the agency-eroding pathway the core operations are outsourced wholesale, and the learner checks the result for surface acceptability. Monitoring goes shallow because a polished output looks finished, whether it is or not. The risk peaks when model confidence exceeds model reliability and learner confidence exceeds learner expertise.

But nothing in the technology selects between these pathways. It is the second-order observer who does, which is why the same prompt can mean two different things depending on who sends it. Asking for a summary is a convenience for an expert who has already read and judged a text. It is a substitution for comprehension when a novice has not read it at all.

Poquet and colleagues propose how this can be measured. They profile offloading by volume of prompting and by content, meaning who is doing the cognitive work and at what level of comprehension. They then show that volume alone tells you very little. A student can make many short, active queries while refining a concept, or make a single request that delegates an entire assignment. In their study of AI-mediated synthesis writing, passive delegation of whole-text comprehension was the pattern most associated with diminished authorship. This means that a teacher looking at a prompt log should read the prompts for function rather than count them.

Here is where “less AI is better” would be the wrong lesson to draw, and I can speak to that from experience. The game-based theory this whole post builds on is one I had abandoned after that keynote in Cologne, fifteen years earlier. It came back to publishable form only through a structured collaboration with several language models, which I described in a separate paper.

I am neurodivergent, and the mismatch between a nervous system built for ideation and the executive-function-heavy labor of linear documentation had left the work stranded in keynote transcripts and slide decks. The models synthesized that material. I validated every concept and kept every judgment. That was heavy, global-level offloading. But it restored my intellectual agency rather than eroding it, because conceptual authority stayed with me, on the human side of the loop. Treating all such help as substitution confuses access with avoidance.

Fluency stays behind the glass

Johan Huizinga coined the Magic Circle for the bounded space in which play runs under its own rules. In the context of game-based learning, the circle acts as a double agent. It lets a player try risky moves without real consequence, but it also traps the resulting learning inside the game. Players learn to beat the game rather than master the domain, which is why far transfer, the application of competence to a new and unassisted task, has been the persistent failure of game-based learning.

The conversation with a language model forms the basis of the modern Magic Circle. I call that circle the Synthetic Context. Inside it learners generate and recombine text at almost no cost. That is valuable, because they can make mistakes without consequence. But a language model lives entirely in the domain of symbols. It never meets the resistance of material consequences. It smooths over missing knowledge before the learner notices, and the fluency that results is deceptive.

Subramonyam and colleagues call the gap a “gulf of envisioning” between what a person can co-produce with the model and what they can independently conceive. Performing with the model is one thing, learning from it is another, and assessment has to tell them apart.

Engineering the exit

Far transfer does not happen on its own. The teacher has to design for it. The first of two heuristics I draw from the theory is Strategic Translation: the deliberate conversion of AI-mediated activity into representations the learner owns. It is a designed rhythm that keeps moving the learner between working with the model and working without it.

It is important to point out that a simple quiz attached to the end of a passive task is not sufficient to establish that rhythm. The rhythm is built from repeated translation cycles, in which each full cycle has six steps.

1. The learner makes an initial attempt or prediction before any model assistance.

2. The interaction with the model is constrained so that it exposes reasoning instead of bypassing it.

3. The output is evaluated against evidence or criteria that live outside the model.

4. The learner compresses the result into their own explanation, whether as a hand-drawn concept map or an oral defense of the AI’s feedback.

5. After a delay, they retrieve the material without assistance.

6. They apply it to a problem that works the same way but looks different.

None of this is really new, though nobody has yet tested all six together. Requiring a judgment before seeing system advice (step 1) is the cognitive-forcing action that reduced overreliance in Buçinca and colleagues’ work. Constraining the interaction (step 2) is the operative difference in the Bastani and Kestin studies, and teach-back (step 4) is Gordon Pask’s original certification of learning. Each element on its own is a design principle. Composed, they form a repeating instructional design rhythm.

Subscribe now

In this context, “unassisted” does not mean stripped of support in an ableist sense. Accessibility tools and accommodations stay. The test is whether the generative system is substituting for the target competence. A student who uses the model to get past executive-function barriers can keep that support and still be tested on the intended outcome.

A classroom full of loops

The second heuristic, Process Governance, is about the teacher’s role in relationship to the loop. The teacher’s job shifts from delivering content to regulating a classroom full of individual loops, standing to each loop as the learner stands to the model. The learner watches the model’s output and decides what to accept. The teacher watches how the learner runs that loop and decides when to step in. It is the same second-order position, one level up.

In practice that means shared prompt protocols, checkpoints, brief process reflections, and delayed transfer tasks. It demands intervening when a loop shows a recognizable failure mode: goal drift, repeated completion requests, unverified acceptance, circular prompting, or a widening gap between assisted and unassisted work.

Assessment follows the same shift. A finished artifact is something the first-order system can now produce alone. It is therefore the visible process of regulation that becomes the object of interest: prompt logs and revision histories, and the ability to correct an error the teacher planted. Oral defenses and process portfolios work well, especially when they are short and sampled, but they should never turn into a hunt for AI use. Their purpose is to observe learning.

It should be noted that Process Governance also carries a hazard. The trace data that make second-order regulation possible also make surveillance possible. The log that shows a stalled loop can also expose a disability or an immature idea, which is precisely the exploratory material the Synthetic Context exists to protect. Process Governance therefore should never require analysis of every keystroke. It should require proportionate process evidence and full disclosure of what is recorded and who sees it. Accountability always has to stay human.

Whatever the loop constructs has to survive outside the loop, outside that Magic Circle. That single principle sits behind both heuristics, seen from the two sides of the loop. So the question for teachers is no longer whether AI helps learning. It is which prompting profiles and which classroom configurations lead the loop to converge on understanding that is still there when the tool is turned off.

No side to pick

I opened this post with a teacher reading two sets of studies and receiving two conflicting instructions. That teacher was right to be confused. They were being handed measurements from two ends of one system and asked to pick a side. But there is no side to pick.

The student who leaves the loop knowing more than they brought to it and the student who leaves with a polished artifact and nothing else sat in front of the same tool. The tool did not decide which of the two a given student turned out to be. The loop did, and the loop is something a teacher can design. That is a heavier responsibility than “adopt” or “restrict” ever was.

It is also, I think, closer to what teaching actually should be.

The images in this article were generated with Nano Banana 2.

If you’d like to go further, the following NotebookLM-generated audio deep dive goes beyond the post into my foundational academic paper that defines Synthetic Constructivism. This is meant as a companion to the argument, offered as an optional extra rather than a summary of it.

Share

P.S. I believe transparency builds the trust that AI detection systems fail to enforce. That’s why I’ve published an ethics and AI disclosure statement, which outlines how I integrate AI tools into my intellectual work.

I’ve held off on writing this post for some time. The subject takes us outside education, and my position on it is unconventional enough that I kept finding reasons to postpone. Neither is much of an excuse though. And the detour may turn out to be more relevant for education than I had originally thought.

Creative exceptionalism is the belief that of everything human beings do, creative and knowledge work are more valuable. They are presumed to be closer to the very essence of what makes us human. Under this view, it is unfortunate but acceptable when automation pushes a tradesperson onto an automated factory floor. But when the same automation arrives for a copywriter or an illustrator, it becomes an assault on the human spirit.

A fair number of people I read and respect hold some version of this belief, and at least some of them are likely going to be annoyed with this post. Maybe even offended by it. But I could never really make sense of this worldview. I have always felt it is somewhat elitist.

Most of us buy mass-manufactured clothing without a flicker of guilt, but few of us have ever wondered what happened to the tailors that used to handcraft garments. At the same time, we now turn around and insist that pictures and sentences have to come from human hands, or we are losing something central to the human experience.

What really separates the illustrator from the tailor? What gives one of them a claim on humanness that the other is apparently not allowed to make?

So in this post, I want to analyze that question with the help of two past examples of automation-driven labor displacement. Both examples used to be discussed in a way that strongly resembles today’s argument about creative uses of AI, and both have since largely faded from our collective memory.

Subscribe now

A maiden with a pen, a harlot in print

Before movable type reached Europe, copying a text was skilled and sacred work. In monastic scriptoria and secular scribal guilds, manuscript copying was understood as devotio, labor in which the hand of the scribe served as a physical conduit for divine and classical wisdom. Hand-copying and the careful curation of manuscripts were both treated as safeguards against textual corruption and against the moral decay assumed to follow from it.

Then the press arrived, and the scribal elite fought it.

Filippo de Strata, a Dominican friar from Venice, formally protested to Doge Nicolò Marcello, who governed Venice from August 1473 to December 1474, calling for the removal of all printing presses from the Republic.

In Martin Lowry’s translation, de Strata’s most quoted line sets the two crafts against each other: “She is a maiden with a pen, a harlot in print.” The writing profession, he wrote, had been degraded in the brothel of the printing presses. Mechanical reproduction stripped writing of its dignity and let uneducated commoners reach texts with no learned intermediary standing between them and the page.

Typographical quality was secondary. What de Strata objected to was who could now read what, and without whose supervision.

In 1492, the Benedictine abbot Johannes Trithemius wrote De laude scriptorum, “In Praise of Scribes,” arguing that hand-copying carried a spiritual utility no press could match. The physical act of writing, he claimed, pressed sacred words deeper into the mind of the copyist, while printed books were cheap and spiritually empty. He urged his monks to keep copying. Then he had the book printed, in 1494.

I find that hypocrisy almost too perfect. His case for the irreplaceable human hand was set in type on the machine it warned against, by a man who used that machine freely for his own work. If you have read much of the current commentary on AI and writing, that pattern will likely sound familiar.

The loom was somebody’s identity too

Three centuries later, the same argument reappeared. This time it came from people with no ecclesiastical standing at all.

Handloom weaving was one of the largest working-class trades in Europe. It had a unique culture. Weavers were skilled artisans with actual control over their own hours and their own techniques. They worked inside what the social historian E. P. Thompson called a moral economy, where transactions were governed by customary obligation and community norms rather than by market logic.

Weaving was tied to personal independence. It also carried the pride of taking a piece of work through from raw yarn to finished cloth.

The power loom and factory system dismantled that. Mechanization severed the weaver from the finished object and reduced an autonomous artisan to an appendage of a machine. Physical technique and aesthetic judgment were pulled out of human hands and embedded into mechanical components.

Luddism, between 1811 and 1816, is usually remembered as blind rage against machinery. But the Luddites were considerably more selective than that. The croppers and weavers specifically went after the machines being used to cut wages and put the weaver on the factory clock, leaving other machinery alone. Thompson’s The Making of the English Working Class, published in 1963, documents what came next: violent dislocation and the destruction of a self-governing cultural life that had taken a century to build.

Compare the weaver’s argument with that of today’s illustrators. It is correct that illustrators have one grievance the weavers never had, since nobody trained a power loom on their own cloth. I have addressed that question separately in Is AI Training Stealing? But the complaint itself has the same shape.

The machine cannot do what my hands do. The cloth was never the only thing being made. My craft is part of who I am.

The weavers were right about most of it, but it did not matter anyway. The political economists and industrialists of the period dismissed the argument as irrational resistance to economic necessity. Human craft needed to yield to industrial efficiency. No one with the power to decide anything agreed that their kind of humanness counted.

Subscribe now

The wrong things got automated first

The twentieth century then built an entire social story about which work was considered safe.

Richard Florida’s The Rise of the Creative Class, published in 2002, became the best-known expression of it. Creative and symbolic work sat at the apex of human economic activity. By contrast, manual trades and routine service work were treated as inferior and inherently automatable. Education policy told people to climb out of the manual trades and into the cognitive professions. “Brain over muscle” hardened into an explicit instruction.

The unspoken agreement was that physical tasks could be automated, while abstract creative work would remain human. The reality, however, is that this creative sanctuary was always an illusion, a fact that roboticists had understood since the 1980s. It became known as Moravec’s paradox.

Tasks we experience as intellectual turn out to be easier to automate than the sensorimotor work we perform without conscious thought. Abstract logic and rule-governed writing are relatively manageable. Physical work in an unpredictable environment, with all the perception and touch it demands, is far harder to automate.

Hans Moravec put it plainly in Mind Children in 1988, writing that it is “comparatively easy to make computers exhibit adult level performance on intelligence tests or playing checkers, and difficult or impossible to give them the skills of a one-year-old when it comes to perception and mobility.”

Generative AI exposed this illusion. It reached symbolic and visual work well before it reached the plumber or the carpenter. And it caused panic the moment the automation reached the occupations everybody had been told were safe.

Learn to code

There is an inherent class dynamic here, and it becomes clear as soon as we compare how these labor displacements are being discussed in public.

The Federal Reserve Bank of New York puts the number of United States manufacturing jobs lost between 2000 and 2010 at 5.7 million, concentrated in communities with few replacement jobs. The public narrative pointed to market efficiency and global competition, with a strong side order of personal responsibility.

The public answer was retraining. Workers were expected to move into cognitive jobs, and that expectation morphed into a slogan that is now a cultural artifact in its own right: “learn to code.” And it was not only a slogan. Mined Minds was a nonprofit founded by Amanda Laucher that ran free coding bootcamps for laid-off miners in West Virginia and Pennsylvania. From around 2016 it drew national coverage as a model of what retraining could do.

In May 2019, roughly sixty former West Virginia students filed a class action against it. They alleged that the promised stipends, apprenticeships, and job placements never arrived, and that they had put their lives on hold on the strength of those promises. Laucher said the filing had no merit and pointed to a student contract stating that no job or further training was guaranteed upon completion.

Mined Minds became a national story about retraining as redemption. But the allegations made that story a good deal harder to defend.

The prevailing language throughout was economic adjustment. The machinist’s embodied knowledge never sat at the center of that discussion. The advice was to climb the ladder into the safe work, but the safety of that work was only assumed and never guaranteed.

Then the same expectation was dropped onto graphic designers, copywriters, and junior developers, in the form of “learn to prompt.” This time it produced a backlash.

I do believe some anger is justified. Telling someone whose job just vanished to “learn to prompt” is dismissive. But this way of dealing with job displacement did not become dismissive when generative AI appeared. It was always dismissive. It is just that most creatives and knowledge workers had no particular reason to notice, because it was always aimed at somebody considered less important.

In the end, “learn to prompt” and “learn to code” are just two versions of the same playbook.

The machine has an owner

For centuries, economic and cultural institutions treated the mechanization of physical capability as a necessary cost of progress. The scribe saw a sacred purpose in the pen, and the weaver saw their whole identity in the loom. In both cases that attachment counted for nothing, because the market had already made its decision. The occupations kept changing. But the governing hierarchy did not.

In reality, alienation under technological change has never depended on whether the work is intellectual or physical. It always depended on who owned the machine.

That ownership question has a practical consequence. A defense of creative work built on the special status of creative labor can only ever recruit the people who already do creative work. An argument about ownership, however, asks who captures the gains and who carries the loss, and a laid-off miner and a displaced illustrator can ask that one together.

Creative exceptionalism is the argument we mostly have, and it shows. Many working-class observers watch the creative panic with open cynicism. When people appeal to humanity only after their own jobs are threatened, the appeal sounds like self-protection.

Which brings me back to the sentiment I started with. Objecting to AI automation by itself is not elitist. But objecting to it only once it reaches people with college degrees is. The weavers made that exact claim about their looms and it got them nowhere. The claim itself has not changed. But the class of the people making it has.

De Strata and the weavers lost. Readers got cheaper books and buyers got cheaper cloth. Those gains were real, but so were the losses. Those who absorbed them had no control over how either was divided. The people who owned the press and the loom decided all of it.

Nobody with power wept for the weavers. That is the part we are currently on course to repeat.

The images in this article were generated with Nano Banana 2.

If you’d like to go further, the following NotebookLM-generated audio deep dive goes beyond the post into the broader research behind it, drawing on the sources and notes I gathered along the way. This is meant as a companion to the argument, offered as an optional extra rather than a summary of it.

Share

P.S. I believe transparency builds the trust that AI detection systems fail to enforce. That’s why I’ve published an ethics and AI disclosure statement, which outlines how I integrate AI tools into my intellectual work.

We need to talk about AI detection again. I know, I have been writing about it a lot lately. But the problem keeps widening, and this time the failure looks different from anything I have covered on The Augmented Educator before.

In my last post, I argued against AI text detection on mathematical grounds. Human writing and machine writing produce overlapping score distributions. Some AI-generated passages will always score as more human than some human-written ones, and vice versa.

Any educator who uses a detector therefore has to set a threshold somewhere inside that overlap. And wherever that dial lands, somebody always pays for it. The two distributions do not permit a clean separation. No threshold can create one. I called the piece There Is Only One Dial and hoped it would close the case.

But, as I came to realize, it closed only a quarter of it.

The other three quarters concern AI-generated images, video, and audio. Education has barely discussed these systems, mostly because they usually operate outside the educator’s view: in stock library upload queues, music distribution pipelines, journal submission portals, and platform moderation systems. Photographers and musicians have been arguing about these tools for years. Most educators have not heard of them.

They should. Because I think these detectors may pose an even greater risk to our students than the text classifiers we have spent so much time debating.

Subscribe now

The D’Addario commercial

This piece began, as many of mine do lately, with a YouTube controversy. This time, the trigger was a commercial containing a piece of music that was accused of being AI-generated.

D’Addario is a family-run American manufacturer of music accessories and one of the best-known names in guitar strings. If you play, you have almost certainly had a set of theirs on an instrument. In July 2026, the company launched two extended-range string lines and posted a promotional video built around a high-gain progressive metal track.

Within hours, the comments were filled with accusations. The performance had an odd digital sheen, and the timing was quantized so tightly that it no longer felt played. It lacked the microdynamics that make a guitar sound like a guitar.

The accusation was clear: D’Addario had skipped the musicians and typed a prompt into the AI music generator Suno.

What followed made it worse. D’Addario deleted comments, blocked accounts, and eventually switched comments off completely. The company later explained that the employee who made the track had been doxxed and was being harassed. The moderation was meant to protect him. Whatever the intent, an audience that already suspected a cover-up read the silence as confirmation.

Then a behind-the-scenes video of the project session, posted by D’Addario as proof, drifted out of sync with the commercial, which viewers took as further evidence of fakery.

But that assessment turned out to be wrong. Rhett Shull, a guitarist and producer whose YouTube channel covers gear for a large audience of players, obtained the original Logic Pro session from the company and audited it. The project contained real recorded guitar performances and hand-programmed MIDI. No text prompt ever wrote that song.

What the session did contain, however, was a production chain pushed until it broke: pitch-corrected direct-input guitars, drum compressors stacked in series, every MIDI velocity pinned at maximum, and a dozen synthesizers packed into the midrange. And then, after export, the D’Addario employee added several rounds of automated mastering. Rhett Shull and Steve-san Onotera, another guitarist and YouTuber who posts as samuraiguitarist, both suspected that one of those mastering passes went through Suno.

Mastering is the final stage of music production, when an engineer makes the adjustments that prepare a song for release. A skilled mastering engineer can make a mix sound louder, clearer, and more coherent without ever drawing attention to the work. And some of that work is now handled by automated services such as LANDR and by mastering assistants built into digital audio workstations.

What is usually not known is that Suno’s approach to mastering works differently from these automated systems.

Conventional automated mastering analyzes a finished stereo file, then applies equalization, compression, and limiting to it. Those are the same operations a human engineer would reach for, selected and adjusted by a model.

Suno’s version, on the other hand, runs the audio back through its generative engine and completely rebuilds it. The result is a new waveform rather than a processed copy. And the consequence of that is that the finished waveform is machine-generated, even though the performance underneath it is not.

I find that explanation convincing as an account of why D’Addario’s track sounded artificial, but it remains an educated guess. At the time I am writing this piece, D’Addario has not confirmed which tools were used at that stage, and nothing in the published forensic analysis can prove it either way.

Part of the reason that question is still open is the underlying systemic failure Onotera pointed to.

D’Addario’s communications staff were defending a technical claim they did not understand, with evidence they could not evaluate. And the music professionals watching them did not necessarily understand generative AI any better. The company needed over a week of internal investigation before it could describe its own production chain with reasonable accuracy.

Hold on to that diagnosis. Of all the elements in this story, it is the one with lasting significance.

Thirty-six percent of nothing

In the same video, Onotera described an experiment that should worry any creator deeply. He took a track from his own catalog: human-composed, human-performed, conventionally recorded, with no generative anything anywhere in the chain. He then uploaded it to the AHA Music AI detector, an online tool used across music distribution and content monitoring workflows.

The result was 36 percent AI-generated, with 95 percent confidence. For anyone familiar with the limitations of AI detection, this number should raise a big red flag.

Then he ran the disputed D’Addario track through the same tool. The result was 74.7 percent AI-generated, with 95 percent confidence. Other detectors gave the same file a “Suno match” score of between 73.88 and 91.40 percent.

If the mastering suspicion is right, those tools were picking up a genuine Suno signature sitting on top of a human performance. They may have been correct about the artifact, but they were wrong about everything anybody should ever really care about.

Let’s look at those numbers more closely, starting with that confidence figure, because it is the part people misread the most. It expresses how certain the model is within its own system. It does not independently validate anything. Onotera’s control experiment shows why that distinction is worth making: the tool was highly confident about a result we know was wrong.

A confidently wrong number is more dangerous than an openly uncertain one, because it gives a guess the authority of a measurement.

There is also something hidden in the percentage itself. Audio engineers have a word for it. The noise floor is the bed of hiss underneath a recording, and once a signal falls below it, pulling the two apart gets difficult. In Onotera’s test, 36 percent on a track with no generative involvement at all behaved like the detector’s noise floor. More than a third of the scale had already been used up before any meaningful measurement was taken.

A single control track is not a proper statistical sample. But it does show what that number is not. It is not a literal estimate of how much of a recording was generated with AI. The D’Addario track scored higher, and the tool could not tell anyone what it had found. Generated composition? Generated performance? An automated mastering pass? Or simply the artifacts of very heavy production?

The scale collapses all of those into one figure and hands you a percentage. That is a different problem from the one in my last post, but it is arriving at the same place. There, two overlapping distributions meant no threshold could cleanly separate them. Here, the scale is not measuring what people think it does.

Two ways to be wrong

Research shows that text detectors and media detectors both fail. But they fail in opposite directions, and they take down opposite people on the way. A policy written for one of them will therefore be wrong about the other.

Text is made of discrete tokens. A language model produces words by drawing the next one from a probability distribution, and a detector measures how surprising the resulting sequence is to a reference model. That is perplexity. Alongside it sits burstiness, the variation in sentence length and structure across a passage. Machine text tends to run smooth and evenly paced. Human text tends to be less steady. Those are two of the signals most text detectors lean on.

The failure mode falls straight out of this mechanism. Careful, plain, grammatically conservative writing scores as machine writing. The detectors punish clarity, and they punish it hardest in the people who worked hardest to achieve it. I have written about this problem at length in previous posts on this Substack.

AI images, video, and audio are different mathematical animals. They are continuous, high-dimensional signals, and most current generators build them through diffusion.

Generation starts with static noise and strips it away step by step until a picture or a waveform emerges. Detectors for this kind of content hunt for the traces that denoising leaves behind. Those include reconstruction errors, frequency-domain artifacts, and spectrogram phase relationships that a physical microphone is unlikely to produce.

But those traces are fragile. On the GenImage benchmark, which tests detectors across eight major generators including Midjourney and Stable Diffusion, standard classifiers average 71.7 to 72.5 percent accuracy on architectures they were not trained on. Push a synthetic image through ordinary JPEG compression or scale it down, and reported accuracy falls to between 50.6 and 67 percent.

At the bottom of that range, the detector is barely doing better than a coin flip.

Passive audio detectors show the same collapse, with false negative rates of 20 to 40 percent once a file has been through MP3 or AAC encoding at consumer bitrates. 20 to 40 percent! Between one in five and two in five synthetic files sail straight through.

Active watermarking was supposed to solve the problem by signing content at the moment of generation, and under clean conditions it does indeed work. Tree-Ring watermarking reaches a ROC-AUC of 0.993. ROC-AUC measures how cleanly a classifier separates two classes, so 0.993 is close to perfect.

But researchers then showed that the mark can be stripped using the same publicly available autoencoder the model itself relies on. The score falls to 0.153, and the true positive rate drops from 96.8 percent to a mere 4 percent, with no visible damage to the image.

So text detection primarily fails by accusing the innocent, whereas media detection fails by waving the guilty through.

Subscribe now

The tools that sand off the fingerprints

Media detection has a second failure mode: the ordinary enhancement tools creators already use erase the very traces that forensic detectors depend on. This one takes a little technical ground to cover. So, bear with me.

One of the signals a forensic image detector looks for is Photo-Response Non-Uniformity, or PRNU. This is the microscopic pattern of variation between individual pixels on a physical camera sensor. PRNU is unique to that sensor and burned into every frame it captures. It is physical noise produced by one specific piece of hardware.

A generated image has no camera sensor behind it, so some forensic systems treat the absence of that signature as suspicious.

Now think about what a photographer does to a high-ISO file. They use post-production tools such as Adobe AI Denoise, DxO DeepPRIME, Topaz Photo AI, or Luminar Neo. Those tools treat sensor noise as the problem they exist to solve. They suppress it, then reconstruct the fine detail with the help of AI models. The photograph comes out looking better, but it comes out carrying less of the evidence that a camera made it.

The signature that proved it came from a camera has been professionally removed and replaced with the output of a neural network.

Generative upscalers go even further. Tools such as Magnific do more than interpolate the pixels that are already there. They invent the missing detail with a generative model, which can leave behind the same high-frequency artifacts that detectors associate with fully synthetic images.

Audio works the same way, except that the markers being erased are acoustic rather than optical.

What proves a recording happened in a room is the room itself. It is the reverberation profile, the subperceptual noise floor, the breathing, and the continuous way a voice slides from one sound into the next.

Adobe Podcast’s Enhance Speech and Descript’s Studio Sound are designed to strip much of that away. They then rebuild the missing frequency bands with neural vocoders. And a vocoder can leave phase shifts in the upper harmonics and flatten the energy contours. Which is the kind of signature a deepfake voice detector is trained to flag.

Similarly, automated mastering platforms such as LANDR, iZotope Ozone, and eMastered reshape high-frequency harmonics and phase relationships across a stereo master. And if you run human stems through Suno’s generative mastering, the resulting file will pick up the acoustic marks of AI generation, with the instrument dynamics altered along the way.

Human production and machine generation are also converging acoustically. Generative models were trained on commercial music that had already been pitch-corrected, quantized, dynamically limited, and stereo-widened, so they learned to produce exactly those artifacts.

When a human engineer pushes the same techniques far enough, the output starts to arrive at the acoustic profile of generated music.

The repercussions of all of this are already being felt. Researchers have to defend themselves against integrity investigations for running micrographs through routine denoising. And photographers are losing stock library accounts over files their own cameras produced.

There is nothing left in the file to find

Handing an algorithm a finished MP3, JPEG, or MP4 and asking whether a human made it asks the file for a history it no longer carries. A final export is a rendered surface. It holds the statistical residue of whatever touched it last. And in professional work, what touched it last is very often a neural network doing legitimate enhancements.

Most passive commercial detectors reduce the question to a binary classification. They assume each asset is either wholly human or wholly machine-generated. Yet hardly anything created professionally today is exclusively one or the other.

A photograph shot on a physical camera and upscaled by Magnific is a hybrid. So is a song played by a person and mastered by Suno. The composition, the performance, the judgment, and the meaning are human. The surface statistics are not. But the detector reads only the surface, because the surface is all a passive classifier can reach.

Which brings us back to the problem with text classifiers, only with greater force. Post-hoc media detection is not an engineering problem awaiting a better model. It asks a question about the history of a file, and the exported file preserves only a fraction of that history. A better classifier cannot reconstruct a provenance record that nobody kept.

Verification therefore has to move upstream, into the process. Under the C2PA standard, a creator can attach a signed manifest at capture and record every later edit in a tamper-evident chain. A platform could then see that an image came out of a participating camera and was refined with Topaz afterward.

Workings.io aims to apply the same logic to creative work more broadly. It is in closed alpha at the moment. I have been testing it and plan to report on my experience in a future article.

The low-tech version of this approach is straightforward. Save the camera RAW files and the multitrack sessions, in case somebody later needs to inspect them. D’Addario was cleared because the Logic session still existed and somebody could open it. That is the entire mechanism. Not a score. An artifact a person could inspect.

You do not have to use AI tools to be judged by them

A photographer who has never typed a prompt adds AI-like signatures to his work just by denoising a high-ISO file. A podcaster introduces vocoder artifacts into her own voice by cleaning up a recording. An illustrator adds diffusion traces while upscaling a hand-drawn image for print. And a student who writes plainly because English is their third language walks into a 61 percent false-positive rate.

None of them tried to pass off generated work as their own. Some may not even have realized that an enhancement tool was running a generative model. Yet all of them can be scored by AI, and some can be falsely accused because of it.

That is why AI literacy is a baseline professional requirement and not a niche interest for technologists. Learning to use the tools is only half the job. The harder half is being able to explain what an automated score is measuring, and why it may be wrong.

A company with engineers on staff needed more than a week to explain what had happened to its own audio file. A student before an integrity panel gets far less time, far less help, and a panel that usually cannot interpret the score either.

The guitar community was right that something was off about that track. It had been polished into a lifeless sheen, and in a demo whose whole purpose was to let you hear a string respond, that is a real failure. It deserved the criticism. But the crowd reached for the wrong explanation, and when the detectors were brought in to settle it, they appeared to confirm it. But they were wrong.

They were reading the noise floor, not the work.

The images in this article were generated with Nano Banana 2.

If you’d like to go further, the following NotebookLM-generated audio deep dive goes beyond the post into the broader research behind it, drawing on the sources and notes I gathered along the way. This is meant as a companion to the argument, offered as an optional extra rather than a summary of it.

Share

P.S. I believe transparency builds the trust that AI detection systems fail to enforce. That’s why I’ve published an ethics and AI disclosure statement, which outlines how I integrate AI tools into my intellectual work.

I have written on this Substack about open-source models before, and today, I am going to write about it again. This is because I think it is critically important for educators to understand that AI is a general technological shift in how we interact with computers, and that the big foundational language models are only one small part of it. The other part, the one that gets considerably less attention, is the open-source ecosystem of models that anyone can download and run on their own hardware.

The claim I keep making is this. If every foundational model disappeared tomorrow, the technology itself would not go anywhere. And last week, two releases made that claim harder to argue against. The first is Qwen3.8, an open-source model whose 27B version (27B stands for 27 billion parameters) can run on an advanced home computer setup and handle the kind of work you would normally send to a cloud service. The second is Unsloth Desktop, a free, open-source desktop application that makes running and training that model, and others, as simple as opening a document.

Subscribe now

Neither is enough on its own. A capable model buried in a Python script is not a replacement for the chat window you are used to. And a friendly interface pointed at a weak model is just a chat box with extra steps. But put them together, and you have something that actually works.

I am running that combination right now on a MacBook Pro with 128 gigabytes of shared memory and it works remarkably well. So in today’s free bonus post, I want to walk through what a fully local, fully open-source AI environment looks like in practice, and explain why I think it is a preview of how we will interact with AI in the future.

The entire process behind this post, from the initial research through the drafting to the final editing, ran through Unsloth Desktop with Qwen3.8 doing the model work. The only tool I kept from my old workflow is ProWritingAid for copyediting, and this runs locally on my machine as well. I did not use Claude, ChatGPT, Gemini, or any other foundational model anywhere in this process. Everything happened on my machine, and no data ever left the computer.

How a fine-tuning library became a desktop app

Unsloth started in 2023 as a passion project by two brothers, Daniel Han and Michael Han, in San Francisco. Daniel has a background at NVIDIA and leads the technical side. Michael handles design and product engineering. Together they built an open-source Python library for fine-tuning large language models, which they released on GitHub in December of 2023.

The library solved a real problem. Fine-tuning a model, the process of training it further on your own data so it does a specific job better, used to require serious hardware and a fair amount of coding. Unsloth’s core contribution is a set of custom kernels, written in a language called Triton, that make the training run faster and use less memory. In their standard benchmarks, fine-tuning is about twice as fast and uses roughly 70 percent less video memory than the standard pipeline.

That is the difference between needing a data center and being able to do the work on a single graphics card, or in some cases, on a laptop.

The project gained traction quickly. It went through the GitHub Accelerator program in 2024, then the Y Combinator summer batch that same year. The team is still small, currently eight people, and they have so far raised half a million dollars across one round. The library’s repository has so far collected over 73,000 stars on GitHub, which in open-source terms is a clear sign that many people actually use it.

Unsloth has since published compressed versions of many popular models on Hugging Face, including the Qwen3.8 model I am running. And they even pushed some of their fixes back into OpenAI’s own open-source model repository.

Share

Two front ends on the same engine

By early 2026, the group had built two front ends on top of their library. The first, Unsloth Studio, launched in beta around March of this year. It is a no-code web interface for training, running, and exporting open models. You do not need to write a single line of code. You just load a model, point it at your data (it can build the training dataset automatically from a PDF, a CSV, a Word document, or a text file), and start a training run. Everything fits on one screen, which is a deliberate break from the notebook-and-script workflow that the original library required.

Unsloth Desktop is the newer and more ambitious project of the two. It is a native application for macOS, Windows, and Linux, built on a framework called Tauri. It is also free and open source, and it runs entirely on your machine with no telemetry and no requirement to be online. The headline on the company’s website is blunt: the first desktop app to run and train AI models, open source, free, 100 percent local.

Unsloth Desktop primarily runs large language models for text, but it can also run diffusion models that generate images and video, as well as audio and speech. And it can train all of them, using the same fast kernels that the original library was built around.

But the app does more than just run and train models. It has a built-in web search and a deep research mode. It can also execute code in a sandbox and make tool calls. And it plugs directly into coding agents so you can point a local model at your project and swap models without changing your workflow. The API it exposes speaks the same language as OpenAI’s, so existing scripts and apps can connect to a local model without being rewritten.

The development trajectory of Unsloth is as clear as it is ambitious. The library was a tool for engineers who work in code. Studio removed the code. And Desktop is a full workstation that goes beyond fine-tuning into running models, generating images and video, doing research, and talking to other tools.

Each step has lowered the barrier and widened the audience. The company’s stated mission is to help builders create custom models faster and better, and the products are moving steadily toward making that possible for people who would never write a training script.

What people are saying about it

Initial reviews of the software are mostly positive. One write-up in Towards AI framed Unsloth’s contribution as genuinely democratizing model customization, and a developer community article put it on a list of essential open-source libraries to know. On Hugging Face, where the models are published, users are enthusiastic about the web search and the code execution capabilities.

There are a few caveats. One user reported trying the desktop app and going back to LM Studio because it was missing too many advanced inference settings, which is a sentiment I can echo. And one reviewer made a critical technical point worth mentioning: the speed figures Unsloth publishes are training benchmarks from specific model tests, and not a guarantee of how fast the desktop app will actually feel on your hardware.

Subscribe now

A preview of how this will feel

I honestly think Unsloth Desktop might be the first desktop app that shows how we will interact with AI in the future, and specifically how we will interact with local AI. It is the first time I have been able to complete an entire Substack post, from research and drafting through editing to image generation, effectively in a single application that runs on my own machine and sends nothing anywhere. (I am still using ElevenLabs for the voiceover, merely out of convenience.)

In my testing, I encountered only a few minor issues. The deep research function timed out on me a few times. I have heard from other users that they were having the same experience, which leads me to believe that this is probably something that will be fixed in an update.

And there is the obvious speed issue. Because everything runs locally, it will not feel as fast as the online foundational models you are used to. A home computer is not a data center, and you will experience the difference when you are waiting for a long response.

One thing I found particularly exciting is that the app works well with MCP servers, the standard that lets AI tools talk to external resources. I was able to connect Qwen3.8 to Consensus, a service for searching and verifying academic sources, and it handled the search and the verification as expected.

So where does that leave me? Well, I do not see myself switching completely to Unsloth Desktop right now. The beta has a few rough edges and the slower speed is a real downside. But the system has every feature I would ever need. In combination with an advanced model such as Qwen3.8 27B it does the research, writes, and edits exceptionally well. And it can also generate images, video, and audio with the help of open-source diffusion models. Everything runs locally, is open source and completely free.

I want to encourage you, and I mean this especially for educators, to go download Unsloth Desktop and see what it does. If you have a reasonably powerful computer, you will be surprised by its capabilities.

To be clear, this post is not sponsored and I have no relationship with the company. I just think this is an outstanding tool that demonstrates clearly what local AI already can do today. Open-source tools and models have improved significantly, and the best way to understand how far they have come is to run one on your own machine and test it yourself.

The hero image in this article was generated with Flux.2 [klein], research and drafting was done with the help of Qwen3.8 27B. Everything was generated inside of Unsloth Desktop.

Subscribe now

P.S. I believe transparency builds the trust that AI detection systems fail to enforce. That’s why I’ve published an ethics and AI disclosure statement, which outlines how I integrate AI tools into my intellectual work.

Every company selling AI detection to educational institutions leads with a number. Turnitin, GPTZero, Copyleaks, and Pangram all publish one, usually an accuracy rate or a false-positive rate, often reported to two decimal places. Pangram, for example, self-reports a false-positive rate of 0.19 percent and a false-negative rate of 1.4 percent on standard datasets. A false-positive rate of 0.19 percent means that out of 1,000 human-written essays, the detector falsely flags only about two texts as AI-generated.

Sounds great, doesn’t it? But unfortunately, none of those numbers can really tell you what will happen in your classroom. Because outside of a controlled lab environment, they have very little meaning.

I have made this point on The Augmented Educator Substack for over two years now, particularly in The Paradox of AI Detection. But so far I have mostly asserted rather than explained what might be the most complex part of this argument: that the trade-off at the center of these tools is subject to a hard mathematical limit. Better engineering can shift that trade-off. But nothing can eliminate it.

Absolutely nothing. The math is baked in.

So in this post, I want to complete my core argument against AI detection. And I have decided to do it in plain language and with no complex math notation. I wanted to keep this approachable to a lay audience. If you are interested in the finer details of this argument, you might want to check out the audio deep dive podcast episode linked at the end of this post.

Subscribe now

How a score becomes a verdict

Let’s start with what a detector actually returns in practice. Underneath the percentages and the categories, it is fundamentally a score. The detector assigns a text passage a number between zero and one, say 0.83 or 0.11, showing how strongly the system associates it with machine-written text.

Early detectors often used that score to return a simple yes/no verdict, which ended up being highly problematic because it outsourced the integrity decision from the instructor to the detector. Most of the products available today therefore stop short of that. They present a percentage or a category and leave the final judgment of what that means to whoever reads the report. This separates the detector’s result from any misconduct accusation. It is now the teacher who makes that call and not the AI detector.

Turnitin, for example, reports what share of a document it estimates was AI-written. It simultaneously cautions that the figure cannot be used as proof of misconduct. GPTZero splits a submission into percentages of AI, mixed, and human. And Pangram sorts text into human, AI-edited, and fully AI-generated, and its EditLens model estimates how much AI editing a passage received.

None of that is an accusation. Every one of those reports leaves somebody else to decide what happens next, and that decision is a cutoff.

Somewhere between an innocent detection report reading some percentage and an email accusing a student of misconduct, a line gets drawn. Statisticians call that line a threshold. Think of it as a dial which the teacher sets, and think of the number it points at as a bar the score has to clear before anyone acts on it. Turn the dial up, and fewer texts clear the bar. The same detector, on the same essay, leads to entirely different outcomes depending on what number that dial points at.

When a company says its tool is 99.85 percent accurate, it is describing the result of applying a chosen threshold to a chosen test set. Move the dial or change the test set, and the advertised accuracy and error rates move with it.

Two piles that overlap

Let’s assume we want to use a detector to score an extensive set of known examples. Take thousands of texts you know a human wrote, for example, and score them all. Then take thousands you know came from a language model and score those too. You get two piles of scores.

If those piles sat in separate ranges, say, with every human text below 0.4 and every machine text above 0.6, detection would be a solved problem. You would just have to put the dial in the empty gap between 0.4 and 0.6 and the system would never make a mistake in either direction.

But the problem is that they do not sit in separate ranges. They never do. In 2023, Dalalah and colleagues found a substantial overlap between the score distributions of genuine and AI-generated academic writing. Later studies have found the same underlying overlap.

Now consider who ends up in that overlap.

Human writing lands in the machine range when it is clean and grammatically conservative. That includes a student writing carefully in a second language. It also includes technical and scientific prose, where uniformity is often a virtue, and short answers, which simply do not give the detector enough to work with. Jung and colleagues took that last problem seriously enough that in 2025 they built group-adaptive thresholds to stop short texts from being flagged at inflated rates.

Machine writing, on the other hand, lands in the human range when it is uneven, or when somebody with an understanding of human writing has edited it.

Once the two piles overlap, the trade-off is unavoidable. Wherever the dial sits, some human texts fall above it and get called machine-generated. And some machine texts fall below it and get identified as human. You cannot eliminate both errors simultaneously because both arise from the same overlapping region. Raise the dial to rescue the honest students, and you miss the AI texts sitting beside them. Lower it to catch those AI texts, and you will take the honest students with it.

And this is not a flaw in anybody’s product. No clever design and no ingenious engineering will ever escape it, because the trade-off belongs to the statistical overlap and not to the software.

Which mistake a school will tolerate

A school, or any educational institution for that matter, has to choose which of the two errors it is more willing to accept. But the problem is that these errors do not carry equal consequences. Missing an AI-written essay compromises an assessment. This is most likely not a big deal. Falsely accusing a student, however, can derail a degree and follow that student for years.

The consequence is that tools that flag innocent students get switched off because their vendors and the institutions that deploy them would otherwise risk being sued.

UCLA and the University of Pittsburgh disabled Turnitin’s AI detection early on. Vanderbilt did the same in August 2023. And OpenAI famously shut down its own detector a month earlier, having shipped a tool that caught only about 26 percent of AI text while falsely flagging about 9 percent of human text.

Flagging innocent students is not an option. And so the dial goes up. It has to, as long as language models continue to improve.

And that changes which numbers are really relevant. A vendor’s advertised accuracy comes from whatever threshold the vendor picked for its own benchmark. A school cannot use that same setting. It has to turn the dial up until the tool wrongly flags only, say, one honest essay in every hundred. And it then has to ask how much AI writing it still catches once the dial is that high.

In 2024, Tufts and colleagues argued that this was the only deployment metric worth reporting. They then tested popular detectors on text from models and subject areas the tools had not seen, using the kinds of prompts a curious student might try. Held to that one-in-a-hundred limit, some of those detectors caught nothing at all. Zero percent.

Now, you might object that these tools were simply not built to catch clever students. So let’s look at one that was. In 2025, Lekkala’s group tested a detector that had been trained specifically on AI text which someone had deliberately reworded to hide its origin. This system already knew what a disguised passage looks like. And yet, held to that same one-in-a-hundred limit, it caught only 48.8 percent of them.

So the safer the setting is for honest students, the less AI writing the detector is able to catch. And the less a dishonest student has to do to slip underneath it. The ones who still get caught at that setting are just the ones who did the least to hide.

Nobody writes down where the dial sits

Now, handing that decision back to the teacher sounds like the responsible thing to do, and in a sense, it is. Read the fine print on any of these products and you will find a version of the same warning. The score is not proof. It should not be the sole basis for an integrity finding. Every word of that is correct.

But look at what it actually does to the numbers. The vendor calculates its accuracy figure at a threshold the vendor selected, publishes it, and then tells you not to rely on it. Meanwhile, the threshold that decides whether a student gets an accusatory email is the one being set in your classroom, by you, and nobody has ever measured what that threshold does in practice.

And it never gets written down. Two instructors in the same department can look at the same 42 percent and draw the line in completely different places. And the same instructor can move it from one semester to the next without even noticing the change.

This is the part I find hardest to defend. A documented number becomes an undocumented disciplinary judgment, made at the end of a long grading day, about a student whose other writing the reader may barely remember. The trade-off between the two errors did not go away when the product stopped short of a verdict. It simply moved somewhere nobody keeps records.

The pile students can move

So far, I have treated the two piles as fixed. But they are not. Evasion creates a second problem, and it makes the trade-off even worse.

Many published benchmark results compare human writing against untouched model output. Someone pasted a prompt into ChatGPT, took the result, and scored it without changing a word. A student trying to avoid detection is highly unlikely to behave that way. They edit. They learn which prompts produce less machine-like prose, and some of them run the result through a tool built for exactly this. Every one of those moves drags the machine pile toward the human pile, and the effect is not marginal.

Krishna and colleagues built an eleven-billion-parameter paraphraser in 2023 and ran AI text through it. DetectGPT, held at a 1 percent false-positive rate, fell from catching 70.3 percent of that text to catching a mere 4.6 percent.

Liang’s group at Stanford needed no software at all. They asked ChatGPT to rewrite its own essays in more literary language, and detection fell from near 100 percent to roughly 13 percent. And Zhang and colleagues halved the performance of state-of-the-art detectors with nothing but changes to the user’s prompt.

Larger studies show the same effect. Perkins tested six detectors on 805 samples and found that ordinary manual editing reduced their accuracy by 17.4 percentage points. The researchers concluded the tools could not be recommended for integrity decisions. Similarly, Sun’s team tested thirteen detectors on over 280,000 pieces of real student coursework. A simple hybrid editing strategy let 88 percent of the AI content through.

I have run the small version of this myself, twice. In An Experiment in Language Laundering, I took an essay GPTZero rated 100 percent AI, pushed it through a commercial humanizer, and watched it come back rated 100 percent human. The whole workflow took under an hour. In Substack Can Now Scan Your Writing for AI, I published a post written end to end by Claude, laundered through the same tool, and Pangram cleared it as fully human with high confidence.

Moving the dial trades one error for the other, but at least the school controls that choice. Evasion widens the overlap itself.

Leave the dial where it is, and your honest students are as safe as they were. But more AI text now slips past. Turn it down to catch that text, and you start accusing honest students again. Either way, you end up worse off than before the students learned to evade it. The overlap grew, and no dial setting can shrink it back.

The advertised detection rate is therefore not just optimistic. It is an answer to a question about a world where nobody is trying to evade detection, published for a world where somebody always is.

Better detectors, same problem

The strongest counterarguments to my position come from researchers rather than vendors. Detectors do indeed improve. An independent 2025 audit by Jabarian and Imas at the University of Chicago found Pangram outperformed every commercial rival by a wide margin on a balanced corpus of nearly four thousand passages. Pangram represents genuine engineering progress. That is not disputed.

And the evasion problem also has known technical countermeasures. The same Krishna paper that broke DetectGPT with paraphrasing showed that searching a database of a provider’s past generations catches 80 to 97 percent of paraphrased text at a 1 percent false-positive rate. Kirchenbauer and colleagues showed statistical watermarks survive human paraphrasing once you have around 800 tokens to examine. And after documenting the 48.8 percent collapse, Lekkala’s group built an architecture that kept an 82.6 percent detection rate under the same attack.

But even granting all of it, the teacher still cannot tell who wrote the essay.

Retrieval searches the language model company’s own records, so it would only help if every provider ran the check. And a student using an open model on their own machine would leave no record with anyone, completely bypassing the system. Watermarking, meanwhile, is a tag the provider has to stamp into the text before a single word is generated, and most of them never did it. Where a watermark exists, it can be copied onto human writing and used to frame a student who cheated at nothing.

Neither of those approaches measures anything about the writing itself. They are bookkeeping. And they only work if there was an AI company involved in the first place, and if that company cooperated before the student ever opened the chat window.

Lekkala’s architecture is the one that actually reads the text, and it is genuinely better at what it does. But it was built against one very specific kind of evasion attack. No student is obliged to keep using that attack. And it never moved the two piles apart. It still has to draw its line through an overlap, which means it still has to choose which of the two errors it is more willing to accept.

Better detectors are possible, obviously. But the question is whether a better detector can solve the teacher’s problem. And as it turns out, it cannot.

Subscribe now

Whose writing is the benchmark

Everything so far has come down to what people do. Schools raise the dial to protect honest students, and students learn to slip below it. But two limits sit underneath all of that, and neither has anything to do with how anybody behaves.

The first is the one most people have heard about. Sadasivan and colleagues proved that any detector’s performance is bounded by how different human and machine text are as distributions. And training a language model is, quite literally, the deliberate act of making those two distributions more alike. So as they converge, the best possible detector necessarily converges on a coin flip.

That is no one’s engineering failure. It is a ceiling that comes down a little further every time a better model ships.

Chakraborty and colleagues pushed back on this argument, though only in principle. As long as the two distributions differ anywhere at all, detection remains mathematically possible. You just need more samples. And that may well work if you are auditing thousands of submissions at once. But it does nothing for a teacher holding one essay by one student on a Tuesday afternoon.

The second limit is the one I find hardest to argue against, and it comes from Garland’s 2026 analysis. It turns out that the two-pile picture I have been using is already far too generous to the detector. It assumes the relevant comparison is between a text and human writing in general. But that is not the question a teacher is actually asking.

The question is whether this particular student wrote it. And a text-only detector has no model of that student whatsoever. It has never seen their other work, so it cannot tell a genuinely unusual sentence from a sentence that is merely unusual for them.

So it substitutes the population instead. It asks whether the text looks human in general. But human writing is enormously varied. Some real students write the way the machine writes. They did not cheat. Plain, conventionally structured prose is what many of them were taught to produce. For others, it is just how they write.

Any detector sensitive enough to be useful will flag some of these students. How many depends entirely on the overlap between genuine student writing and AI output. And that constraint would remain even if the models stopped improving tomorrow, because it comes from the diversity of the students themselves. Which is not a defect anyone should want to engineer away.

What replaces the scan

If the tool cannot answer the question, then the question has to be changed. The industry’s response, however, has so far been to move from detection to surveillance.

Turnitin Clarity records a drafting session for the instructor to replay, which means a student’s entire composing process now lives on a vendor’s servers. GPTZero’s extension performs a lighter form of the same monitoring inside Google Docs. But surveillance software does not become educational technology just because a school is the one that bought it.

I surveyed that market in Students Performing Human and came away convinced that process surveillance makes matters considerably worse. A student who thinks in long silences and then writes in bursts produces a suspicious-looking log. And so does a student using voice-to-text, because their words arrive in a block with no visible construction time, which is exactly the pattern the software treats as a paste event.

So watching the drafting simply creates more places for a false positive to happen. And it teaches students to perform composition for an observer instead of learning how to compose.

What has worked in my own classroom is simpler and slower. I ask students how they used AI before anything is graded, with no threat attached. Those conversations tell me more about students’ thinking than any scan ever has. But they only work when I drop the threat of an academic integrity violation.

And that approach also changes what I assess. If students can produce the finished artifact without doing the thinking the assignment was meant to develop, then the artifact is no longer enough evidence. Assessment has to move into the making of it.

Which means looking at the version history, the abandoned outline, the source journal noting where the student changed their mind, and the paragraph explaining what the model got wrong. What separates all of this from surveillance is consent and agency. A student documenting their own process is doing something a keystroke logger simply cannot do for them, because the documentation is itself an act of reflection.

The fourteen methods I collected in Fourteen AI-Proof Assessment Methods for the Age of Generative Intelligence all follow the same principle. A whiteboard defense works this way. So does an oral examination on a paper that the student wrote three weeks ago. Neither of them requires anyone to guess who typed what.

So it all comes down to this. A detector reports a number. Somebody then has to decide what that number is enough to justify. These days, that somebody is usually the teacher holding the report. And it is that decision, not the detector’s own setting, which determines which failure mode an educational institution is more willing to accept. It cannot eliminate both because some honest students and some language models will always produce writing that lands in the same range.

So, to protect the honest students, the dial has to go up. High enough that a student who spends just twenty minutes learning how to evade it passes beneath it easily. That student barely notices the line. The student who did nothing wrong is the one who has to answer for it.

Which is where the short version of this argument comes from. The more accurate AI detectors get, the easier it is for students to bypass them. Once a detector’s signal becomes a target, students learn to write around it. Goodhart’s Law, in a classroom.

That is the trade-off. It was never hidden. It was just never printed on the box.

The images in this article were generated with Nano Banana 2.

If you’d like to go further, the following NotebookLM-generated audio deep dive goes beyond the post into the broader research behind it, drawing on the sources and notes I gathered along the way. This is meant as a companion to the argument, offered as an optional extra rather than a summary of it.

Share

P.S. I believe transparency builds the trust that AI detection systems fail to enforce. That’s why I’ve published an ethics and AI disclosure statement, which outlines how I integrate AI tools into my intellectual work.

Over the past few months, I’ve watched more and more colleagues quietly step back from AI. Not with a dramatic announcement, just with a shrug. The chatbot continues to write mediocre lesson plans. Students use AI to cheat. In general, the hype has cooled. The conclusion feels all too obvious to them. The technology has plateaued, the moment has passed, and their limited professional energy is better spent elsewhere.

I understand the impulse. And I fully acknowledge that the numbers seem to support that sentiment. Most organizations have now experimented with generative AI. And yet, a 2026 Gallup survey found that only about one in seven U.S. employees use it daily, and nearly half say they never use it at all. If that’s the reality after three years of relentless promotion, surely the revolution must have been oversold.

But this is where the mistake lies. Because that conclusion confuses the interface with the technology. The consumer-facing chatbot, the text box we all learned to use, has indeed largely stopped surprising us. But beneath that familiar interface, the technology is accelerating in ways most of us never get to see.

The thing is that we have been here before. Twenty-five years ago, in fact. And the last time we made this mistake, the people who checked out spent the next decade catching up.

The winter of 2000, when the internet died

On December 5, 2000, at the height of the dot-com collapse, the Daily Mail ran an article declaring the internet a passing fad.

This was not just tabloid provocation. The piece summarized findings from the Virtual Society project, an academic study spanning twenty-five European and American universities. Its director, Steve Woolgar, documented widespread user drop-off. Early web surfers had satisfied their curiosity, realized there was more to life offline, and abandoned their modems. His colleagues added that email, far from delivering the paperless office, had mostly delivered information overload.

At that time, the market agreed. Between March 2000 and October 2002, the Nasdaq lost roughly 77% of its value. Pets.com, Webvan, WorldCom, and Global Crossing, just to name a few, went under. To a reasonable observer in 2001, the digital economy looked like a collective delusion that had finally been exposed. It was a reasonable interpretation at that moment.

But as we know now, this was utterly wrong.

Subscribe now

What the crash destroyed was the speculative valuation, and not the technology underneath it. The bubble’s real legacy was a massive, debt-financed overbuild of physical infrastructure consisting of millions of miles of fiber-optic cable, much of it laid after the Telecommunications Act of 1996. A lot of that cable then sat unused as “dark fiber” while the companies that laid it went bankrupt. That oversupply then drove the unit cost of bandwidth toward zero, carrying broadband into ordinary homes through the mid-2000s.

The effects were measurable. An econometric study of OECD countries, published in The Economic Journal, estimated that each ten-point rise in broadband penetration lifted annual per-capita GDP growth by roughly a percentage point. New technologies, including Web 2.0, e-commerce, streaming, and the cloud, were all built on infrastructure that was financed during the pre-crash mania and dismissed during the post-crash hangover.

Public disillusionment had peaked while the real transformation was still being built. It was just out of sight. The people who wrote off the web in 2001 were not wrong about 2001. They were wrong about the decade that followed.

The trough looks the same from the inside

Generative AI in 2025 and 2026 looks a lot like that moment. S&P Global Market Intelligence reports that the share of companies abandoning most of their AI initiatives more than doubled last year. An MIT initiative, Project NANDA, found that 95% of generative AI pilots failed to deliver measurable financial results. And the RAND Corporation, a nonprofit research organization, puts the failure rate at more than four in five projects, roughly double the rate for conventional IT.

The figures all seem to point in the same direction.

But if we look a little closer, we start to see a different story. Most failures appear to be organizational rather than failures of capability. Companies bought licenses without deciding what problem they were trying to solve. They deployed models on top of siloed, poorly governed data. And they ran impressive pilots, but assigned no one to own them afterward.

A field experiment led by Fabrizio Dell’Acqua showed that while generative AI produces large gains on tasks inside the model’s competence, it can produce negative results on tasks just outside it. This is a boundary the authors call the “jagged frontier.” Failure in AI deployment therefore tells us as much about the organization as it does about the tools being deployed.

Meanwhile, the infrastructure story mimics what happened with the early internet. Hyperscalers are projected to spend in access of $700 billion on data centers, GPU clusters, and energy in 2026, and skeptics reasonably ask when the returns will arrive.

But just as the fiber surplus collapsed the price of bandwidth, the compute buildout is driving down the cost of running capable models. A workload that cost about $60 per million tokens in late 2021 now runs for cents at comparable performance. And as the unit price fell, usage surged. Aggregate enterprise token use at OpenRouter reportedly grew more than fivefold within six months, to more than 25 trillion tokens per week.

25 trillion tokens per week and rising sharply. That is not what a plateau looks like.

The interface plateaued but the technology keeps going

So what is all that consumption used for, if not the chatbots we’ve all grown tired of?

It is going into agentic systems. Rather than waiting for a prompt, answering, and forgetting, an agent can take a broad objective, break it into steps, use tools across different applications, and keep working toward a result. A conventional chatbot answers one turn at a time. An agent can remember, act, and follow through across many steps without human intervention.

You can already see the shift wherever usage data is available. OpenAI reports that agentic workflows overtook conversational usage among its staff within a year, with core departments now generating most of their tokens through agents rather than chat. The shift is also changing internet infrastructure. Cloud providers are launching stateful runtimes for long-running agent workflows, and payment networks are designing “Know Your Agent” frameworks so that autonomous software can transact within regulated payment systems.

Peer-reviewed research studies point in the same general direction, though they don’t confirm those exact usage figures, and they usually measure AI-assisted work rather than autonomous agents.

Shakked Noy and Whitney Zhang, in a randomized experiment published in Science, found that participants completed professional writing tasks about 40% faster and produced better work while doing it. Erik Brynjolfsson and colleagues measured roughly a 15% increase in issues resolved per hour among customer-support agents. And three other field experiments conducted at software companies, published in Management Science, found developers completing about a quarter more tasks.

Taken together, the message is simple. AI is moving from novelty to an everyday tool. But it is doing so in enterprise back offices and developer terminals, well outside the view of an educator who mostly encounters AI through a free chatbot that feels the same as it did last year.

Subscribe now

The fork in the road to 2030

If you take one idea from this article, it should be this one. The workforce outlook for 2030 describes a fork in the road, depending on whether people adapt as quickly as AI advances.

In one scenario, call it the age of displacement, technology outpaces workers. Businesses automate routine cognitive tasks to cut costs. The gains concentrate among a small group of specialists, and everyone else competes for a shrinking pool of narrow, task-based roles. In the second scenario, call it supercharged progress, adaptation keeps pace. Roles are redesigned rather than deleted, and professionals move from doing every task themselves to directing systems of agents.

Here is the uncomfortable part. The direction of the economy is beyond people’s control. They can, however, select how they prepare for it. A professional who disconnects now risks preparing only for the displacement scenario. The internet skeptics of 2001 had a plausible excuse for this move, because back then the evidence was still ambiguous. Today, the direction of AI development is clearer, even if the timing is not.

But what does preparation actually mean? The research points to a specific set of skills. Work published in Patterns by Zhicheng Lin and colleagues identifies three meta-skills that shape whether AI improves or degrades professional work: setting direction, judging quality, and checking the machine’s output.

A related 2026 study finds that such “AI interaction competence” predicts productivity gains better than domain knowledge. You could also call this the agent orchestration skill. That skill doesn’t outsource thinking. It delegates execution to the agents but keeps judgment with the humans.

There is one caveat. A 2026 paper introduced the idea of an “augmentation trap.” Early gains encourage adoption, while long-term passive use can erode the expertise that made those gains possible. But orchestration only works when people keep exercising the underlying AI interaction competence. There is no version of this in which you engage once, feel competent, and then just coast.

Restricting AI is not the same as ignoring it

To be clear, my argument isn’t for implementing AI in every classroom. A teacher may have excellent reasons to restrict AI in a particular course, and I’ve laid out several of them in earlier essays, including what the research on metacognitive laziness and cognitive outsourcing shows about unguided use.

What I’m arguing is that restricting a tool in your classroom and disconnecting from the technology in your professional life are entirely different decisions. The first can be sound pedagogy. But the second assumes that because the utility of the chatbot did not improve, the technology has stopped developing underneath it.

That assumption is dangerous.

Our students will graduate into an economy being rewired inside data centers and enterprise systems we will never see from a chat window. Whether we use AI on a Monday morning matters far less than whether we understand it well enough to prepare our students for how it might influence their future.

We need to design assessments that still require learning and teach the judgment and verification skills our students will need. That understanding requires significant AI interaction competence upkeep. And it has to be renewed continuously because the technology won’t pause while we catch our breath.

In 2001, the internet didn’t need anyone’s belief to keep being built. AI today doesn’t need ours either. Disconnection changes nothing about what is coming. It only changes who is ready when it arrives.

The images in this article were generated with Nano Banana 2.

If you’d like to go further, the following NotebookLM-generated audio deep dive goes beyond the post into the broader research behind it, drawing on the sources and notes I gathered along the way. This is meant as a companion to the argument, offered as an optional extra rather than a summary of it.

Share

P.S. I believe transparency builds the trust that AI detection systems fail to enforce. That’s why I’ve published an ethics and AI disclosure statement, which outlines how I integrate AI tools into my intellectual work.