The enduring je ne sais quoi of LLM writing

You’re trying to manipulate me by doing easy things that you know will work, and the fact that it slightly does work just makes it even more galling!

- Scott Alexander

 

Most people know LLM text when they see it, so why are detection programs so ineffective? And why have LLM coding abilities run so far ahead of their (perceived) linguistic abilities?

In 2019 Sarah Constantin pointed out that if only conscious, focused logical thought can detect a bot, maybe some people will become more aware of when they’re thinking actively vs not. The examples in the article (from the just released GPT-2) are worth looking at, because the magic eye technique for detection works better after you see how the trick works, and what the LLM was doing back then was... a little bit more obvious:

We believe this project is the first step in the direction of developing large NLP systems without task-specific training data. That is, we are developing a machine language system in the generative style with no explicit rules for producing text.

Skimming through it looks fine. Sarah points out that only by concentrating on the logic in the sentence do you realise that a rule for producing text is something like "if the next word starts with a vowel then use an an" - totally different to task specific training data (a collection of books on rocks). The next year GPT-3 was able to generate full paragraphs without obvious logical inconsistencies inside them, and the glitches in the matrix moved to a higher level over time until now its easy for experts in an area to be lulled into a Gell-Mann Amnesia state in related discussions.

Despite the difficulty in spotting their lack of a consistent world model, LLMs are failing the Turing Test hard in a way that humans can detect better than AI text detectors. In an ACL 2025 study¹ five people who used LLMs heavily in their own writing judged which half of nonfiction articles were produced with frontier models, and the majority got 299/300 right. This was better than all of the automated detectors tested except Pangram, which matched them at 99.3% but wrongly flagged 3 of the human written articles against the experts 0. Articles rewritten by the models based on expert feedback fooled the detectors more, humans got all 30, GPTZero 14, Fast-DetectGPT 7, Binoculars 2, and RADAR none.

The expert explanations suggested that structure, excessive formality, thin originality, an oddly uniform clarity, and recurring AI vocabulary were the tells. A year later the ones I currently notice most are contrasting negation, stilted metaphors, and meaning signifiers: Here’s the thing, that sentence was empty calories, not a balanced meal. When writing code comments LLMs love to relitigate history (no theory of mind to look at the conversation as a bystander), and develop their own jargon heavy dialect for discussing changes and pontificating on unimportant aspects (low insight to word ratio is a symptom of not understanding meaning). When writing short stories I would characterise the tells as emotional over-signalling and overall aesthetic cowardice. And generally just acting like a complete simp in conversation.

The two best-known slop word lists also disagree with each other, each one flagging whichever kind of writing it happened to be built from: one says academics are the worst offenders, the other says Jane Austen. The obsious word tells are squished over time, for example when was the last time you saw a model say delve even though that was very common a few years ago. What separates machine from human now is mostly sentence architecture rather than vocabulary, and humans are better at picking that up. Statistical detection degrades with each model version but the expert readers judgement does not².

Human writers drop two-clause replacive (it's not a bug, it's a feature) sixfold when they switch from arguing to explaining. Across 392,000 words by fourteen pre-2021 essayists it runs at 73.6 per million but in in 854,000 words of research papers it falls to 12.6. A 2026 frontier model doesn't make that switch, across 1.36 million words of its own technical prose the same construction runs at 60 per million, delivering the human argumentative rate in an expository register. Feed the same opening sentences to a small 7B base model before and after instruction tuning, and the argumentative phrase "rather than" rises 8.7×, the copular replacive 3.8×, the em-dash 20×. That suggests that rather than being lots of ad copy fed into pretraining (the majority of the training corpus for modern models is web text which has two-clause replacive at around 30/M) this argumentative form is being selected for in post-training.

The contrasting negation in “The fault, dear Brutus, is not in our stars, But in ourselves, that we are underlings” is elegant and draws the mind down from fate and the heavens to Cassius. But Shakespeare was the OG, if you try writing like that today the literati will call you a Bromidiom. But LLMs do have an an overwhelmingly reliance on cliche. Everything is a shadow, an echo, a whisper, a void, a heartbeat, a pulse, a river, a flower—you see it spinning its Rolodex of 20-30 generic images and selecting one at random which Nostalgebraist calls flashy eyeball kicks. RLHF training rewards literary sounding descriptions above the models ability to deliver them tastefully.

A lot of greatness in creativity happens at the ends of the distribution, but a model optimizing for the most likely reward drifts toward the average. With more specific guidance from a high taste user the LLM has more more options other than obvious rhetorical tricks. But if you have ever pushed a diffusion image model with more and more rules you'll see it improves for a while and then things get... weird. My theory is that you can't control the default ick at the same time as generating a clever story - constraints on the ick force prose subtlety which selects for folk wisdom style ideas that flow well. An idea generation planning phase helps, but the constraint of having to then write the ideas subtly without diluting them still ends up both softening the structural choices and leaking LLM tells into the output. I have much better results with a first "ideas" pass with lots of context and few restrictions to generate a smart but ham-fisted story, and then a second "style" pass which takes whatever story the first created and rewrites it based on a number of style rules. The stories that I make multiple course corrections to are still better.

AI detection depends on probing things which are expensive to imitate unless you understand meaning. Obviously humans are good at this, but also our brain is sculpted by reinforcement learning and runs a dopamine based prediction-error-based algorithm, so y'know - we do often fail or wildly succeed with the same lazy shortcuts. LLM tics will continue to be squashed, but equally humans are going to co-evolve into communicating at a higher level denser LLM style because it works, and this will have as much influence on future language as our OG Shakespeare did.