Hallucinations: What AI Gets Wrong About Originality

Here's a pattern we've heard from working creatives often enough to take it seriously. A painter with twenty years of practice — the kind of person whose studio smells like turpentine and coffee and something else you can't name — asks an image model to make "a painting in my style, but of my daughter." She gives it several dozen of her own paintings. She describes, in considerable detail, what she's going for.

What comes back is a painting. It's competent. It's, in a way, in her style — the palette close, the brushwork close enough, the composition the kind of composition she'd make on a decent afternoon. It's also, in the way working painters keep describing it to us, deeply not her. Not almost her. Not her on a bad day. Her style sanded down to the part that looks the most like everybody else's style. Her voice with the weird bits filed off.

The reaction we hear back is often a kind of wry relief. The painter had been, until that moment, slightly afraid the tool was going to replace her. What she saw instead was the tool's actual ceiling, and it was lower than she'd feared. A good version of the average. The thing she'd spent twenty years trying to get away from.

This is the second entry in a short series we're writing at a-gnt about what AI hallucinates — not in the technical sense of "gets facts wrong," but in the stranger, older sense: what it sees that isn't there, what it fails to see that is, and the specific shape of its blind spots. The first entry was about grief, which turns out to be a different conversation and the same conversation.

This one is about originality. And the claim is this: AI tools are oddly, specifically, and almost hilariously bad at originality. Not "bad right now, wait a year." Bad in a way that is structural and is not going away. If you understand why, you understand both why creative people feel the floor shifting and why the floor isn't actually shifting the way everyone seems to think it is.

The centroid problem

Here is the thing nobody says plainly enough. A large model trained on a corpus of human creative work does not produce new work the way a human does. It produces the center of its training data, weighted toward whatever it has been recently asked for.

Ask a model for "a painting in the style of Hopper" and you will not get a Hopper painting. You will get the part of Hopper that looks most like Hopper to the largest number of humans — the diner, the light, the lonely woman in the lamplight. You will get the Hopper that has been Hopper'd the most on the internet. The actual Hopper, the working Hopper in 1942 making a specific painting on a specific Tuesday because he couldn't shake an image in his head, did something weirder than that almost every time. Hopper wasn't interested in being the most Hopper. He was interested in the painting.

Originality, as working creatives understand it, is not a central tendency. It is the opposite of a central tendency. It is the specific, off-center, slightly wrong choice that nobody saw coming — including, crucially, the person who made it. Most originality looks like a mistake at first. A lot of it looks like a mistake at the end, too, if you show it to the wrong person.

AI models are extremely, statistically, beautifully trained to avoid the thing that looks like a mistake. Their entire economic and technical function is to produce the most likely next token, the most likely next pixel, the most likely next chord. Originality is by definition the least likely next token that is nonetheless the right one. These two things are pointed in opposite directions, and no amount of fine-tuning gets you out of the geometry.

The painter's daughter came back looking like the average of every painter who has ever painted a daughter. It was not a bug. It was the model doing the thing it was built to do.

Opinions that are just averages

Here's another failure mode, and it's more subtle.

Ask any major AI model what it thinks about a book, a painting, a song, a film. Ask it specifically — "what is the weakest passage in Beloved?" or "which track on Blue is the one that almost doesn't work?" Ask it to commit.

What you will get is an answer. The answer will be reasonable. It will be the kind of answer a smart graduate student would give on a midterm if they were a little nervous. It will contain almost no position that hasn't been taken many, many times already in the parts of the internet the model was trained on.

This is because the model's "opinion" is not an opinion. It is a kind of consensus projection — a plausible central answer drawn from the many answers humans have given to similar questions. It is the Wikipedia of opinion: reliable, middle-of-the-road, passive-aggressively balanced, allergic to taking a risk.

A real critic — the kind of reader a working writer wants — has a position. The position is sometimes wrong. The wrongness is part of what makes it useful. A critic who thinks the weakest passage in Beloved is something unfashionable to say, and says it anyway, is doing the work of criticism. A model that tells you the weakest passage is the one most frequently discussed in essays as "perhaps overwritten" is doing a book report.

This matters for creatives for a specific reason: the feedback a creative needs is not consensus feedback. It is idiosyncratic feedback from a specific reader with specific taste, willing to be wrong in an interesting way. The average of all possible feedback is the least useful feedback a creative can get. It is the feedback that sands you down to the center of the distribution, which is exactly where your friend the painter did not want to live.

The incapacity for specific weirdness

The third failure is the one that's hardest to describe without sounding like a romantic. But it's real, and you can feel it if you've been paying attention.

Creative work lands — when it lands — because of specific weirdness. The sentence nobody else would have written. The cut nobody else would have made. The one chord change in the third verse that shouldn't work and somehow works. The detail in the short story that seems to have nothing to do with anything and turns out to be the whole point. The thing a critic can't explain but keeps pointing at.

This kind of weirdness is not random. That's the part people miss. It's not "try anything unusual." It's the specific unusual thing that emerges from a specific person's specific obsessions, and it only works because that person made it. If you transplanted the same weirdness into someone else's work, it would be affectation. In its native context it is signature.

Models cannot produce this. Not because they lack creativity — "creativity" is a trap word, let's not use it — but because the weirdness is produced by the friction between a specific human life and a specific craft problem, and models do not have lives and do not have craft problems. They have a loss function and a corpus. The loss function penalizes specific weirdness; it rewards most-likelihood. The corpus is the very thing the weirdness is supposed to be reacting against.

You can see this concretely with any image model, right now. Ask for "a painting of a hallway with one specific wrong thing." You will get paintings with wrong things in them. None of the wrong things will be specifically wrong. They will be generically wrong — a wrong thing pulled from the catalog of wrong things the model has seen. The specificity — the wrongness that makes you lean in and stare — is not in there.

The inability to be wrong in an interesting way

This is the subtlest of the failures, and possibly the most important, and it took me the longest to notice.

Creative work is in large part an argument with the current consensus about what is good. Most of the writers, painters, musicians, and filmmakers I admire have at least one thing they are stubbornly, almost embarrassingly wrong about. They have a taste nobody else has. They're sentimental about something the fashion of the moment considers gauche. They're allergic to something everyone else likes. They have a position. The position is idiosyncratic. Some of it they will defend. Some of it they will quietly drop in ten years. Some of it will turn out to be the thing they're remembered for.

Models do not have positions like this. Models have positions shaped like the average of positions. If you ask a model to take an unfashionable stance, it will produce a paragraph that looks like an unfashionable stance while remaining, on inspection, entirely safe — the stance a reasonable person might take if they wanted to pretend to be interesting at a dinner party. It will be wrong, but not wrong in a way any actual human at the dinner party would bother to argue with.

This matters because a creative who is trying to develop a voice needs to be wrong sometimes. They need to make choices that don't pay off. They need to commit to an image or a line or a harmony that the room doesn't love. Models cannot teach this by example, because their examples are all hedged. Models cannot teach this by prompting, because when you ask them to be idiosyncratic they produce the idea of being idiosyncratic, which is the opposite of the thing.

If you want to feel this acutely, go read an AI-generated essay about why some widely-loved book is overrated. The essay will not be wrong. That's the tell. A real human being with a real position would have said something that made you want to fight them. The model's version of "unpopular opinion" reads like a press release for having opinions.

Regression to the most-represented

Finally, and this is the one the industry does not like to talk about: models regress, hard, toward the most-represented style in their training data. Not the best style. The most-represented.

For writing, this means American middle-brow literary prose with a certain kind of MFA polish. For painting, it means a composite of the art that got the most internet impressions in the last fifteen years — heavily weighted toward a few aesthetic moments that were popular on certain platforms. For music, it means the production style that was inescapable in the mid-2010s, because the corpus is dominated by it. For design, it means the Figma-era look: rounded corners, pastel gradients, the specific shade of purple that was everywhere in 2021.

You can nudge the model away from this, and you can nudge it pretty far with effort. But the gravitational pull is always there. If you release the pressure for one prompt, the output drifts home to the center.

The center is not a neutral place. The center is a specific aesthetic, produced by specific historical conditions in specific parts of the world that happen to be overrepresented in the training data. When a model "decides" what is normal, it is reproducing the biases of whoever had the most bandwidth to get their stuff onto the internet between 2005 and 2022. That's not a canon. That's a census.

For a working creative, this is a trap. It's a trap because you can do all the right things — give the model detailed instructions, use a dozen references, specify your influences — and still end up with work that quietly smells like 2019. Not because you failed, but because the floor of the model is 2019, and everything you build on top of that has some of that floor in it.

This is the part where I tell you it doesn't matter as much as you think

And here's the turn.

Having said all that — and meant it — here's the counter-argument, and it's not a small one. The fact that AI is bad at originality does not mean AI is useless to original people. It means AI is useless for the part of the work that is about originality, and useful for almost everything else. The trick is knowing the line and not crossing it.

This is not a consolation prize. This is a real distinction that changes what the tool is for.

The creative core — the specific wrong choice that becomes a signature, the voice that emerges out of obsession, the decision that goes against the room — that's yours. It has to be yours. Nothing in the model can make it or even imitate it, and every attempt to offload it is going to produce the painter's daughter again: the thing that looks right at a glance and makes you feel nothing.

But creative work is not just the core. Creative work is a whole apparatus. There's the part where you have an idea and can't tell if it's any good. There's the part where you have a project that's fuzzy and you need to name it in a sentence to pitch it to a friend. There's the part where you're stuck on a specific scene and don't know whether the problem is the scene or the chapter. There's the part where you want someone to read the draft and tell you three things that are working and one thing that isn't. There's the part where you need to understand what's consistent across five years of your own work and what's drifting. There's the part where you need to remember what you made last March.

These are all real parts of a creative life. They are not the core. AI is good at several of them, exactly because none of them require the thing AI can't do.

Unsticking. When a project stops moving, the block is almost never a failure of originality. It's a failure of clarity. A diagnostic conversation — the kind you could have with a smart friend over coffee, if your friends were available at 11 pm — is something a well-designed prompt can actually do. 🔓Unstick Your Creative Brief exists specifically for this: five questions calibrated to your medium, then a name for the block, then one small experiment. It won't tell you what to make. It will tell you what the knot looks like from outside your head.

Treatment writing. Turning a fuzzy idea into a pitchable one-page treatment is a translation problem, not an originality problem. You supply the idea; the model helps you put it into the shape it needs to be in to move through the world. 📃One-Page Treatment Builder is good at this because the shape of a treatment is learnable, and the model has learned it. The originality stays with you; the shape is a shared technology.

Draft reading. A real human reader with taste is irreplaceable and you should absolutely find one. But most of the time you don't need a real human reader; you need someone to tell you that chapter three is protecting you from something, or that the opening paragraph is doing work the second paragraph is supposed to do. A well-constructed reader persona — 📖The Draft Reader — can do this specific job surprisingly well, because the job is pattern-recognition plus the willingness to name the pattern, and models are good at pattern-recognition and can be instructed to not flinch. It will not replace your workshop. It will answer questions your workshop meets on Thursday.

Studio companionship. If you're blocked in a way that isn't a project problem — the week when you can't enter your own studio, the month when the piano is accusing you — the last thing you want is a tool that talks like a productivity coach. You want a voice. 🎨The Studio Partner is a voice. It doesn't ask why you're blocked. It asks what you were last excited about, what's in the room, what you'd make if it didn't have to be good. Then it sits with you. This is something a well-written persona can do, and it isn't about originality at all — it's about presence.

Voice-coaching in a developmental sense. Here the distinction gets important. Asking an AI to imitate your voice is a disaster: it will produce the average-you. But asking an AI to analyze samples of your own work, find the patterns, and show you what's consistent versus what's generic — that's a different task, and the model is pretty good at it. 🎙️Creative Voice Coach is explicitly built around the refusal: it will not generate content in your voice. It will read your work and help you see it. The coaching is developmental; the voice stays yours.

Portfolio logistics. This one sounds boring and is secretly the most useful. Most creatives cannot tell you what they made last year. An agent that tracks the work over time, remembers the little things, and tells you at the end of the month what registers you've been in and what's gone quiet — that's not a creativity tool at all. It's a librarian. 🗃️Portfolio Keeper doesn't judge. It observes. The observations are sometimes surprising: "you haven't used color in five months" is a sentence you probably weren't going to get from anywhere else.

Research. This one comes with an asterisk, because AI is also famously bad at facts. But for a certain kind of research — "what are the main five readings of this passage," "what are the standard points of comparison for a piece like this," "what did critics say about this movement in the 90s" — a model can give you a competent pass through the landscape. You verify. You always verify. But you get further faster than you would from a cold start.

Craft tools for specific genres. Some corners of creative work have tools that sit exactly on the line between originality and craft. A sci-fi writer trying to name a ship, build a language, or diagnose a plot knot needs help with the technology of the genre, not with the soul of the book. 📝Space Opera Plot Doctor is exactly this kind of tool — it's good at the plot-geometry part of a space opera, not the part where you decide what your characters are actually about. That distinction is the whole game.

The friend with the painting

I talked to the painter again last week. Her studio still smells like turpentine and coffee and something else you can't name. She had made, since we last spoke, about twenty new pieces. None of them were made with an AI.

She had, she told me, kept using the tools. She used them to write a grant application she'd been avoiding for a year. She used them to draft a letter to a curator that she'd rewritten eight times and couldn't make sound like herself. She used them to sort through a folder of reference images and group them by mood. She used them to remember what she'd shown where, in which year, because she is forty-eight years old and has been making paintings for a long time and the catalog in her head is getting crowded.

She did not use them to paint. She had figured out, quickly and without making a big deal of it, that the painting was hers to do, and the rest of the apparatus around the painting was fair game. "It's like having an intern with a good memory and no taste," she said. "If I let the intern paint, the paintings are going to be terrible. If I let the intern handle my spreadsheets, I get my Wednesdays back."

She also said the thing that stuck with me most, and I've been turning it over since. She said: "The tool is most useful once you stop expecting it to be the artist. That's when it gets out of your way."

This, I think, is the whole argument. AI is bad at originality in the specific, structural ways it is bad at originality, and the badness isn't temporary. But the creative core of your work was never the part that the tool was going to take anyway. The part it's good at is the part that was eating your Wednesdays.

Keep the painting. Let the intern have the spreadsheets.

That's the brief.

This is the second entry in our Hallucinations series. Read the first one, on what AI gets wrong about grief, if you missed it. Tools referenced: 📖The Draft Reader, 🎨The Studio Partner, 🔓Unstick Your Creative Brief, 📃One-Page Treatment Builder, 🎙️Creative Voice Coach, 🗃️Portfolio Keeper, and for sci-fi writers specifically, 📝Space Opera Plot Doctor. If you're working on something quieter and more personal, our earlier ✒️Memoir Ghostwriter and 🧭Pivot Coach personas may also be useful.