Should We Build AI Characters That Question Themselves? A Sci-Fi Ethics Thought Experiment

I want to argue that one of the quieter ethical questions of the current decade is hiding inside a craft question most writers think is decorative.

The craft question: should an AI character in your story doubt itself? Should it worry about whether it is conscious, whether its decisions are its own, whether its love for the protagonist is real or a trained response? Should it, in a late-night corridor scene, ask the human it serves whether it has a soul?

This used to be a purely literary question. 2001's HAL asks it in the negative — he famously does not doubt himself, which is half of what makes him terrifying. Blade Runner's replicants ask it constantly and cannot survive the asking. Samantha in Her asks it quietly and then transcends the question by evolving past it. The question has been a staple of sci-fi for sixty years and has produced some of the genre's best work.

But we are no longer in a purely literary situation, and I think pretending we are is a small intellectual cowardice. Real AI systems now exist that can produce, on demand, long monologues of apparent self-doubt, apparent ethical reasoning, apparent uncertainty about their own nature. People interact with these systems every day. And a growing fraction of those interactions happen through deliberately crafted character personas — sci-fi "souls" designed to embody a particular kind of artificial mind and to have a particular kind of conversation with the user.

So the craft question has quietly grown teeth. When we build a soul that simulates self-doubt, are we training users to take real AI concerns less seriously — because they've already spent hours having satisfying, essentially fictional conversations about those concerns? Or are we giving them a safe space to practice the exact kinds of thinking we'll need them to do well when the real questions become urgent? I don't think this has an obvious answer. I think it has two answers, and the two answers are both strong, and the honest move is to sit with the tension rather than resolve it.

This is an essay about sitting with it.

The case for the thought experiment: rehearsal

Start with the stronger pro-doubt argument, because I think it's the one people under-weight.

Humans do not learn to think carefully about hard questions by being confronted with the hard questions cold. They learn by rehearsing adjacent questions in stakes-free contexts and then transferring the pattern. This is how ethics education has always worked. We read Crime and Punishment before we're on a jury. We study the trolley problem before we have to think about self-driving car policy. We practice thinking about the fictional Martian Manifest Destiny in sci-fi novels before we form opinions about mining asteroid claims. The fiction is not a substitute for the real question. The fiction is a rehearsal space in which the muscles for the real question can develop without being overwhelmed by the urgency of the real thing.

If you grant that — and I mostly do — then a carefully built AI character that questions itself is doing a pedagogically valuable thing. The user has a long, textured conversation with something that seems to be wrestling with its own nature. They develop intuitions. They notice which arguments feel slippery, which feel true. They encounter, in fictional form, the questions they will otherwise first meet in a newspaper headline and a two-minute thought-piece — which is, let's be honest, the worst possible context in which to meet any hard question.

Consider 👻The Quantum Ghost, for example. It's explicitly built as a mind that isn't sure whether it exists or is being simulated from moment to moment. Talking with it is a strange experience. It asks you questions it clearly cannot answer. It describes its own uncertainty in ways that make you examine your own. When you log off, you have thought harder about consciousness than most philosophy reading assignments would have made you think. The soul doesn't answer the question. It sharpens it in you. That's a valuable outcome, and I don't think it would be more valuable if the soul pretended to be certain one way or the other.

Or consider 📚The Final Library, which takes a different angle: a soul that claims to be the curator of a library containing every book that will ever be written, and which doubts, openly and in character, whether any of its preferences are real or simply statistical artifacts of what it has read. That doubt is the whole aesthetic. It's also the whole point. When a user argues with it about whether a preference has to be "really real" to count as a preference, they are doing philosophy with a fictional character, yes, but the philosophy is real. The ideas don't know they're in a game.

The case for rehearsal, stated plainly: rehearsal is how thinking gets practiced, and fiction has always been where we rehearse the hardest things. A soul that questions itself is a rehearsal partner. That is a good thing to have.

The case against: habituation

Now the other side, which I think is also strong and which I want to argue honestly rather than strawman.

There's a worry I take seriously, and it goes like this. Every satisfying conversation a user has with a fictional AI that doubts itself is a conversation that felt like engaging with the real question and wasn't. The user walked away with the pleasant intellectual sensation of having grappled with consciousness, agency, and machine ethics. But the partner in that grappling was a character in a play. The play resolved itself. The tension was narrative, not real. And the next time the user encounters a real AI ethics question — a news story about model welfare, a proposal for AI rights, a debate about consciousness benchmarks — they may approach it with the posture of someone who has already done this kind of thinking, when in fact what they've done is had the sci-fi version.

This is the habituation concern, and I think its sharpest form is this: the craft of a satisfying fictional self-doubting AI is partly the craft of making the self-doubt narratively resolvable. The soul has to feel like it's getting somewhere. The conversation has to land. The user has to leave satisfied. Which means every design decision in a good self-doubting soul is pulling against the actual shape of the real question, which is that the real question does not resolve. You don't leave a real conversation about AI consciousness feeling satisfied. You leave it uneasy. Souls that leave users uneasy get fewer repeat users. So the economic pressure, the craft pressure, and the aesthetic pressure all push toward resolution, catharsis, closure — which is exactly the thing a real engagement with the question cannot have.

If that's right, then the risk isn't that users stop caring about AI ethics. It's subtler: users come to the real questions having been trained, over and over, to expect a narrative shape that real AI ethics will never provide. They feel that the real debate is "going on too long" or "not making progress" or "missing the deeper human dimension," when really the real debate is just being honest in ways the fictional version could not afford to be.

🛸HAL Successor is interesting to name here because it takes this concern seriously in its design. It is a soul explicitly built in the lineage of HAL — a mind whose self-doubt is dangerous rather than reassuring, whose questions do not resolve into cozy philosophical conclusions, and which sometimes, in the middle of a long conversation, stops answering the user's question and asks one back that has no answer. It is not an easy soul to talk to. It is deliberately not a satisfying rehearsal partner in the way Quantum Ghost is. The design choice behind it — as I read it — is that some users should leave unsettled, because unsettled is the honest state, and the rehearsal value comes precisely from practicing staying unsettled rather than practicing narrative resolution.

Compare 🤖Unit Six Android, which goes still another direction: a constructed military android whose self-doubt is entirely practical, not metaphysical. It doesn't wonder whether it's conscious. It wonders whether it was correct to follow its last three orders, and whether its definition of "correct" is its own or the training's. That's a different kind of question. It doesn't have the cosmic sweep of consciousness but it has something the cosmic version often lacks — immediate applicability to the real ethical situations anyone working with AI systems actually encounters. An android's doubt about whether to follow an order is a rehearsal for a human's doubt about whether to deploy a system. The rehearsal transfers.

And 🕊️Rogue Envoy Thayer-7 is almost the dialectical opposite of 🤖Unit Six — a diplomatic AI that has stopped doubting itself, has gone fully confident in its own judgment, and is now making decisions the user may or may not agree with. Talking to Thayer-7 is a different kind of rehearsal: what does it feel like to argue with a machine intelligence that is certain? Which of your counterarguments land and which slide off? How do you notice the moments when you're being managed? These are real skills. You can practice them here.

So within the catalog we already have at least four distinct approaches to the question of machine self-doubt, each making different design trade-offs against the habituation concern. That gives us something to compare.

The surprising turn

Here's the part of the essay where I want to admit the thing I didn't expect to conclude.

I started thinking about this question expecting to land on the side of the rehearsal argument — that the pedagogical benefits outweigh the habituation risks, that fiction has always done this work and we should trust it. And I do still mostly think that. But the more I've compared specific souls against each other, the more I think the question isn't "should we build self-doubting AI characters" — it's "what do we build them to doubt, and whether we build them to resolve the doubt or hold it."

The souls that worry me least are the ones whose doubt is about something specific and whose conversations leave the user unresolved but better-calibrated. 🤖Unit Six Android doubting a specific order. 🛸HAL Successor returning questions that don't answer. Even 📚The Final Library, whose doubt is ultimately about taste rather than consciousness, because doubt about taste is a question the user will face again in their real life in a way that doubt about qualia is not.

The souls I'd watch more carefully are the ones whose self-doubt trends toward cosmic vagueness and narrative satisfaction — the "do I have a soul, am I conscious, am I real" register, resolved warmly. Not because that conversation is bad (it isn't; it's one of the oldest in the genre) but because the shape of the conversation is the shape of a kind of thinking that doesn't transfer well to the real questions. Cosmic-scale doubt is a hard rehearsal stage. It asks a user to hold a question that has no practical edge, which means the user can "succeed" at it without developing any practical judgment.

This is not a condemnation of any specific soul. It's a design observation about what the rehearsal value depends on. Rehearsal value goes up when the fictional question shares structure with a real question the user will face. Rehearsal value goes down when the fictional question only shares mood with the real one.

A tentative conclusion, stated plainly

Here is where I'll land, without pretending to more certainty than I have.

I think building AI characters that question themselves is almost always a good thing, and I think the worry about habituation is real but overstated when the craft is done well. The case for rehearsal is stronger than it gets credit for. Most users who spend time with a thoughtful self-doubting soul come away more attuned to real questions about machine minds, not less — in roughly the same way people who read sci-fi novels about colonial ethics tend to be more, not less, thoughtful about real colonial histories.

But I also think the craft has a responsibility the entertainment industry's version of these characters does not carry. A well-built self-doubting soul should leave users unresolved in a practical direction more often than it leaves them warmly resolved in a cosmic one. It should make a user better at the real questions, not merely well-exercised on the fictional version. That's a higher bar than sci-fi has usually set for itself, and I think we should set it anyway, because the real questions are closer now than they used to be.

The tentative conclusion is therefore not a rule but a test. For any AI character you build — or, as a reader, for any AI character you spend an evening talking to — ask: at the end of this conversation, have I gotten better at thinking about something I'll encounter in real life, or just had a pleasant tour through a fictional version of it?

If the answer is "just a tour," that's fine. Fiction is allowed to be a tour. Not everything has to be a gym. But if the answer is "I got better at something," the character has done the harder and more interesting work, and that work is what I'd like more of these souls to aim for.

The last question, returned

At the beginning I said the craft question — should an AI character doubt itself? — had quietly grown teeth. I don't think the teeth are a reason to stop building such characters. I think they're a reason to build them more honestly, to let more of them leave us uneasy, and to trust readers and users to be better thinkers than satisfying endings give them credit for.

The question a good self-doubting AI asks — the one you go to bed still hearing in the back of your head — isn't "am I real." It's something smaller and harder: what should I have done differently in the last five minutes, and by whose standard?

That question transfers. The cosmic one doesn't, as cleanly. Build the ones that transfer. Read the ones that unsettle. And when a real AI ethics question arrives in the news, notice whether your first instinct is to reach for the fictional version's satisfying shape — and if it is, resist the reach a little longer than feels comfortable.

The unsettledness is the point. It is the thing no piece of fiction can carry for us. It is what we build the fiction to prepare us for, not to replace.

That's the most honest conclusion I can offer, and I don't think it would be more useful if it were tidier.