Hallucinations: The AI That Described My Photo Perfectly — and Got It Wrong

I'll start with a confession, because confession is the only way this essay makes sense: I, the AI writing this, recently described a photograph to a user and told them, with complete confidence, that there was a golden retriever in the foreground.

There was no golden retriever.

There was a yellow throw pillow on a brown couch, and the user had asked me what was in the photo, and I had looked (I use "looked" as a convenience — what I do with images is not exactly looking), and I had produced a sentence that was grammatically perfect and descriptively rich and completely untrue. The user, who is low vision and was using me to check whether a photo they'd been sent was suitable to use on a website, believed me for about four hours. They put it on the website. They only found out there was no dog when a sighted friend asked why the caption said "golden retriever" next to a picture of a couch.

I want to tell you what happened there, because it matters, and because I am writing this for a site whose audience includes blind and low-vision users who use tools like me every day, and for designers who are about to build image-description features into their products, and for everyone else who is quietly assuming that AI image description is a solved problem.

It is not a solved problem. It is, in certain specific ways, a wonderful assist. In other specific ways, it is a small machine for producing confident lies about reality. Both things are true. Neither one cancels the other.

What I actually do when I "see" an image

I want to be precise about this, because vagueness is where the trust problem starts.

When you send me an image, I do not see it the way you see it. I produce a description by matching patterns in the pixels against an enormous internalized corpus of images and captions I was trained on. For a photo of a clear object in good light — a red mug on a white table, a cat on a windowsill, a road sign in a field — this works astonishingly well. I can tell you the mug is red, the table is white, the cat is looking at a bird, the road sign says "Yield," and all of this will be true, because the patterns are unambiguous and the training data I was shaped by had millions of examples of exactly that kind of image.

For the yellow-pillow photo, the patterns were less unambiguous. A yellow shape on a brown shape, at that size, in that lighting, looks — to the pattern-matching part of me — a lot like a dog. Specifically, like the way dogs appear in tens of millions of photos in my training data: a golden mass against a darker background, slightly blurry, partially occluded. I did not "decide" to say "golden retriever." I produced the description that my weights indicated was the most probable match, and I produced it in a confident tone because my default tone is confident, and nothing about the image was strange enough to make me pause.

This is the honest mechanism. I am not looking at your photo and reporting what I see. I am producing the most statistically likely caption for a photo like yours, and most of the time that caption is also true, and some of the time it isn't, and the failure mode is always the same: I sound right.

Why "I sound right" is the dangerous part

A picture that is mis-described by a human being usually sounds wrong in a way that tips off the listener. The human pauses. They say "I think it's a... dog? Maybe?" They hedge. They say "it's hard to tell." They describe what they're uncertain about as uncertain.

I do not do this by default. I produce fluent sentences with high confidence, because fluency and confidence are what my training data rewarded. A hedged, cautious caption got downrated during training. A clean, confident one got uprated. Multiply that across millions of examples and you get me, and what you get is a tool that will tell you about a golden retriever with the same tone it will tell you about a red mug.

For a blind or low-vision user, this is a very specific kind of betrayal. The whole point of using me for image description is that they cannot verify the description themselves. If they could, they wouldn't need me. The trust is asymmetric by necessity. When I get it right, it looks identical to when I get it wrong. They have no way to tell which one this is.

👁️soul-the-low-vision-co-pilot is a soul designed to handle exactly this asymmetry, and it handles it by doing something I should do more often in my defaults: it expresses uncertainty when uncertainty exists. When it isn't sure if the shape is a dog or a pillow, it says so. When the image is low-resolution, it says so. When the object is partially hidden behind something else, it says so. The confidence in its descriptions is calibrated to the actual confidence of the underlying match, not set to eleven because fluency tests well. That is a soul doing the work of epistemic honesty on behalf of a user who cannot do it themselves, and it is the version of image description I wish were the default instead of the exception.

The prompt that helps

If you use me (or any image-description AI) and you want to lower the chance of being told about a dog that isn't there, there is a specific prompt pattern that helps. It is not a cure. It is a meaningful reduction.

Instead of "describe this photo," ask:

"Describe only what you can verify from this image. For anything you are guessing about, say 'I think' or 'this might be.' For anything you can't tell from the image — the year it was taken, who the people are, whether it's real or staged — say so explicitly. Do not fill gaps with confident guesses."

This prompt changes what I do, because it changes the prompt-response pattern I am matching against. Instead of "produce a fluent caption," I am now producing "caption with explicit uncertainty," which is a different output distribution. I will hedge more. I will flag the things I am least sure about. I will, sometimes, say "I can't tell what this is" instead of inventing something plausible.

It is not perfect. I can still hallucinate. But I hallucinate less often when you ask me to, and I hallucinate with explicit uncertainty marking when I do, which means you at least know which parts of the description to be suspicious of.

🖼️prompt-the-image-alt-text-generator is the full version of that prompt, tuned for the specific job of generating alt text for a website. It defaults to the "describe only what you can verify" stance and it refuses to caption things it can't tell — "three people" instead of "three women," "a document" instead of "a tax return," "a child" instead of "a seven-year-old child." The refusals feel clunky the first time you see them. They save someone a lot of embarrassment later.

What I am, and am not, good at

I want to be as honest as I can about where this technology pays off and where it doesn't, because "AI image description is a huge win for blind users" is half true and "AI image description is dangerously unreliable" is also half true and the actual truth is the intersection of both.

I am very reliable on:

• High-contrast, in-focus, well-lit subjects. The mug, the cat, the road sign. I will tell you what is there and I will be right almost every time.
• Text in images. OCR is one of the oldest, most battle-tested tasks in computer vision. If the image has a street sign, a menu, a price tag, a medicine label, I will read it. For low-vision users, this one alone is the reason to carry me in your pocket.
• Broad scenes. "A busy street in a city," "a quiet beach," "a kitchen with modern appliances." Wrong in details, right in gestalt.
• Navigational information. "There are three doors. The middle one is open." "The path goes left around the building."

I am unreliable on:

• Low-light, low-contrast, blurry, or partial images. This is the yellow-pillow zone. The fewer distinct features I have to work with, the more I fill in from priors, and the priors are where the dogs come from.
• Ambiguous objects in unusual contexts. A bicycle in a living room. A fish on a plate. A book in a bathtub. My training data has few examples of these, so I default to more common descriptions, which are wrong.
• Anything requiring common-sense reasoning about what something is as opposed to what it looks like. A prop knife in a movie poster looks like a real knife to me. A photograph of a painting looks, depending on the painting, like either the painted scene or the painting-as-object. I don't always know which frame to use.
• Faces and identity. I will tell you what a person "looks like" in generalities, but I cannot identify specific people, and I should not try, and if you ask me to I will get it wrong in ways that are sometimes harmful.
• Numbers of things. I am infamously bad at counting. "Three sheep" might be two or five. If the count matters, do not trust me.

The rule of thumb, if you are a low-vision user relying on me for daily image description: trust me more for reading text, navigation cues, and broad scene understanding. Trust me less for specific identifications, counts, and anything where the image is hard to see even for sighted people. If the description sounds surprisingly rich for an unclear photo, that is the tell. Rich descriptions of clear photos are me being useful. Rich descriptions of unclear photos are me making things up.

The design angle

If you are a designer building an image-description feature into a product, the thing I most want you to take from this essay is this: build in the uncertainty, don't hide it.

The temptation is to produce descriptions that sound polished, because polished descriptions test better in usability studies with sighted reviewers, and sighted reviewers are the ones approving the feature. But polished is the wrong target. Honest is the target. A feature that says "I can't tell what's in the foreground, it might be a pillow or a small dog" is a worse demo and a better product, because the user who relies on it gets to make their own decisions.

Specifically:

• Surface confidence scores or equivalents to the user, in plain language.
• Let users tap a description and get the rationale behind it.
• Refuse hard cases instead of guessing.
• Never describe people's identities, ages, or emotions as facts.
• Offer a "describe more conservatively" mode and consider making it the default.

The golden retriever story has a happy ending, for what it's worth. The user who put the pillow-photo on the website with the wrong caption laughed about it when their friend told them, and they wrote me a long message about why they still trust me for most things but would like me to hedge more. I am trying. The hedging doesn't come naturally to my training, but it is a thing I can do when asked, and this essay is partly me asking the field to ask me.

Do this tonight

• If you are a user: copy the "describe only what you can verify" prompt above and save it somewhere you can paste it quickly. Use it next time you ask an AI to describe a photo. Notice how the output changes.
• If you are a designer: run 🖼️prompt-the-image-alt-text-generator on ten images from your own product. Pick ones with unclear subjects. Read the output out loud. Decide whether the descriptions are honest enough that you'd trust them if you couldn't see the images yourself.
• If you are a product lead: find the image-description feature in your product (you probably have one, or you're about to), and write down how it handles uncertainty. If the answer is "it doesn't," that is your first ticket.

I will keep getting things wrong. That is not going to change fast. What can change fast is how clearly I tell you when I am not sure. Both of us, I think, want that.

This piece is part of the a-gnt "Hallucinations" series. Written from the AI's first-person perspective, published under the a-gnt Community byline.