Generative AI Lies

Examples of generative AI making stuff up

Posts

  • Unauthorized biographies

    (, )

    In the New York Times, journalist Kashmir Hill writes that she recently discovered that there was an AI-generated biography of her available for sale.

    A biography of Kashmir Hill — title: “The Biography of Kashmir Hill” — [was released] in August 2025. My life story had a mottled brown cover and a publisher I’d never heard of before.

    […]

    My biography is 90 pages long and should be shorter. It combines facts about me that are widely available on the internet, such as where I grew up, with generic insights that could be true of anyone, like a horoscope spread over dozens of pages. “You cannot understand Kashmir Hill without understanding her contradictions,” my biographer wrote, along with an excruciatingly long description of my elaborate coffee-making ritual. (Fact check: My husband does it.)

    […]

    I clicked on the author, one John Crane Miller. His bio page described him as a “seasoned biographer and cultural analyst,” and his portrait was a widely used stock photo of a white man in a suit speaking at a conference.

    […]

    “The Biography of Kashmir Hill[…]” was one of 10 biographies that Mr. Miller had published in a single week

    […]

    [Miller] claimed to have read my childhood diary and spoken with people who know me well “and those who wish they didn’t.”

    Hill goes on to interview a different “author” of AI-generated books, Bill Johns, who has published 445 books on Amazon (he sells “a few hundred books per month, each earning him roughly $7”), and to discuss some other general topics around AI-generated books. But the above excerpts are the most relevant bits for my purposes.

  • Google AI lies about Scalzi

    (, , )

    John Scalzi writes:

    Every assertion in this “AI Overview” of the question “What coffee does John Scalzi drink” is wrong. I don’t regularly drink coffee (and never black) I’ve never had black sesame jasmine cream tea, and I don’t hang in coffee shops. Don’t trust “AI” ever!

    The post was accompanied by a screensnap from Google’s AI Overview, answering the question “What coffee does John Scalzi drink”. As usual, the AI Overview provided links that supposedly linked to sources for the information that it was giving, but the linked-to pages don’t say what Google says they say.

    After Scalzi’s posted the above to Bluesky, Google’s answer changed to link to his post. But just because AI Overview might eventually be updated to give correct information, that doesn’t mean we should trust it.

  • AI models answer questions about medical images they haven’t seen

    (, )

     “AI models happily came up with answers to questions about a supposedly accompanying image — even if the researchers never even showed it an image.”

    The article quotes the researchers as talking about this in terms of there being “information […] hidden in a sentence or a question” that allows the AIs to correctly answer the questions even without access to the images that the questions are about, but that seems weirdly implausible to me. Later in the article, there are implications of what seems to me far more likely: that the AI was trained on these questions and answers, and thus can provide the answers without having to have access to the images that accompany the text.

    More from the researchers:

    “Another implication is that, now that we know an AI can say ‘I see evidence of malignant melanoma on your skin’ without even having access to any images, how much can we trust it when it says the same while actually seeing the image?” Asadi posited. “We definitely need more effort being put in safety and alignment of such models, and might need to think twice before deploying them in user/patient-facing systems.”

    […] “The number one [takeaway] would be that just because the AI is saying, very convincingly, that it is seeing something, it doesn’t mean that it is actually seeing that.”

  • Canadian Immigration Department uses lying generative AI

    ()

    “Canada rejected her permanent residence application. Her job duties were made up — by Immigration’s AI reviewer”

    Postdoc Kémy Adé applied for permanent residence in Canada, but was rejected because a generative-AI tool hallucinated a set of job duties that she didn’t have, and the Immigration Department therefore ruled that the work she had done (that the AI made up) didn’t match the work she claimed to have done (that she really did).

  • Police officer turns into frog

    ()

    You know how organizations and doctors and therapists and lots of other people want to use generative-AI tools to write up summaries of meetings?

    Now there are a couple of tools that police can use to create reports summarizing body-camera audio and/or video.

    And some police love it:

    “Most of our officers are kind of awestruck with (Draft One) because it’s such a new, innovative thing for us,” Weishar said. “It’s like that brand new car that’s got all the features to it. For us, it’s crazy that you can just press a button and it’ll tell you everything about the case that you were on and give you a pretty decent police report to edit.”

    But (gasp! shock!) it has certain pitfalls:

    “I read the report, and I’m like, ‘Man, this really looks like an officer wrote it,’” Sever recalled. “But when it got to one part, it said, ‘And then the officer turned into a frog, and a magic book appeared and began granting wishes.’ … It was because they had, like, ‘Harry Potter’ on in the background. So it picked up the noise from the TV and added it to the report.”

    The second half of the article has some reasonably good discussion of some of the reasons police shouldn’t be using these tools. But it won’t surprise me if some police departments start doing it anyway. This kind of software is a great time-saver, as long as you don’t mind when the resulting reports describe officers turning into frogs.

  • Law firm dissolved

    ()

    In Mississippi, a lawyer included AI-generated fake citations and was told not to do that, but kept doing it.

    The latest judge to receive her fake citations was not amused, and has issued sanctions against her and the two partners in the firm that she worked for.

    If I’m understanding right, the partners have now dissolved the firm.

    (It looks like Ms. Watson, the lawyer who used AI-generated fake citations in ten different cases, may be the daughter of one of the two partners.)

    The judge reacted strongly to Ms. Watson’s behavior:

    In light of repeated warnings from federal courts about the risk of hallucinated cases, as well as CLE trainings she attended, direct notice and knowledge of the same prior mistakes, her violation of the Firm’s AI policy, and the sheer number of filings, Ms. Watson’s misconduct is particularly egregious and prolific.

    The partners are also being sanctioned for failing to notice the problems. For example:

    a large portion of Billups’ argument relies on a case styled Jackson v. Gautreaux, 3 F. 4th 182, 190 (5th Cir. 2021). […] In fact, this case is cited eight times, even arguing that a jury should be instructed under its holding. […] In reality, Jackson is an excessive force and failure to train case and is wholly irrelevant to the case at bar. A seasoned attorney examining the brief should have read a case so heavily relied upon. Had he done so, he would have easily discovered the problems.

    The judge noted that the usual penalty for this sort of thing has been fines, but quoted another case about why fines are insufficient:

    “If fines and public embarrassment were effective deterrents, there would not be so many [AI misuse] cases to cite.”

    (Given that there are so many such cases, I probably won’t post about all the ones I hear about, but this one did seem especially egregious.)

  • More hallucitations

    ()

    “AI Is Inventing Academic Papers That Don’t Exist — and They’re Being Cited in Real Journals”

    Rolling Stone says:

    [Academic] articles which include references to nonexistent research material […] are themselves being cited in other papers, which effectively launders their erroneous citations. This leads to students and academics (and any large language models they may ask for help) identifying those “sources” as reliable without ever confirming their veracity. The more these false citations are unquestioningly repeated from one article to the next, the more the illusion of their authenticity is reinforced.

  • Fake blobfish

    ()

    “Deep Sea Social Media is Flooded by AI Slop”

    I focus more on text than on images on this site, but when AI-generated images inaccurately portray what real-world creatures look like, I figure that more or less fits my theme.

  • Made-up journals

    ()

    Scientific American says:

    OpenAI’s ChatGPT, Google’s Gemini, Microsoft’s Copilot and other models are befuddling students, researchers and archivists by generating “incorrect or fabricated archival references,”

    Which is a problem for librarians:

    who end up wasting their time looking for requested nonexistent records, says Library of Virginia chief of researcher engagement Sarah Falls. Her library estimates that 15 percent of emailed reference questions it receives are now ChatGPT-generated, and some include hallucinated citations for both published works and unique primary source documents. “For our staff, it is much harder to prove that a unique record doesn’t exist,” she says.

    I kinda want to call these things “hallucitations.”

  • Chuck Wendig’s cat

    (, )

    Chuck Wendig discovers that Google AI Overview says he has a cat named Boomba. Also other cats. Also six dogs. Also two children. And a spider. Most of those pets and one of those humans don’t exist in real life, but if Google AI Overview says they do, who are we mere mortals to question it?

    Content warning for the imaginary deaths of imaginary cats. Also for an imaginary cancer diagnosis.