When German journalist Martin Bernklautyped his name and location into Microsoft’s Copilot to see how his articles would be picked up by the chatbot, the answers horrified him. Copilot’s results asserted that Bernklau was an escapee from a psychiatric institution, a convicted child abuser, and a conman preying on widowers. For years, Bernklau had served as a courts reporter and the AI chatbot had falsely blamed him for the crimes whose trials he had covered.

The accusations against Bernklau weren’t true, of course, and are examples of generative AI’s “hallucinations.” These are inaccurate or nonsensical responses to a prompt provided by the user, and they’re alarmingly common. Anyone attempting to use AI should always proceed with great caution, because information from such systems needs validation and verification by humans before it can be trusted.

But why did Copilot hallucinate these terrible and false accusations?

  • Ilovethebomb@lemm.ee
    link
    fedilink
    English
    arrow-up
    44
    ·
    2 months ago

    I’d love to see more AI providers getting sued for the blatantly wrong information their models spit out.

    • catloaf@lemm.ee
      link
      fedilink
      English
      arrow-up
      13
      ·
      2 months ago

      I don’t think they should be liable for what their text generator generates. I think people should stop treating it like gospel. At most, they should be liable for misrepresenting what it can do.

      • RvTV95XBeo@sh.itjust.works
        link
        fedilink
        English
        arrow-up
        53
        ·
        2 months ago

        If these companies are marketing their AI as being able to provide “answers” to your questions they should be liable for any libel they produce.

        If they market it as “come have our letter generator give you statistically associated collections of letters to your prompt” then I guess they’re in the clear.

      • TheFriar@lemm.ee
        link
        fedilink
        English
        arrow-up
        30
        ·
        2 months ago

        So you don’t think these massive megacompanies should be held responsible for making disinformation machines? Why not?

        • futatorius@lemm.ee
          link
          fedilink
          English
          arrow-up
          1
          ·
          1 month ago

          Yeah, all these systems do is worsen the already bad signal/noise ratio in online discourse.

          • medgremlin@midwest.social
            link
            fedilink
            English
            arrow-up
            6
            ·
            2 months ago

            Which is why, in many cases, there should be liability assigned. If a self-driving car kills someone, the programming of the car is at least partially to blame, and the company that made it should be liable for the wrongful death suit, and probably for criminal charges as well. Citizens United already determined that corporations are people…now we just need to put a corporation in prison for their crimes.

            • futatorius@lemm.ee
              link
              fedilink
              English
              arrow-up
              1
              ·
              edit-2
              1 month ago

              If a self-driving car kills someone, the programming of the car is at least partially to blame

              No, it is not. It is the use to which the system has been put that is the point at which blame can be assigned. That is what should be verified and validated. That’s where some person is signing on the dotted line that the system is fit for use for that particular purpose.

              I can write a simplistic algorithm to guide a toy drone autonomously. So let’s say I GPL it. If an airplane manufacturer then drops that code into an airliner, and fail to test it correctly in scenarios resembling real-life use of that plane, they’re the ones who fucked up, not me.

          • futatorius@lemm.ee
            link
            fedilink
            English
            arrow-up
            1
            ·
            1 month ago

            No liability should apply while coding. When that code is deployed for use, there should be liability if it is unfit for its intended use. If your AI falsely denies my insurance claim, your ass should be on the line.

      • Ilovethebomb@lemm.ee
        link
        fedilink
        English
        arrow-up
        20
        ·
        2 months ago

        I want them to have more warnings and disclaimers than a pack of cigarettes. Make sure the users are very much aware they can’t trust anything it says.

      • Stopthatgirl7@lemmy.worldOP
        link
        fedilink
        English
        arrow-up
        16
        ·
        2 months ago

        If they aren’t liable for what their product does, who is? And do you think they’ll be incentivized to fix their glorified chat boxes if they know they won’t be held responsible for if?

        • lunarul@lemmy.world
          link
          fedilink
          English
          arrow-up
          5
          ·
          2 months ago

          Their product doesn’t claim to be a source of facts. It’s a generator of human-sounding text. It’s great for that purpose and they’re not liable for people misusing it or not understanding what it does.

          • Stopthatgirl7@lemmy.worldOP
            link
            fedilink
            English
            arrow-up
            14
            ·
            edit-2
            2 months ago

            So you think these companies should have no liability for the misinformation they spit out. Awesome. That’s gonna end well. Welcome to digital snake oil, y’all.

            • lunarul@lemmy.world
              link
              fedilink
              English
              arrow-up
              6
              ·
              2 months ago

              I did not say companies should have no liability for publishing misinformation. Of course if someone uses AI to generate misinformation and tries to pass it off as factual information they should be held accountable. But it doesn’t seem like anyone did that in this case. Just a journalist putting his name in the AI to see what it generates. Nobody actually spread those results as fact.

        • futatorius@lemm.ee
          link
          fedilink
          English
          arrow-up
          1
          ·
          1 month ago

          If they aren’t liable for what their product does, who is?

          The users who claim it’s fit for the purpose they are using it for. Now if the manufacturers themselves are making dodgy claims, that should stick to them too.

      • kibiz0r@midwest.social
        link
        fedilink
        English
        arrow-up
        8
        ·
        2 months ago

        If we’ve learned any lesson from the internet, it’s that once something exists it never goes away.

        Sure, people shouldn’t believe the output of their prompt. But if you’re generating that output, a site can use the API to generate a similar output for a similar request. A bot can generate it and post it to social media.

        Yeah, don’t trust the first source you see. But if the search results are slowly being colonized by AI slop, it gets to a point where the signal-to-noise ratio is so poor it stops making sense to only blame the poor discernment of those trying to find the signal.

      • futatorius@lemm.ee
        link
        fedilink
        English
        arrow-up
        1
        ·
        1 month ago

        Unless there is a huge disclaimer before every interaction saying “THIS SYSTEM OUTPUTS BOLLOCKS!” then it’s not good enough. And any commercial enterprise that represents any AI-generated customer interaction as factual or correct should be held legally accountable for making that claim.

        There are probably already cases where AI is being used for life-and-limb decisions, probably with a do-nothing human rubber stamp in the loop to give plausible deniability. People will be maimed and killed by these decisions.