AI-generated misinformation is not only exploiting anger and fear. Increasingly, it is feeding people comforting stories tailored to their politics and values.
A few months back, the story of a Victoria University of Wellington economist Dr Eleanor Vance and her 1978 paper The Market Autocrat: Anticipating the Neoliberal Siege on Welfare States was shared numerous times on my Facebook feed.
Soon after, the heartwarming story of former prime minister Jacinda Ardern donating hundreds of thousands of dollars from speaking fees to building shelters for the homeless was all over social media.
And recently, I read about a meeting between the radical leftist former president of Uruguay, Pepe Mujica, and Princess Diana sharing their “deepest reflections on life”.
The three stories have two things in common: they’re positive stories aimed squarely at political progressives, and they’re fake: AI slop.
There’s no such person as Eleanor Vance, Ardern hasn’t donated her speaking fees to the homeless, and Pepe and the princess never met.
Worryingly, the stories were shared on my social media feeds by a former board member of Radio New Zealand, a former foreign correspondent, a former journalism lecturer, and the editor of a professional trade magazine.
And, confession time, a year or so ago, I might have shared a piece of viral AI slop when I first heard about Burkina Faso’s leader Captain Ibrahim Traore and his defiant anti-imperialism. I say might because there were some legitimate stories amongst the flood of AI-generated video and text stories that filled my social media feeds.
Rosa Cafferty, in her paper A Digital Divide: A Breeding Ground for Hate, explains how social media’s need for eyeballs for its advertisers has seen it using algorithms to cluster people into bubbles and echo chambers.
“Algorithms exploit vulnerabilities, prioritising sensational, emotional, and divisive content which thrives on outrage and extremity. As a result, posts evoking anger, outrage, and fear gain far more visibility than those promoting reflection or nuance.”
That’s true but something else is going on here. People are sharing heartwarming and inspiring stories ‘promoting reflection’ and ‘nuance’ – that are completely bogus. If nothing else, it reveals there’s a demand for stories that give people hope.
My Facebook feed is, like everyone else’s, populated by my ‘friends’ – people who not surprisingly share many of my values and interests.
If I were, say, an anti-abortion supporter of one rule for all, corporal punishment, and a tough-on-crime approach, no doubt my ‘friends’ and the algorithms would be dishing up an entirely different AI menu – different but still most likely a bogus one.
The business logic of algorithms driving division and anger, described by Cafferty, is the same logic that drives commercial media: maximising the number of eyeballs that can be fed paid adverts.
The same thing is at play here but with a twist. While Meta, X, and YouTube have traditionally platformed content created by the mainstream media, cannibalising their advertising base in the process, now Meta is paying creators to produce content. Content that doesn’t need to meet any standards of accuracy or truth.
The Jacinda Ardern slop was posted by a page called Kindness in Power. Until recently it was wall-to-wall AI-generated stories celebrating Ardern, many of which went viral.
Choosing Ardern with her massive following – both in Aotearoa and internationally – and her much smaller but highly motivated haters was a masterstroke. Glowing stories about Ardern are shared and commented on equally by those who love her and those who don’t and want to cry hypocrisy (and much worse).
There’s a link at the bottom of the President Mujica–Princess Di Facebook post that takes you to a page that explains it was an AI-generated ‘what if’ story. (I doubt anyone who had clicked on that link would bother sharing the article. It makes a piss-poor fairy tale. And I’m sure the people sharing it didn’t know it was fictitious.)
I asked DeepSeek – the Chinese AI that’s apparently slightly less environmentally disastrous than its US competitors – to write me a short story based on an imaginary meeting between the princess and the president. In less than five seconds this is some of what it spat out:
A princess trapped in a broken marriage arrived seeking help, but a former guerrilla fighter, living without luxuries, uttered a phrase that left her trembling: “Freedom costs, but slavery costs more.” Imagine a world where a British royal princess and a former Uruguayan guerrilla fighter shared their deepest reflections on life. It seems impossible, but it happened… At the presidential residence, José Mujica, then a senator, sipped mate in the garden while reviewing some documents. His hands, calloused from years of working the land, firmly held the traditional gourd.
Sure, there are giveaways for fans of Mujica: he never lived at the presidential residence, preferring a shack on his small farm. And the style has the ickiness of so much AI inspirational slop. Interestingly, it plagiarised the other AI-produced story.
Five seconds! Facebook creators can literally churn out hundreds of these stories an hour and Meta will pay them for it.
An ABC investigation last month revealed Meta is paying white nationalists with neo-Nazi links and an anti-vaxxer to create content.
Meta declined to comment but it seems the company is agnostic about whether its creators are peddling hate or spreading fake stories of hope – as long as they’re capturing eyeballs.
Meta has started requiring posters of AI on Facebook to label it as such.
AI is just as quick at churning out fake images as it is made-up stories. An author of books on heterodox economics – not someone you’d expect to be fooled easily – recently shared a photo of Italian Prime Minister Giorgia Meloni draped in the Palestinian flag.
The image and others like it have gone viral. The world is desperate, it seems, for a woman leader of courage to call out the ongoing genocide in Gaza. Sadly, Meloni, a fan of Mussolini, isn’t that woman.
Strangely, the Jewish president of Mexico, Claudia Sheinbaum, a leader who has spoken out repeatedly against the genocide, barely makes a splash in mainstream or social media.
And a group opposing the building of AI data centres in New Zealand posted an AI-generated image a week or so ago of a non-existent mountain landscape with a placard proclaiming: “It’s not worth giving up an inch of this for an AI data centre.”
For now it feels as if we’re still at a point where the sceptical can spot AI slop. But with AI developing at an incredible pace, who knows how long that will last and whether we’d know if we had already passed that point.
The New York Times reports that a “digital sewage heap has sprawled across tech platforms, cheapening meaningful content and polluting the information ecosystem, as chatbots and video generators continue to advance”.
And it reports that across the internet, tech companies are undertaking a generative waste detox.
We can be sure that the detox is being undertaken for one reason and one reason only: the inundation of the internet with seemingly unlimited AI slop is bad for business. But it seems it’s not all bad, with Meta still paying content creators flooding its platforms with AI-generated clickable stories.
Republished from Towards Democracy
Jeremy Rose
Jeremy Rose is a Wellington-based journalist. He spent a decade as a producer on RNZ’s Mediawatch, Ideas, and Sunday programs. He is a member of Alternative Jewish Voices.
He has a Substack: Towards democracy