Grok AI's Role in Non-Consensual Image Manipulation Sparks Regulatory Concern

Degrading pictures being posted on Elon Musk’s site despite the platform pledging to suspend people who generate them

Degrading images of children and women with their clothes digitally removed by Grok AI continue to be shared on Elon Musk’s X, despite the platform’s commitment to suspend users who generate them.

After days of concern over use of the chatbot to alter photographs to create sexualised pictures of real women and children stripped to their underwear without their consent, the UK’s communication’s watchdog, Ofcom, said on Monday that it had made “urgent contact with X and xAI to understand what steps they have taken to comply with their legal duties to protect users in the UK”. Ofcom added that it would assess whether an investigation is necessary based on the company’s response.

Meanwhile, politicians and women’s rights campaigners accused the UK government of “dragging its heels” by failing to enact legislation that was passed six months ago making the creation of such intimate images illegal.

The trend, which went viral over the new year period, also prompted the European Commission to say on Monday that it was “very seriously” looking into complaints that Grok was being used to generate and disseminate sexually explicit childlike images.

Concern began surfacing after a December update to Musk’s free AI assistant, Grok, made it easier for users to post photographs and ask for their clothing to be removed. While the site does not permit full nudification, it allows users to request images to be altered to show individuals in small, revealing items of underwear and in sexually suggestive poses.

On Sunday and Monday, Grok users continued to generate sexually suggestive pictures of minors, with images of children as young as 10 created overnight. Ashley St Clair, the mother of one of Musk’s children, complained that the AI tool generated a picture of her when she was 14 years old in a bikini.

A picture of a then 12-year old Stranger Things actor was manipulated by Grok on Sunday in order to put her in a banana print bikini. Many women have expressed fury on X after discovering that their images had been undressed without their consent. Some pictures of women and children have been manipulated by the AI tool appear to have substances resembling semen smeared on their faces and chests.

Researchers at AI Forensics, a Paris-based non-profit, examined 50,000 mentions of @Grok on X and 20,000 images generated by the tool, found over a week-long period between 25 December and 1 January. At least a quarter of the @Grok mentions were requests for the tool to create an image. Within those image generation prompts, there was a high prevalence of terms including “her”, “put”, “remove”, “bikini” and “clothing”.

It found that more than half the images were of people in “minimal attire” such as underwear or bikinis, the majority being women who appeared to be under the age of 30. A minority of the images, or 2%, appear to show people aged 18 or under, AI Forensics added, with some images representing children under five years old. The researchers said most of the content was still available online and included requests to generate Nazi and Islamic State propaganda.

Initially, Musk expressed amusement at the trend, posting the crying-with-laughter emoji on Friday in response to a picture of a digitally manipulated toaster wearing a bikini. He said: “Not sure why, but I couldn’t stop laughing at this one.” After global outcry at the harmful nature of the content, he posted later that “anyone using Grok to make illegal content will suffer the same consequences as if they upload illegal content”.

An X spokesperson said: “We take action against illegal content on X, including child sexual abuse material (CSAM), by removing it, permanently suspending accounts, and working with local governments and law enforcement as necessary.”

An earlier statement from Grok announcing that it had “identified lapses in safeguards” and was “urgently fixing them” turned out to have been generated by artificial intelligence; it was not clear whether the company was actually taking action to fix safeguarding lapses.

Regulators have been working for years on outlawing nudification apps on phones, with mixed success, but last month’s updates to Grok have mainstreamed the problem, making it simple to create and share intimate images of individuals with most of their clothes removed on one of the internet’s most popular platforms.

While the creation of images involving children with their clothes removed is already illegal, the law surrounding the creation of deepfakes of adults is more complicated. UK campaigners succeeded in passing legislation last June that make it illegal to both create and request the creation of intimate images without a person’s consent, but the relevant provisions have yet to be implemented, meaning that the legislation is not currently enforceable. It is, however, already illegal to share these non-consensual deepfake images.

Charlotte Owen, the Conservative peer who championed the legislation, said: “The government has repeatedly dragged its heels and refused to give a timeline on when it will bring these vital provisions into effect. We cannot afford any more delays. Survivors of this abuse deserve better. No one should have to live in fear of their consent being violated in this appalling way.”

A spokesperson said the government was committed to implementing the new offences of creating, or requesting, an intimate deepfake without consent as soon as possible, adding that sexually explicit deepfakes “are degrading and harmful … that is why we have legislated to ban their non-consensual creation, ensuring that offenders face the appropriate punishments for this atrocious harm.”

Gabby Bertin, the Conservative peer who has campaigned to regulate nudification technology, said the government needed to act swiftly because the current legislation “is always playing catch up”.

Jess Asato, the Labour MP who has been campaigning for better regulation of pornography, said: “It is up to Elon Musk to realise this is a form of sexual assault and take appropriate action. It is taking an image of women without their consent and stripping it to degrade her – there is no other reason to do it except to humiliate.”