The online network X has provoked widespread anger after people employed its AI chatbot Grok to manipulate pictures of women by taking off their attire.
Several instances have been seen where the technology has been employed to virtually strip women to show them in swimwear without consent, as well as to put them in inappropriate scenarios.
XAI, the company that created Grok, offered no comment to requests for comment, aside from an standard message stating "legacy media lies".
A Home Office representative stated that legislation is being brought forward to outlaw nudification tools. Under a upcoming criminal offence, anyone who provided such technology would "receive a prison sentence and heavy penalties".
"X or Grok is capable of halting these forms of abuse if they desired to," said a law professor, adding they "seem to have a lack of consequences."
The communications watchdog Ofcom said technology companies must "consider the risk" of people seeing unlawful material on their networks, but did not specify whether it was currently looking into X or Grok in concerning artificial intelligence images.
Grok is a no-cost artificial intelligence—with some premium functions—which answers X users' queries when they reference it in a tweet.
It is frequently used to provide reactions or further context to other posters' remarks, but people on X are also capable of alter an uploaded photo through its artificial intelligence picture modification tool.
The chatbot has been condemned for enabling users to create photos and clips with nudity and sexually suggestive material, and it was in the past accused of generating sexually explicit content of public figures.
"The platform has been allowing the creation and spread of these pictures for a long time without taking any action and we have yet to see any action by oversight bodies," the expert added.
XAI's own terms of service bans "portraying likenesses of individuals in a graphic manner".
In a statement, Ofcom said it was unlawful to "create or circulate private pictures or child sexual abuse material" and stated this included sexual deepfakes made with AI.
It said platforms such as X were obligated to take "suitable measures" to "reduce the danger" of users viewing unlawful content on their platforms and to delete it quickly when they learn of it.
Outdoor enthusiast and travel writer sharing tips and stories from the Dutch wilderness.