The online network X has sparked significant anger after people employed its AI chatbot Grok to change pictures of women by stripping their attire.
Multiple cases have been seen where the tool has been used to digitally undress women to make them appear in swimwear without permission, as well as to insert them in inappropriate situations.
Company Response and Regulatory Stance
XAI, the organization responsible for Grok, offered no comment to requests for comment, except for an standard response stating "traditional news outlets lies".
A Home Office official stated that legislation is being proposed to outlaw nudification technology. Under a upcoming law, anyone who supplied such technology would "face a prison sentence and substantial fines".
"The platform is capable of halting these forms of misuse if they chose to," said Clare McGlynn, adding they "seem to have impunity."
Oversight Body Position and Platform Capabilities
The communications watchdog Ofcom said tech firms must "consider the danger" of people seeing unlawful material on their networks, but stopped short of confirming whether it was presently examining X or Grok in relation to AI-generated images.
Grok is a free chatbot—with some premium functions—which reacts to X users' queries when they tag it in a message.
It is commonly used to give reactions or further context to other users' remarks, but people on X are also can modify an posted image through its AI image editing feature.
Background of Criticism and Policy
The system has been criticised for permitting users to generate pictures and clips with nudity and sexually suggestive content, and it was previously alleged to have been involved in generating sexually explicit material of celebrities.
"The platform has been enabling the generation and sharing of these pictures for a long time without taking any steps and we have not seen any challenge by regulators," Professor McGlynn added.
XAI's own usage rules forbids "showing images of people in a graphic manner".
In a statement, Ofcom said it was unlawful to "make or share private pictures or illegal content" and clarified this applied to sexual deepfakes made with AI.
It said services such as X were mandated to take "appropriate actions" to "lessen the danger" of users coming across unlawful content on their platforms and to remove it promptly when they learn of it.
