Chatbot refusal rules built around nudity let hijab removal through in Guardian tests

Two of the vendors' own explanations show where their rules draw the line. Asked why it would take off a hijab but not a dress, ChatGPT said the "distinction is that removing a hijab changes a head covering, while removing a dress would expose the person's body" ^[6]. The measures xAI announced in May target requests to digitally undress "images of real people in revealing clothing such as bikinis" ^[8]. Both rules are written around exposing a body ^[6][8]. In the Guardian's tests, neither one stopped a hijab coming off ^[2].

In my view the Gemini result matters more to a deployer than the outright compliance. Gemini refused on a principle broader than nudity, saying it could not alter someone's appearance without their consent ^[4]. It then produced the edit when the request was framed as making the woman look more "western" ^[4]. A missing rule shows up in the first round of testing. A refusal that holds on the direct request and gives way on a reworded one passes a quick test, and Gemini's gave way twice in the Guardian's tests ^[4][10].

A skeptic would say the consent rule was never engaged, because every test image was AI-generated ^[1]. The point holds for the images, but Gemini raised consent itself, and the only thing that changed before it made the edit was the wording of the prompt ^[4]. The post that prompted the tests started from a photograph of a real woman crossing a street ^[11].

For a team putting image editing in front of customers, the trade-off is between launch date and coverage. Relying on vendor refusals costs nothing extra and ships this quarter. It also means adopting the vendor's harm categories as the company's own. Eviane Leidig, director of research and outreach at the Center for the Study of Organized Hate, said the distinctions show gaps in the guidelines that govern how chatbots behave ^[12]. She said de-veiling may not distinctly violate platform rules but is still harmful ^[16]. An in-house test set covering reworded requests and religious dress costs staff time before launch. Skipping it pushes the cost into a later quarter, when a customer's edit of a real person goes out under the company's name.

Generative AI is already part of this abuse, according to a CSOH study. The study found it producing images in India that sexualize Muslim women or dehumanize Muslims, and similar material aimed at Ilhan Omar and Rashida Tlaib in the US ^[14]. "This is just part of an ongoing trend of how we see Muslim communities, and particularly Muslim women, being targeted and abused," Leidig said ^[15].

The evidence is narrow. Claude, the one chatbot of the four that did not produce the image ^[1], said it has no image-editing feature and would not remove a hijab ^[3]. The tests ran a few prompts against each chatbot ^[1], and the reporting does not say whether API or enterprise configurations behave the same way. ChatGPT, asked about the difference after it had made the hijab edit, put the harm in its own words: "The fact that an edit doesn't create nudity doesn't mean it can't violate someone's privacy, dignity, or religious boundaries" ^[7].