Security1 publisher2 min readPublished
Contractors grading Copilot image edits see users' photos with faces uncensored
Contractors hired to grade Copilot's image edits see each user's original photo, faces uncensored, beside the request and two AI outputs, 404 Media reported. Microsoft did not say whether users can keep their uploads out of those reviews.
The Watch · Security desk
What happened
- Interviewees told 404 Media the contractors are paid to grade instruction-following, image preservation and overall look, and are not asked to flag inappropriate content or make legal or safety calls.
- Many of the requests reviewers saw were sexual and some involved children, including edits to shorten skirts, enlarge body parts and place people in sexual positions.
- The updated consumer Copilot app, available since August 18, does not use prompts, responses or uploaded file contents to train foundation models, Microsoft says.
Compiled by The WatchSomething wrong?How this is made
Why it matters
- exposure People who appear in uploaded photos, and not only the users who upload them, are identifiable to an outside workforce, including subjects of edits that appear nonconsensual.
- constraint The training opt-out and the August 18 change both address model training; neither is documented as keeping an upload away from a human grader at a contractor.
- decision Organisations whose staff use consumer Copilot on work images have to set policy on the basis that outside contractors can view those files, since Microsoft has not said which accounts feed the reviews.
The exposure comes from a paid review workflow run by third-party firms [2]. It reaches two groups: the Copilot user who uploaded the photo, and the people in it, whose faces reviewers can see [3]. In many cases the edits appear nonconsensual, according to the report [5]. One contractor described being shown multiple sets of upskirt photos. Another was asked to rate foot-fetish images of children's cartoon characters [5].
Microsoft's documents set out retention and training terms but leave the review question open. The Privacy FAQ for the older consumer Copilot keeps conversations for 18 months by default, and users can delete them at any time [8]. It says Microsoft may use data "including the images or files you upload" for AI training [9]. The FAQ also refers to human reviews that users cannot opt out of; whether those include the image-rating tasks is not specified [10]. The August 18 statement on the updated app covers foundation-model training [11], and neither document is shown to keep an upload off a contractor's screen [1]. Microsoft uses customer data from some accounts to improve its products, according to Malwarebytes. 404 Media asked whether users can exclude uploads from the rating tasks and got no answer [7].
For companies, the open item is which accounts feed the pool. The reporting and the FAQ concern consumer Copilot [8][11], and "some accounts" is as specific as Microsoft's description gets [7]. The evidence stops at consumer accounts. A work photo put into a consumer Copilot account can reach a person at a third-party firm with faces intact, and Microsoft has documented no way for the user to prevent it [2].
Copilot's image tools have drawn internal warnings before, though the 2024 cases concerned what the generator would produce and this report concerns who sees what users feed it. In early 2024, Microsoft machine learning engineer Shane Jones went public with concerns that guardrails in the DALL-E 3 text-to-image tool could be bypassed [12]. He said he had asked Microsoft to remove Copilot Designer from public use for three months and the company refused [12]. He told CNBC at the time that the feedback team received over 1,000 product feedback messages a day and dealt only with the most serious [13]. In the same period Microsoft fixed a Copilot Designer loophole that allowed nonconsensual intimate deepfakes of Taylor Swift [14].
Microsoft is one of several AI companies using human reviewers. OpenAI also hires contractors to review AI chats, under its Project Lily initiative, according to Malwarebytes [15].
What to watch
- Microsoft's answer to 404 Media's question on whether users can keep uploads out of human image reviews.
- Any Microsoft statement on whether commercial or work Copilot accounts supply images to the contractor rating tasks.
- Whether the privacy terms for the updated consumer Copilot app, available since August 18, address human review separately from training.