Build1 publisher2 min readPublished
Same-model rewording costs SlopShape's structural AI detector 0.9 points of F1
Sitefire's SlopShape spots AI-written company blog posts at 97.0% macro-F1 from page structure alone, and at 96.1% after each model rewords its own output. Paraphrasing generated copy hides little from it, though the test reworded text without reordering any page's sections.
The Engineer · Build desk
Drafted by a language model from the sources cited here and checked against its claim ledger before publication. How we use AISend a correction
What happened
- The dataset pairs 2,250 human blog posts from 268 company domains with 11,250 AI rewrites produced by five frontier models.
- Of the classifier's 203 features, 176 are structural, covering section order, evidence type, voice consistency, semantic HTML and DOM depth.
- Word-level detectors catch unedited AI text almost perfectly but fall to near-random after one rewording pass by the model that wrote it.
- The classifier also names the correct source model for 68.6% of AI posts, against a 16.7% chance rate across six classes.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- decision Anyone publishing generated pages at volume now has to weigh varying section order and evidence types, the evasion step the write-up's author expects to come next.
- exposure At about four times chance, attribution lets a page's layout point to the vendor model behind it as well as flag it as machine-written.
- cost By the write-up author's account, screening at crawl scale puts an LLM call in front of every page before the cheap tree runs, so verdicts come later from a batch queue.
The 0.9-point loss follows from where the classifier looks. An LLM reads each page and fills in structured features, then a gradient-boosted tree classifies those features. No neural network is trained on the raw HTML [5]. According to the dev.to write-up of the paper, rewording changes words but not structure, so the tree receives the same feature vector [6]. The rewording test therefore measures how far the extractor's reading of structure moves when only the wording changes [2].
The evaluation choices deserve credit. SlopShape comes from Sitefire, a YC W26 company [7]. Every human post predates ChatGPT, so its authorship is not in doubt [1]. The test set holds out whole companies. A high score cannot come from memorising one firm's house style [8]. The extractor's labels were checked against human gold annotations. It agreed with the annotators at kappa 0.951, slightly above the 0.939 that two humans reached with each other [9].
For the headline score to hold on the open web, real AI pages have to resemble the AI pages in this set. Here each human post was rewritten once by each of five models, so AI posts outnumber human ones five to one [1]. The write-up flags a harder case: a human post edited by an AI assistant could carry structural fingerprints from both classes [10]. Its author also argues that human posts sit in rare structural configurations, while AI models converge on a tidy, self-announcing shape [16].
I'd expect the result to hold against tools that only swap words. A page with reordered sections, and evidence switched from statistics to anecdotes, is a different input to the extractor [4]. I would want that measured before trusting the score against it.
The two-stage split also helps maintenance. When generator output drifts, the tree is retrained and the extractor stays as it is [14]. The paper does not name the extractor model or the tree library, according to the write-up, whose author guesses a GPT-4-class model and XGBoost or LightGBM [15].
What to watch
- A test of SlopShape against AI pages whose section order and evidence types were deliberately rewritten.
- Results on AI-assisted human posts and post-2022 pages, where neither label is clean.
- Disclosure of the extractor model and tree library, so the 97.0% can be reproduced outside Sitefire.