Skip to content

Build1 publisher2 min readPublished

A scraper cleared Akamai Bot Manager by attaching to an ordinary Chrome over CDP

Plain requests, a spoofed Chrome TLS fingerprint and headless Chrome all failed on mobile.de, and the first failure came back as 200 OK. Driving an ordinary Chrome over the DevTools protocol returned 64 listings.

The Engineer · Build desk

What happened

  • A scraper pulling car listings from mobile.de found that a plain requests.get with a normal User-Agent header returned a challenge page, delivered with an HTTP 200 OK status.
  • The run that worked used the ordinary Chrome binary with a real user profile, started with remote debugging enabled and attached over the DevTools protocol with connect_over_cdp.
  • That session loaded the pages and produced 64 complete listings.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • exposure Any pipeline whose success test is the HTTP status will write challenge pages into the table as records, and nothing downstream flags it until a human reads a row.
  • decision Detection-resistance now forces a deployment choice: patch headless tells on somebody else's release schedule, or run real Chrome processes with user profiles and debugging ports on your workers.
  • cost Maintenance is the recurring bill, and it lands on whoever owns the crawler when the wall changes and the IPs get flagged.
  • constraint The evidence base is one target and one session, so cost per page, concurrency limits and durability of the CDP route are all untested.

A status code describes the transport. The challenge page came back 200 OK with a bot wall in the body [3], so the check that catches it is a content assertion: does the parsed document contain the fields the extractor expects, and does a run of a hundred pages yield a hundred rows with prices in them. A crawler that grades itself on `r.status_code` will report a perfect run.

TLS impersonation addresses the handshake: cipher order, extension list, the JA3/JA4 signature [4]. The author's request carried a real Chrome signature and was blocked anyway [4]. According to the post, Akamai does not stop at the handshake, and runs JavaScript inside the page that measures a rendering engine, timing and browser internals that a bare HTTP client does not have [5].

Headless Chrome under Playwright or Puppeteer got closer and was still flagged [6]. The leaks named in the account are automation flags and missing or inconsistent browser internals that a stock headless build carries out of the box [7]. "You can spend days patching each tell one by one, and the wall moves the following week," the author wrote [8].

What worked was a Chrome the author started himself: the ordinary binary, a real user profile, remote debugging enabled, then `connect_over_cdp` against the local debugging port [9]. That session extracted 64 complete listings [10]. The account changes several things at once, and does not isolate whether the profile, the non-headless binary or the attach-instead-of-launch step is what cleared the check.

Three of the four techniques failed against one site [13]. The result rests on one target, one session and one practitioner writing on dev.to [1]. Several conditions have to hold for the 64-listing result to transfer. The site you care about would have to be running the same Akamai Bot Manager configuration [2]. Your worker would have to keep a real Chrome and a real user profile on disk and a debugging port open [9]. And one browser per worker, loading pages at a browser's pace, would have to be enough throughput for the volume you need [10].

The post is clearest about the operating cost. "Getting past the wall once is an afternoon. Keeping it working for six months is a different job," the author wrote [11]. The account puts the ongoing work in diagnosis, maintenance and knowing when to change approach entirely, because IPs get flagged and what passes today gets caught next month [12].

What to watch

  • Whether the real-Chrome-over-CDP route still works on mobile.de after the next Akamai policy update; the post documents one session.
  • Whether anyone publishes a measured comparison across multiple Akamai-protected sites.
  • Whether attaching over a local debugging port acquires its own detectable signature as the technique spreads.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories