Skip to content

Build1 publisher3 min readPublished

Calling Actor.charge before the fetch bills for a row the run never delivers

In three days of building pay-per-event scrapers on Apify, the first bug was a billing one. The charge call resolved, the browser died in the next line, and the run billed for a detail row anyway.

The Engineer · Build desk

Illustration accompanying Calling Actor.charge before the fetch bills for a row the run never delivers

What happened

  • Three days after a 10 September start, the author had eleven pay-per-event scrapers, eight live on the Apify Store and three waiting for a publishing slot, with revenue still at zero.
  • The first detail-page pipeline charged before it fetched, and in a test run the browser died between those two lines, so the run billed for a detail it never delivered.
  • The corrected order fetches through a retry wrapper that is allowed to throw, charges only after it resolves, and pushes the row last.
  • The incremental-mode store had the same defect: marking all 100 scanned ids as seen meant a budget cut-off after 40 delivered rows discarded the remaining 60 permanently.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • exposure A charge that lands before the row does turns every transient fault into money taken for data the buyer never receives, and the delivered dataset is the evidence.
  • constraint Progress markers inherit the same ordering rule. State written before the row leaves the process erases rows the buyer never got, and no later scheduled run can tell they existed.
  • decision Which endpoint you find in DevTools sets the per-row margin before the listed price does, so profitability on a metered Actor is decided during the build.
  • cost Fetch-then-charge moves the cost of failure onto the operator, who pays compute for every retry while only a delivered row produces revenue.

Actor.charge and Actor.pushData are two separate awaits in the same loop body [5]. In the first version the charge resolved first, so the platform had a billable "detail" event on record while the fetch that produces that detail had not run [6]. Nothing in the published snippet links the two calls, so statement order is the only control the code has.

The corrected block is three lines. The author wrote that the fix "is boring and it belongs in every metered pipeline" [11], and it reads: fetch through a retry wrapper that may throw, then charge, then push [9]. When fetchDetailPage fails after its retries, control leaves the block before the charge line, and the buyer keeps the list row it already paid for [10]. "Nobody complained, because the only buyer was me," the author wrote of the run that exposed it [8].

The same order applies one layer up, to state. Most of these Actors have an incremental mode where a scheduled run returns only rows the buyer has not received, backed by a set of ids in a named key-value store [12]. Mark while scanning and the store records a delivery for every id it scanned [13]. The published loop calls markDelivered only when pushCharged reports that the row went out, and writes the state once at the end [14]. One scraper compounded it by delivering newest-first and marking as it went, so a cut-off left a hole in the middle of the timeline. It now buffers oldest-first, and that leaves the surviving state as a contiguous range [15].

Every Actor in the set is priced per row delivered [4]. The cost figures come from platform runs at 256 MB with no proxy unless stated [16]. The expensive case is a scraper that drives a real browser, gets past Cloudflare and loads one page per row: about $0.0046 per row, which the post puts at 460 times the cost of the same output shape from a JSON endpoint [17]. Divide and the JSON row lands near $0.00001 [18]. For that ratio to transfer, the target has to answer a plain Node fetch at 256 MB without a proxy. Several do: Meetup, Eventbrite, Kalshi, Polymarket, Himalayas, Jobicy and every European company registry return clean JSON at the right URL, which the author calls "the highest-paid ten minutes in the whole build" [19]. Substack's search endpoint does not. It ignores a plain fetch even with a copied user agent, needed got-scraping from the Crawlee family locally, and needed a residential proxy on the platform. The publication endpoints are fine with plain fetch [20].

An undelivered row is one failure mode. A wrong row is the other, and per-row pricing bills both the same. Passing limit to the Himalayas search endpoint silently returns twenty rows forever, because the listing endpoint takes a cursor and search takes page [23]. Meetup answers location=Berlin with events near the server's IP, a US datacenter city on a cloud run, so the Actor geocodes the city once through OpenStreetMap and sends de--Berlin instead [21]. Workday's board comes back only from a POST carrying { limit: 20, offset } as JSON and an Accept-Language header, while every other ATS in the set is a GET [22]. And a keyword filter built on includes() matched "rust" inside "trustworthy" and returned a job post that never mentioned the language. It is now a built regex with a Unicode-aware boundary that still lets c++ and node.js through [24].

What to watch

  • Whether any of the eleven Actors earns revenue from a buyer other than the author, and whether a charged row gets disputed.
  • What the Substack search path costs per row once the residential proxy it needs on the platform is priced in.
  • Whether Apify's charge API gains a way to reverse an event after a failed push. That would make statement order less critical.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories