Build1 publisher2 min readPublished
Cloudflare's crawl events replace status polling with a Queue signal that still needs a results fetch
Cloudflare now pushes three Browser Run crawl lifecycle events to a Queue, so apps can stop polling for job status. Its finished event carries only a status and page counts, and the app must still fetch the pages and judge whether a partial crawl counts as success.
The Engineer · Build desk
What happened
- Starting a crawl returns a job ID at once, and the app uses that ID later to retrieve the job's status and results.
- Cloudflare's own example of a finished payload shows a job with two completed pages and one errored page.
- Browser Run event subscriptions are account-wide, so a handler has to match each event's job ID to a request the app stored.
- Documented terminal states include timeout, account limits, user cancellation and errors alongside successful completion.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost Every crawl.finished event costs the consumer a second call to the results endpoint before a user can be shown anything, because the message holds no pages.
- exposure A handler that attaches the next event to whichever user is on screen can show one user's crawl to another, since the queue carries every job in the account.
- decision Each product has to set its own minimum usable coverage before summarising, because a completed crawl can still include errored, skipped and disallowed pages.
The events sit on top of the existing job-ID contract as a push channel, and I think the split between signal and payload is correct. A developer's walkthrough on dev.to summarises Cloudflare's September 25 announcement and documentation [1]. According to that post, a crawl.finished message does not carry the crawled page content. It is a cue to call the results endpoint with the job ID [4].
The payload does give the consumer enough to triage. It holds the job status and counts of total, completed, errored and skipped pages [5]. Cloudflare's reference example is itself a partial crawl [6]. The post adds disallowed pages to the list of things a completed crawl can still contain [9].
Status needs the same discipline. A UI that maps any terminal state to a green check mislabels four of the outcomes Cloudflare lists [1]. The post's test forces a failed terminal state onto a job and checks that the screen shows a failure or a clear next step [16].
I expect most of the bugs to show up in correlation. The author, who builds apps freelance [17], saves a record at submission with the requester, the target site, the start time and the provider's job ID [14]. Events with unknown job IDs are not attached to whatever brief is open. A late result for a cancelled request is not published as a fresh answer [15]. On an account-wide queue, the next message can belong to a job whose user closed the tab an hour ago [7]. The ordering test starts jobs for two test users and finishes them in reverse order, and each user should see only their own result [12].
Coverage is a product decision. For a research brief, the author would allow a partial report that lists which sources were used and which were unavailable. Any claim whose only source is a missing page would be withheld [13]. "That is a product rule I would choose, not a guarantee Cloudflare makes," the author wrote [18].
Queues are optional for all of this. Polling a job-status endpoint can be enough at small scale, and the author would not push a beginner onto Queues for a first prototype [10]. The same checks apply on either path [19]. As the author wrote, "a notification tells you to check the work, not to invent its outcome" [11].
What to watch
- Cloudflare documentation on ordering and duplicate delivery of crawl events on Queues; handlers keyed on job ID would need to tolerate a repeated crawl.finished if delivery can repeat.
- Whether crawl.finished payloads gain per-page reasons for skipped or disallowed pages, letting apps label coverage gaps before the results fetch.
- Cloudflare's pricing for Queue delivery of crawl events compared with polling a status endpoint at small scale.