Refusing a BigQuery key cost findmypylibrary 15,000 requests per snapshot

One BigQuery query would have returned name, description, download count and release date for every package at once ^[4]. Refusing the Google Cloud account and key put 15,000 HTTP requests into every snapshot instead, one per package ^[6]. Constraint 3 charges that cost to the user's machine on each refresh.

The crawl is why `fetch.py` is async ^[18]. Sequentially the log clocks it at over an hour; `httpx.AsyncClient` with a semaphore of 25 concurrent requests brings it to a few minutes ^[6]. An hour spread across 15,000 requests is about a quarter of a second each ^[1], and 25 in flight puts the wall clock near 2.4 minutes ^[2]. The log does not report response times under load. That figure transfers only if PyPI serves 25 concurrent clients at the latency it serves one.

The floor of the corpus is around 68,000 downloads a month ^[8], while the top result in the demo output shows 163,835,611 in 30 days, roughly 2,400 times as many ^[10][4]. The log records the product owner's answer on corpus size as "15,000, or whatever covers all actively downloaded libraries" ^[7]. The file held exactly 15,000 rows, so the working definition of "active" cost nothing to write ^[8].

Read the demo output as a claim about one query. "fuzzy string matching" returns RapidFuzz at 0.90, pfzy at 0.81 and fuzzywuzzy at 0.73 ^[10], and the words of the query appear in all three one-line summaries ^[11][5]. The first ranker was a pure-Python BM25 ^[12]. The query the README advertises is `findmypylibrary "parse messy pdfs"` ^[13], where the user's vocabulary and the package's need not overlap, and the log does not show that result.

The BigQuery key gets the refusal documented at full length ^[4], while the embeddings decision appears twice as an outcome ^[3].

Click groups reject unknown command names, which would have broken the bare query the README already promised, so `click.Group` was subclassed and `resolve_command` retries the arguments as `["search", *args]` ^[13]. Steps 12 and 15 found three more cases ^[14]. The package had been sitting on PyPI as a 0.0.1 name reservation whose README described both commands before either was written ^[15].

`asyncio.as_completed` yields in completion order, so results cannot be zipped back onto the input list to attach download counts, and each task was written to return its own complete record with the count included ^[16]. Every pipeline test during development ran `refresh --limit 40`, with the full crawl kept for the runs that needed real data ^[17]. The first bug was a documented dataset URL that returned HTML because a 301 to a new domain went unfollowed; the fix was the raw GitHub URL, and the log drew the lesson that a 200 should never be assumed and redirects should be followed ^[19].