A #1 ranking only its author could see, and eleven days spent measuring the wrong end

A developer publishing small automation tools on the Apify marketplace had 23 listings live by August, ranking well in store search, and after 89 days he had one active user and no revenue ^[1][2][3]. He then spent eleven days resolving that contradiction in the wrong direction, and the reasoning he used is more instructive than the bug at the end of it ^[25][4].

The logic was that if ranking was fine and users were zero, the fault had to be downstream ^[4]. So he went downstream, properly. He pulled 3,655 listings, then went back and pulled 12,834 to check the first pass for sampling bias, found one, and corrected it ^[5]. He split the corpus by whether the title contained a well-known platform name and got median users of 5 against 2 ^[6]. He measured the base rate for new listings: 11 percent get a first user within 0 to 3 days of publishing, against 74 percent at 14 to 30 days ^[7]. His listings were young, so the zeros looked statistically unremarkable ^[8]. He acted on all of it, renaming five tools and adding output schemas, which moved the platform's own quality score from 74 to 78-79 ^[9]. Nothing moved ^[10].

Look at what that age-cohort number did. A roughly sevenfold difference between the 0-3 day and 14-30 day cohorts is a real-looking effect ^[23]. It also rests on nine listings, of which exactly one had acquired a user, since 11 percent of nine is one ^[28]. That single observation supplied the alibi: the anomaly was reclassified as normal, and the contradiction stopped demanding an answer. Tripling the corpus to 12,834 improved the statistics about everyone else's listings without testing a single claim about his own ^[24][5].

The condition he never varied was that he was logged in ^[13]. Every measurement went out from a browser session, and later a token, belonging to the author of the 23 tools, even though store search is an ordinary public endpoint ^[13][14]. Sent with no credentials, the ranking was not there; it existed only inside his own session ^[15]. His summary of the error is the part worth keeping: re-running a measurement under identical conditions reproduces the same bias as faithfully as it reproduces the same truth, so repetition rules out transient noise and nothing else ^[12].

The cause was an exclusion list. The endpoint takes an `includeUnrunnableActors` parameter, and the documentation defines unrunnable as actors from developers who have not passed KYC, or full-permission actors without a large user base ^[16]. Two conditions, OR'd, and the second is checkable without a token because every listing exposes its permission level: all 23 came back as LIMITED_PERMISSIONS and public, leaving identity verification, which the platform console stated in words ^[17][18][19]. Nothing in the code was broken. The tools were live, runnable and genuinely well ranked, inside an index that is not served to logged-out visitors ^[20].

Note what was never testable. Across 2,047 tool-days there was one user, but no logged-out visitor could reach any of the 23 listings, so the demand question the eleven days were spent on could not have been answered from that data at all ^[26][2][15].

The detection he ends on costs two unauthenticated calls: query the endpoint with and without the flag, compare, and treat present-with-flag but absent-without as exclusion rather than a ranking problem ^[21]. Worth running against your own listings before the next round of naming analysis. His own remedy is paperwork ^[22].