The split, not the model: 0.606 to 0.818 AUROC without touching a hyperparameter

AUROC starts at 0.5, not at zero. Measured against that floor, the time-split baseline held 0.106 of usable ranking signal and the random-split version of the same model held 0.318, three times as much apparent skill ^[5][6][1]. The LightGBM defaults, the Morgan fingerprints and the hyperparameters were identical in both runs ^[8].

Size that against the variation a modeller would normally report. Five random seeds spanned 0.803 to 0.830, a band of 0.027 ^[6]. The choice of split moved the score about eight times further than reseeding did ^[7][2]. On a board that does not print its split type, the largest single term in the headline number is the one nobody documented.

The mechanism is in how the data gets generated. A promising scaffold is modified into a family of close analogues, and a random split can distribute that family across training and test, so the model is graded on compounds that resemble examples it has already seen ^[9]. A chronological split asks it to predict measurements reported after its snapshot of knowledge ends ^[9]. Both produce an AUROC on hERG. Only one of them is a forecast.

The label side is the second piece of arithmetic. Repeat measurement is common here: of 11,972 records retained after unit checks and parsing, 9,788 were distinct compounds, so roughly 18 percent were re-measurements ^[4][7]. FINAL-Bench compared hERG values for the same compound across different publications and found, over 5,185 pairs covering 857 compounds, a median absolute difference of 0.148 log units and a mean of 0.475 ^[12]. A mean sitting three times above the median is the signature of a long tail ^[4], and the tail is where the damage is: at the 90th percentile two papers disagree by 1.338 log units, more than twenty-fold in potency ^[13]. From that spread the project estimates a single-measurement standard deviation of 0.421 log units and prints it beside model scores ^[14].

That floor is why classification got demoted. 696 of the 1,338 test compounds sit within one noise floor of the pIC50 = 5.0 cutoff, and LEADBOARD now scores regression first and classification second ^[15]. Fifty-two percent of the classification test set ^[3] could change class if the assay were run again in another lab, which means a ranking on that metric is partly a ranking of measurement luck.

Two limits on the finding. FINAL-Bench does not claim that published drug benchmarks generally use random splits, and notes that scaffold splits are common and set a different level of difficulty ^[10]; the 0.211 figure comes from one endpoint, built from 12,021 ChEMBL 37 records reduced to 9,788 unique compounds ^[4][7]. And the evidence is self-published on Hugging Face by the party that runs the board ^[2], whose founder's credentials, 12 patent applications and four co-authored papers, are sourced to VIDRAFT's own site ^[16]. The board's authority rests on 18,382 held-out compounds whose labels participants never see ^[3], which leaves the organiser as the only party able to score it.