250 Japanese labels put a 310M encoder 12 points ahead of a hosted decision API

Task shape here is the difference between a label that sits in the surface words and a label you have to read for. The topic task was livedoor, nine classes ^[19]. A character n-gram pipeline with logistic regression on top beat Jev there and lost badly on short-text polarity, a swing of 32.4 points for one feature extractor across two tasks ^[20]. Vocabulary identifies the topic of a long Japanese document, and counting character n-grams finds vocabulary. Short-sentence polarity turns on wording.

The trained system is sbintuitions/modernbert-ja-310m with a classification head, and its numbers are out-of-fold: train on 200 rows, predict the held-out 50, five times, so all 250 reported predictions come from a model that never saw that row ^[10]. The linear baseline ran the same protocol ^[10]. That is what makes the two rows comparable.

The tie on polarity is where that discipline matters. At n=250 the independent confidence interval is about plus or minus 6 points ^[11]. Trained 88.8 against Jev's 94.4 is a gap of 5.6 points ^[21]. That sits inside the interval, so only a paired test on identical row ids decides it, and McNemar on those ids is how every comparison in the run was scored ^[11].

Trained on the same 250 labels, the GLiNER-family classifier scored 88.8 / 92.8 / 75.2 ^[5], gains of 26.0, 5.6 and 30.4 points over its own zero-shot row, all significant ^[6]. Zero-shot, its three-class polarity score sat 11.5 points above the 33.3 chance line ^[22].

Two other models in the run failed on the same rows. SemIf, a Qwen3.5-4B system, lost all three tasks, two of them significantly ^[16]. Laya's median confidence on the nine-class task was 0.029 ^[14], while a uniform guess over nine classes would assign 0.111, so its median confidence sat below its own prior ^[23]. Laya's own documentation gives a zero-shot English score of 0.362 ^[15]. The author of the dev.to post calls this a zero-shot defect and not a Japanese one.

The choice comes down to the labeling cost. Jev costs $0.042 per 1M input tokens with output free, offers roughly 32k of context, returns a confidence number, and needs no training data ^[7]. If every one of a task's 250 rows filled that window, the run would be 8M tokens, about $0.34 ^[24]. The adoption cost is 250 labeled rows per task and five training runs per reported number, against an API call that needs neither ^[10][7].

For 88.8 to transfer to your topic task, three things have to hold: the label set is fixed, the documents are long enough for lexical signal to separate the classes, and you are willing to label 250 rows in the language you actually run. On the third, the author wrote: "A decision API wins when you have no labels; a small trained encoder wins when you have a few hundred" ^[18].