Build1 publisher2 min readPublished
The German sentiment head-to-head turns on how five stars collapse into three labels
A dev.to post scores nlptown's multilingual star model against oliverguhr's German-specific classifier on twenty hand-written German sentences, and three of those sentences expect the label mixed, which exists only on the answer key.
The Engineer · Build desk
What happened
- A dev.to post compared two German sentiment models on the same twenty real-world sentences, hand-written to cover the cases that break sentiment scoring.
- The other model, oliverguhr's german-sentiment-bert, is German-specific and emits positive, negative and neutral directly, skipping the remapping step.
- Three of the twenty sentences carry the expected label "mixed", a class neither model can return.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Both models drop the three mixed rows as labelled, so the test can separate them only across the remaining 17 sentences.
- decision Anyone reusing a star-rating model has to pick the cut points before measuring anything, and giving neutral just one of the five star values is a product decision made in the preprocessing.
- exposure In a queue routed on this mapping, every three-star reading becomes neutral, so a lukewarm complaint is filed alongside a billing question.
- constraint At five points of accuracy per sentence, the set cannot resolve two models that sit within a row of each other, whichever way the totals fall.
Start with the star scale. nlptown's checkpoint is multilingual, trained on product reviews, and returns a rating from one to five [2]. The mapping in the post sends one and two stars to negative, four and five to positive, and leaves the middle star as neutral [3]: two star values each for positive and negative, one for neutral [17].
Sentence 11 shows what that costs. The text is "Das Produkt ist nicht so schlecht, wie ich erwartet hatte." and the expected label is positive, filed under negation [6]. For the star model to be scored correct there, it has to return four or five; a three-star output is remapped to neutral and the row is marked wrong [18]. oliverguhr's model answers the same sentence directly in its own three-way label space [4].
Three rows in the set are labelled mixed [8]. Sentence 16 is "Die Lieferung war super schnell, aber die Qualität hat mich leider enttäuscht.", expected mixed [7]. Mixed is not in either model's label set [2][4]. That caps both at 17 of 20, or 85 percent [15].
Eight of the twenty expected labels are negative [12]. Answer negative to all twenty and you score 40 percent [14]. That is the floor any accuracy figure from this set has to beat before it says anything about reading German. Nine of the sentences sit in the three clear categories, and the other eleven are the negation, sarcasm, mixed and colloquial rows [13].
With twenty items, one sentence is five percentage points [16]. Two models a single row apart are indistinguishable here, and the set was hand-written to hit the cases that break sentiment models [5], so the category mix is not the mix in a live queue. For the ranking to transfer, your tickets would have to fail in roughly the proportions the author chose.
The scenario the post opens on is a support team reading reviews in five languages, one of whose angry customers got marked as satisfied [19]. One multilingual checkpoint covers all five of those languages [2]. The German-specific model covers one [4], so matching that coverage means five models plus a language identification step in front of them [20].
The text available here ends with the eval set, before any predictions or scores [10]. So far the post tells you what the twenty sentences are asking. Whether the German-specific model reads them better is still open. The author wrote that "Sarcasm doesn't translate through a keyword, and a dissatisfied customer doesn't always spell it out directly either" [9].
What to watch
- The per-sentence predictions and accuracy figures from the run, which would show whether the two models differ on the negation and sarcasm rows or only on the clear ones.
- A version of the set sampled from a real ticket queue instead of hand-written cases, to test whether the category mix changes the ranking.
- A scoring rule that credits either half of a mixed sentence, which would raise the 17-of-20 ceiling for both models.