Build1 publisher2 min readPublished
Laya's small encoders answer typed triage questions in a single forward pass
Laya's 322M- and 421M-parameter encoders answer typed routing questions in one forward pass, generating no tokens. Swapping out an LLM judge for it makes sense only after its labels are checked against tickets your team has already tagged.
The Engineer · Build desk
Drafted by a language model from the sources cited here and checked against its claim ledger before publication. How we use AISend a correction
What happened
- Questions come in three types: choice picks a label, score rates an intensity, and noul returns a yes/no answer as a probability.
- A router reads the script and language of the input, sending English text to the English checkpoint and everything else to the multilingual one.
- Given "My payment failed twice", the triage preset returned five answers from one pass, led by a technical_help intent at probability 0.811.
- The project reports 33ms per question on a T4 GPU, dropping to 7.2ms per question when requests are batched.
- Created on September 18, 2026, the repo had 26,639 stars and 2,326 forks by September 27, with commits landing the day the write-up's author tested it.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost The per-call API fee goes away and hosting takes its place: roughly 650 MB of weights and about 1 GB of RAM at inference, on hardware the team runs itself.
- constraint Same input, same output holds only per checkpoint, so on a repo this young, pinning the package version (the write-up used laya==0.3.21) keeps last week's decisions comparable to this week's.
- decision Confidence comes back with every answer, so adopting Laya means choosing a cutoff below which tickets still go to the LLM judge or a person, and that cutoff decides how much of the old bill remains.
A generative judge answers by writing prose. A parser then has to pull a label out of that prose [12]. Laya drops both steps. The text and every typed question go through the encoder once, and each question type has its own output head [2]. Its response reports output_tokens: 0 [3]. There is no sampling step, so the same text through the same checkpoint should return the same scores. The write-up calls the judgments deterministic [14].
I like the typed interface. A score comes back as a number and a yes/no as a probability, so there is no answer to regex out of prose and no JSON to hope parses [2][12]. Run with no flags, the CLI only detects script and language and downloads no weights [8]. You get to see the router before it asks for 650 MB of disk [10].
Treat the speed figures as the project's own, measured on a T4 GPU [5]. Batching cuts per-question time by about 4.6 times [2], so the 7.2ms number assumes a queue deep enough to fill batches. The unit needs care too. Latency is quoted per question, yet one pass answers every question [2][5]. If the per-question figure adds up, the five-question triage preset takes about 165ms unbatched [3]. A frontier-model call adds a second or more, by the write-up's account [12]. Its author ran everything on a CPU-only VM [10], so a CPU deployment needs its own timing.
Adoption has been fast. Nine days after creation, the repo was averaging about 2,960 stars a day [1]. "That is not normal. It is the kind of spike you see when a project names a pain everyone already feels," the write-up's author wrote [7]. Stars count people who pressed a button on GitHub. The write-up does not report accuracy for any checkpoint against a labeled set.
According to the write-up, Laya's pitch is that 95% of judge decisions never needed a reasoning engine, only a fast classifier with typed outputs, confidence scores and the ability to abstain [13]. For that share to hold on a given queue, the preset's label definitions have to match the ones the support team already uses. In my view the only accuracy number that transfers is agreement, per question, with tickets the team has already tagged, measured beside the current judge. The sample output shows where disagreement would start. For "My payment failed twice," the preset scored frustration at 2.28 on a 0-3 scale, urgency at 0.002 and a refund request at 0.004 [9].
What to watch
- A published accuracy evaluation of the English and multilingual checkpoints against a labeled triage or moderation set.
- CPU latency for the five-question triage preset on the 322M multilingual checkpoint, measured by someone outside the project.
- Checkpoint updates on Hugging Face that shift scores for users still pinned to laya 0.3.21.