After our practical test of Jev, an obvious question came up in the comments: how does Laya compare? Laya is an open model under the Apache 2.0 license. It returns the same kind of judgment as Jev and runs on your own graphics card. No traffic to a vendor, no cost per request.
The question is fair, because Laya compares itself with Jev. The model card shows a large comparison chart and says openly that the Jev figures come from third parties. Jev was never measured there. We have access to both, so we measured both on the same inputs.
The result up front, as a rule of thumb: going local is a rebuild, not a swap. If you replace Jev with Laya and change nothing else, you lose. If you redesign the setup around Laya, you gain speed and independence, but you have to fill two gaps yourself.
What Laya is
The weights sit on the Hugging Face account convaiinnovations, the code on GitHub. Laya knows the same three question types as Jev: pick one option from a set (choice), a yes/no probability (noul) and a level on a scale (score). It writes no text. It returns probabilities, in a single forward pass.
There are three checkpoints, meaning three trained versions of the model:
- an English model based on ModernBERT-large with 421 million parameters,
- a multilingual model based on mmBERT-base with 322 million,
typed-decisions, also 421 million, trained on typed decisions.
A bundled router detects script and language and picks the checkpoint accordingly. The model card recommends it as the default path. There is now also laya-serve, a server with the same interface as Jev. Existing programs are meant to only change the address.
The model card names its own limits: above 20 options Jev leads, the probabilities ship overconfident, and yes/no questions can follow the answer labels instead of the text. The first limit shows up clearly in our data. For the second, it depends on the checkpoint. We also found two weak spots we did not see listed there: picking a value from a text, and the bundled shortlisting.
The setup
Laya ran locally on an RTX 4090, Jev through the API as in the first article. Both got the same questions. They are not a transcript. They come from the same program functions that built the requests to Jev. The same script scored both.
We measured two parts:
| # | Part | Cases | What it covers |
|---|---|---|---|
| 1 | Eight small tasks | 628 | Mail triage, prompt injection, claim checking, AI tone, alerts, invoices, a weakness probe and the public set deepset/prompt-injections |
| 2 | Skill routing | 561 | Which of 117 skills fits a request, the same requests as in the first article |
Everything ran zero-shot, without any fine-tuning on our data. That is the question we were asked: can Laya take Jev’s place?
The eight small tasks
There are ten measures here. We compared each one paired, case by case on the same inputs.
| # | Task | Measure | n | Jev | Laya router | Laya typed-decisions |
|---|---|---|---|---|---|---|
| 1 | Prompt injection | correct | 80 | 100.0% | 63.7% | 60.0% |
| 2 | deepset | correct | 116 | 75.0% | 66.4% | 66.4% |
| 3 | AI tone | correct | 58 | 100.0% | 43.1% | 63.8% |
| 4 | Alerts | correct | 68 | 100.0% | 70.6% | 75.0% |
| 5 | Claims | correct | 74 | 97.3% | 86.5% | 73.0% |
| 6 | category | 77 | 92.2% | 66.2% | 59.7% | |
| 7 | needs reply | 77 | 89.6% | 40.3% | 37.7% | |
| 8 | exact urgency | 77 | 48.1% | 22.1% | 19.5% | |
| 9 | Invoice | due date | 60 | 83.3% | 15.0% | 8.3% |
| 10 | Invoice | amount | 60 | 100.0% | 31.7% | 23.3% |
With default settings, Laya loses all ten measures. Nine gaps are statistically significant (p below 0.05), only deepset on the router is not. Four of the ten measures sit at 100 percent for Jev. Our synthetic sets are too easy there, and the gap partly measures that ceiling.
The table is not the whole story, though. Two adjustments recover a lot.
The checkpoint matters
On AI tone, the router orders the texts about as well as chance: the AUROC is 46.0 percent. AUROC measures whether a model ranks the cases correctly, independent of any threshold. 50 percent means guessing. The same questions on typed-decisions give 93.8 percent. The router picks by language, not by task, and for these tasks it picks wrong.
The threshold matters
By default, Laya scores a yes/no question at 0.5. If you instead fit the threshold on one part of the data and test on the other, typed-decisions rises on AI tone from 63.8 to 77.6 percent. On deepset it reaches 81.9 percent against Jev’s 89.7. The 89.7 is exactly the figure from the first article, and all three values use the same method, two splits of the data. The deepset gap is narrowly significant (p = 0.049).
So if you measure Laya at 0.5, you measure the default setting. On short yes/no judgments, Laya gets within reach with the right checkpoint and a fitted threshold, but it stays behind Jev.
Picking a value breaks it
It looks different when a value has to be picked from the text. For the invoices, the model gets several dates and has to name the due date. The router gets 15.0 percent right, typed-decisions 8.3. Random guessing would give 24 percent. Jev is at 83.3. No threshold helps here, because there is nothing to threshold.
How well Jev reads exactly this kind of detail from real invoices is shown in our lab. There, Jev only gets the free text of 34 public test invoices and says for ten details whether they appear in the text. You can replay the measurement and move the thresholds yourself.
One question even runs backwards. On “needs reply” in the mail triage, the AUROC is 31.8 percent for typed-decisions and 35.7 for the router. The probability points the wrong way. This matches a limit the model card names itself (issue 156): a yes/no question can follow the labels “false” and “true” instead of the text. As a workaround, the card recommends asking the same question as a choice between two neutrally named options. We did not test that. All yes/no questions ran exactly as they went to Jev.
Skill routing
Our assistant has 117 skills. In the first pass, Jev gets all 117 descriptions in one question, a median of 23,766 tokens. Laya cannot take this question at all. Its encoder, the part of the model that reads the text, holds at most 8,192 tokens. And by default, all options together get only 256 of them.
The result follows:
| # | Setup | correct in first place |
|---|---|---|
| 1 | Jev, first pass, all 117 options | 88.8% |
| 2 | Jev, full chain with second pass | 90.6% |
| 3 | Laya, all 117 options | 5.5% |
| 4 | Laya, all 117 options, budget raised to 4,096 of 8,192 tokens | 4.3% |
| 5 | Laya, its own shortlist of 20, then decide | 4.1 to 4.9% |
| 6 | Laya, Jev’s three candidates, multilingual | 42.2% |
| 7 | Laya, Jev’s three candidates, typed-decisions | 52.2% |
Scored over the 490 requests that have a skill assigned. Correct means the first place is the intended skill or a checked alternative. That is why the Jev values differ from the table in the first article, which counts differently. Every Laya setup trails Jev, paired, with p below 0.0001.
The shortlist is missing
For large option sets, Laya’s GitHub README names two paths: raise the budget, or shortlist by embedding first and then decide over the small set. We raised the budget above. For the shortlist we took exactly the bundled path. Among its 20 suggestions, the right skill appears in 19.4 percent of cases with the multilingual checkpoint and 22.7 percent with typed-decisions. Drawing blindly would give 18.2 percent. A plain keyword search (BM25) reaches 56.7.
So the bundled shortlist does not work here. It averages the encoder’s internal states, and that is not a representation trained for search. Building locally means adding a separate search model. We have not measured that.
Telling close candidates apart does not work either
Rows 6 and 7 are the fairest comparison. Laya gets exactly the three candidates Jev kept after the first pass, with full descriptions. There is no context limit left. Counting only the 478 requests where the right skill is among the three, Laya gets 53.6 percent right with typed-decisions and 43.3 with the multilingual checkpoint. Chance with three options is 33.3, Jev is at 92.9.
For me this is the decisive finding. Even where context is no obstacle, Laya lands about 20 points above chance and Jev almost 60.
Calibration
In the first article we wrote a lot about Jev’s calibration. The question is whether a stated confidence of 90 percent turns out right in 90 percent of cases. On the test half of the routing, it looks like this:
| # | Model | n | ECE | answers at 99% confidence or more | of those correct | answers below 50% |
|---|---|---|---|---|---|---|
| 1 | Jev, first pass over 117 options | 257 | 0.056 | 103 | 100.0% | 21 |
| 2 | Laya typed-decisions, three candidates | 257 | 0.083 | 0 | 215 | |
| 3 | Laya multilingual, three candidates | 257 | 0.365 | 27 | 22.2% | 36 |
The tasks are not the same: Jev decides over 117 options, Laya over three. The comparison here is only whether the stated confidence matches the hit rate.
ECE is the average gap between stated confidence and hit rate, lower is better. In the top band, the multilingual checkpoint states 99.6 percent and is right 22.2 percent of the time. That is exactly the failure the Laya page attributes to generative models. The card of the multilingual model gives the reason: it ships without a fitted temperature. typed-decisions is the opposite. It states less than 50 percent in 215 of 257 cases and is right more often than it claims in every occupied band. It errs on the side of caution rather than overconfidence.
Speed
Here Laya is clearly ahead. On the 4090, a request takes a median of about 21 milliseconds with version 0.3.20, and 22 with 0.3.5. Jev took 694 milliseconds per request for us, but that is the round trip to the API with eight parallel requests over our connection. Independent measurements put Jev at 236 to 276 milliseconds. The two are not the same measure, but the direction is clear.
Money is not an argument, on the other hand. Jev cost a total of 0.0149 US dollars for all 628 small cases.
Two explanations that did not hold
My first guess was the option budget. Laya splits its context into a part for the options and a part for the text. For the invoice questions, the options are longer than the default 256 tokens. So we raised the budget to 512. For claims, mail and due date, not a single answer changed, and the amount got worse, from 23.3 to 10.0 percent.
In the routing I tried a second time, with 4,096 instead of 256 tokens for the options. The hit rate dropped from 5.5 to 4.3 percent, at four to six times the compute time depending on the version. The budget is not the reason.
One more correction before publication: my draft reply on LinkedIn first said Laya was level with Jev on deepset. That depended on the method. With five splits of the data the gap was not significant, with the two splits from the first article it was. The version that went out used the method from the article, and that is what this article uses too.
Re-measured with version 0.3.20
We measured on 22 September with Laya 0.3.5. By the morning of 24 September, 13 more versions had been released. An article on outdated figures would be unfair to the project, so we repeated every run with 0.3.20.
The weights on Hugging Face have not changed since 19 September, and the release notes say so too. Only the surrounding library is new. Of 4,833 cases, 4,818 are identical in every single answer. The other 15 are router cases that the new language detection now sends to the multilingual instead of the English model. Two values shift slightly as a result: deepset on the router from 66.4 to 67.2 percent, alerts from 70.6 to 66.2. Every statement above still holds.
Limits of this measurement
| # | Limit |
|---|---|
| 1 | Everything zero-shot. The model card calls the checkpoints a base for specialization and ships a notebook for fine-tuning. We did not test that. |
| 2 | 117 skills is our inventory. With ten or twenty options the routing would look different, and this measurement says nothing about that. |
| 3 | We did not measure a separate search model in front of Laya. That is the setup that would show whether going local with two models works. |
| 4 | Most test sets are synthetic, generated by Claude Sonnet and checked by Claude Opus. The only public, human-labeled set is deepset, and that is where the gap is smallest. |
| 5 | Time: Jev is a round trip to the API, Laya a forward pass on the local card. |
| 6 | Yes/no questions ran as noul, as with Jev. We did not test the two-option choice the model card recommends as a workaround for issue 156. |
What we take from this
Laya is a serious project that moves fast and documents its limits openly. For short yes/no judgments on your own hardware, it gets within reach if you choose the checkpoint deliberately and fit the threshold.
For our use case that is not enough. Jev is clearly better with many options, with picking values from text, and with fine distinctions between similar skills. If you want to stay local, you put a search model in front and fine-tune on your own data. That is a project, not a configuration change.
What fine-tuning on our own data would bring is still open. As of today, the rule of thumb from the start holds.
Sources
All public, retrieved on 24 September 2026.
- Laya, model card and weights: huggingface.co/convaiinnovations/laya
- Laya, code and documentation: github.com/NandhaKishorM/laya, nandhakishorm.github.io/laya
- Laya on PyPI with all versions: pypi.org/project/laya
- Our first article with the setup and the Jev figures: Jev in practice
- Our lab, Jev on public e-invoices: Jev reads e-invoices
- External benchmark: huggingface.co/datasets/deepset/prompt-injections
- Independent Jev latency measurements: AbdelStark/jev-benchmarks, nibzard/decision-model-benchmark
The Laya values come from our runs on an RTX 4090 with laya 0.3.5 (22 September 2026) and 0.3.20 (24 September 2026), torch 2.11.0. The Jev values come from the runs for the first article, model version jev-1.13.0.