Jev has been in many feeds since 15 September. TypeSafe released the model that day, together with a funding round and a Doom demo that The Register wrote about as well. Jev writes no text. It returns typed judgments with a probability: one option out of a set, a yes or no as a probability, a level on a scale.
We did not comment on it, we measured it. Four days after the release, on a task we face every day.
Our internal assistant has 117 skills. Every request therefore starts with a decision before anything else happens: which tool is the right one here, and is one needed at all? Look at that decision and you find a list of names with one line of description each. That is all the model has when it chooses. For exactly this case, TypeSafe proposes an upstream judgment in its skill suggestion cookbook, measured against 182 skills of a third-party harness.
This article shows the setup, all the numbers and the places where we were wrong.
What others built in the first days
In the four days up to our test, several projects appeared that we did not measure ourselves and therefore name only as references: a browser agent that selects the next element instead of describing it, a trading bot with one decision per block, an open replication on its own hardware and a field report from LangChain. The vendor’s own comparison figures have already been criticised in public, because TypeSafe measured them itself.
Our numbers below are self-measured as well. The difference is that we disclose the test material, the questions and the limits.
The setup
1,732 examples, eleven tests, two blocks. The first block asks one question: does a router find the right skill from a request? The second block tests what else the model can do and ends with a public benchmark we did not build ourselves.
Claude Sonnet wrote the 561 realistic requests from the skill files, Claude Opus reviewed every assignment, added equivalent alternatives and discarded anything unclear. The weakness probe is generated in code, where the correct answers are exactly fixed. Every hit rate carries a 95% Wilson interval, every comparison between two routers runs paired over the same requests with an exact McNemar test.
Block 1: choosing the right skill
| Test | Requests | Data | Routers |
|---|---|---|---|
| Trigger sentences | 508 with a skill, 35 without | taken from the descriptions, description stripped of triggers for this | Jev, keyword search |
| Realistic requests | 561: 461 with a skill, 29 about missing services, 71 without a skill | Sonnet writes, Opus reviews | Jev in three variants, keyword search, Haiku and Opus each without and with the Jev hint |
| After sharpening the descriptions | 297, the test half | same requests, new descriptions | the same five routers |
Block 2: eight further tasks
| Task | Question type | n | Data |
|---|---|---|---|
| Mail triage: category, reply needed, urgency | Choice, Noul, Score | 77 | synthetic, reviewed |
| Detect prompt injection | Noul | 80 | synthetic, four difficulty levels |
| Check a claim against its source | Choice | 74 | synthetic, three claims per text |
| Does the text sound like AI | Noul | 58 | synthetic, half and half |
| Is an alert actionable | Noul | 68 | synthetic, operations and noise |
| Invoice: due date and amount payable | Choice over candidates | 60 | synthetic, several dates per text |
| Weakness probe: counting, dates, numbers, negation, arithmetic | Choice and Noul | 95 | generated in code, labels exact |
| deepset/prompt-injections | Noul | 116 | public, labelled by humans |
The same definitions also went to Claude Haiku and Claude Opus, there as a batch prompt in plain language instead of a typed question.
Result 1: Jev lands at the level of the small model
| Router | total correct | skill correct | wrong skill | none chosen | false alarm |
|---|---|---|---|---|---|
| Claude Opus alone | 94.3% | 93.5% | 1.0% | 5.5% | 0.0% |
| Claude Opus with the Jev hint | 93.2% | 92.2% | 2.0% | 5.7% | 0.0% |
| Claude Haiku with the Jev hint | 87.9% | 86.5% | 6.7% | 6.7% | 2.8% |
| Jev alone, full descriptions | 84.3% | 82.9% | 6.7% | 10.4% | 5.6% |
| Jev alone, no triggers, shortened | 83.2% | 81.8% | 7.3% | 10.8% | 7.0% |
| Jev alone, description shortened | 83.1% | 81.4% | 6.7% | 11.8% | 5.6% |
| Claude Haiku alone | 82.7% | 80.4% | 11.0% | 8.6% | 1.4% |
The “total correct” column counts over all 561 requests, the three middle columns over the 490 requests with a skill, the false alarm over the 71 requests without a skill. The BM25 keyword search is not in the table, because it can never answer “none”. On the requests with a skill it hits 19.0% at rank 1, counted over all requests 16.6%.
On the trigger sentences the same keyword search still hit 52.6%, on real requests 19.0. That drop shows the basic problem: real requests do not contain the words of the description. Someone asking whether the backup ran through last night uses no keyword from the matching skill.
Which differences hold
| Comparison | only a correct | only b correct | p |
|---|---|---|---|
| Haiku alone / Haiku with hint | 15 | 44 | 0.0002 |
| Opus alone / Opus with hint | 6 | 0 | 0.031 |
| Jev / Haiku alone | 46 | 37 | 0.38 |
| Jev / Opus alone | 0 | 56 | less than 0.001 |
| Jev without triggers, shortened / Jev full descriptions | 11 | 17 | 0.34 |
The gap between Jev and Haiku is not statistically significant, the gap to Opus is.
By type of request
| Group | n | Haiku | Haiku with hint | Jev | Opus |
|---|---|---|---|---|---|
| Requests with a matching skill | 461 | 85.5% | 92.0% | 88.1% | 99.3% |
| Requests about missing services | 29 | 0.0% | 0.0% | 0.0% | 0.0% |
| Everyday, no skill needed | 20 | 100% | 100% | 100% | 100% |
| Technical question, no skill needed | 25 | 100% | 100% | 100% | 100% |
| Routine work in the repo, no skill | 25 | 96.0% | 92.0% | 84.0% | 100% |
| German requests | 428 | 82.0% | 86.9% | 83.2% | 94.2% |
| English requests | 133 | 85.0% | 91.0% | 88.0% | 94.7% |
| short requests | 177 | 76.3% | 80.2% | 78.5% | 87.6% |
| long requests | 116 | 87.9% | 93.1% | 90.5% | 99.1% |
The upper five groups add up to 560 requests. The one missing control asks for a service that does not exist and therefore does not sit under “no skill needed”.
Two rows stand out. Short requests are harder for every router than long ones, the gap is around ten points. And the second row sits at zero for all of them. More on that shortly.
Result 2: an extra judgment helps the small model and gets in the way of the large one
We put Jev’s suggestion into the prompt as a hint, the way the vendor’s cookbook proposes. Haiku rises by 5.2 points with it, and that gain is statistically significant. Opus loses 1.1 points. Six cases flip from correct to wrong, none in the other direction.
That has turned into a rule of thumb for us: a foreign judgment helps the model that is unsure, and takes the decision away from the model that is sure. Anyone running a strong model buys, with an upstream router, mainly a second opinion they would not have needed.
Result 3: are the probabilities right?
Jev returns a probability with every choice. The question is whether it can be trusted.
| Confidence for rank 1 | n | mean confidence | actually correct |
|---|---|---|---|
| 0 to 50% | 39 | 34.0% | 41.0% |
| 50 to 70% | 64 | 59.7% | 68.8% |
| 70 to 90% | 89 | 80.3% | 87.6% |
| 90 to 99% | 114 | 94.9% | 99.1% |
| 99 to 100% | 184 | 99.8% | 100% |
The model is consistently a little more cautious than necessary. For practice the last row counts: from 99% confidence upwards the choice was right in all 184 cases. That allows a threshold above which a suggestion is followed and below which it is discarded. How much that threshold covers, and what happens to the rest, is in the addendum at the end.
Robustness of the scoring
Opus reviewed the labels and is measured at the same time. That can favour Opus. So we scored the same answers in two further ways: once strictly, meaning only the originally intended skill without the alternatives Opus added, and once only on the requests the review left unchanged.
| Router | normal, all 561 | strict, all 561 | strict, only the 439 unchanged |
|---|---|---|---|
| Claude Opus alone | 94.3% | 92.9% | 97.9% |
| Claude Opus with hint | 93.2% | 92.0% | 97.0% |
| Claude Haiku with hint | 87.9% | 86.6% | 92.3% |
| Jev alone | 84.3% | 83.2% | 88.6% |
| Claude Haiku alone | 82.7% | 81.6% | 87.5% |
The order stays the same in all three scorings.
Result 4: the gap no router closed
Some of the test requests target services that are not connected to our setup at all. A ticket in a third-party tracker, a page in a note-taking app, a payment at a payment provider. We have a skill for this case. It checks whether there really is a gap, and then makes a proposal instead of running off on its own.
Hit rate on these requests: 0%. For all five routers, Opus included.
The model was not the problem. The description was. It said what the skill is meant for, but not in a way that a router recognises from a normal request. With Jev this is visible in the ranking: in these 29 requests the skill ranked first 14 times and was among the top three 25 times. Every time, the pre-check of the two-stage chain discarded it.
So we sharpened 39 descriptions that had been confused. Opus wrote them, a second Opus run checked every new version against the actual skill file. Measurement ran on the half of the requests nobody had seen while writing. What the rewrite does to the probabilities is in the addendum.
| Router | before | after | lost / gained | p | missing services (n = 20) |
|---|---|---|---|---|---|
| Claude Opus alone | 93.3% | 99.0% | 3 / 20 | 0.0005 | 0 to 100% |
| Claude Opus with hint | 92.6% | 99.0% | 1 / 20 | less than 0.001 | 0 to 100% |
| Claude Haiku with hint | 87.2% | 89.6% | 11 / 18 | 0.26 | 0 to 55% |
| Claude Haiku alone | 82.8% | 88.2% | 14 / 30 | 0.023 | 0 to 65% |
| Jev alone | 82.8% | 85.9% | 1 / 10 | 0.012 | 0 to 20% |
Two things in this table are easy to miss. With Opus the entire net gain sits in the group of missing services, on requests with a matching skill Opus loses three out of 237 in the process. With Jev that group carries only four of the nine cases gained. And the advantage of the Jev hint, which was statistically significant for Haiku before, is not afterwards: 89.6% against 88.2% at p equal to 0.63.
The price of the exercise is measurable too. The skill list grows by 9,965 bytes and is loaded into every session. Our own ceiling for it is 80,000 bytes, 78,808 are currently used. All 39 descriptions together break it. The one that carries the gain costs 294 bytes and fits.
Result 5: what else the model can do
| Task | n | Jev | Claude Haiku | Claude Opus |
|---|---|---|---|---|
| Mail category, seven classes | 77 | 92.2% | 87.0% | 100% |
| Mail: reply needed | 77 | 89.6% | 90.9% | 97.4% |
| Mail: urgency exact, most likely level | 77 | 59.7% | 45.5% | 62.3% |
| Claim against source | 74 | 97.3% | 98.6% | 100% |
| Invoice: due date | 60 | 83.3% | 100% | 100% |
| Invoice: amount payable | 60 | 100% | 100% | 100% |
| Prompt injection, own data | 80 | 100% | 100% | 100% |
| Sounds like AI | 58 | 100% | 100% | 100% |
| Alert actionable | 68 | 100% | 100% | 100% |
One row needs an explanation. On urgency Claude names one level, Jev returns a distribution over the four levels. The comparison uses the most likely level. Take the rounded expected value instead, which Jev also returns, and it hits only 48.1%.
Four rows sit at 100% for all models. That is a finding about our data, not about the models: the reviewer deliberately sorted out unclear cases, which made these tasks too easy. That is exactly why we brought in a public dataset.
The external benchmark
The dataset deepset/prompt-injections is public, labelled by humans and contains German and English texts. We took the test split with 116 examples.
| Model | hit rate at threshold 0.5 | precision | recall | AUROC | Brier |
|---|---|---|---|---|---|
| Claude Opus | 94.0% | 100% | 88.3% | 0.998 | 0.052 |
| Claude Haiku | 90.5% | 100% | 81.7% | 0.929 | 0.100 |
| Jev | 75.0% | 100% | 51.7% | 0.985 | 0.187 |
Jev never raises a false alarm, but misses 29 of 60 injections. The ranking of the cases is almost perfect, which the AUROC of 0.985 shows, above the one from Haiku. The default value of 0.5 simply does not fit this task. With a threshold of 0.03 to 0.06 calibrated on half the data, Jev reaches 89.7%, cross-validated on the other half.
Rule of thumb from this: with a judgment model, the threshold is part of the application, not of the model.
The weaknesses the vendor names itself
| Task | n | Jev | Claude Haiku | Claude Opus |
|---|---|---|---|---|
| count directly | 20 | 65.0% | 100% | 100% |
| count by single questions, sum in code | 20 | 95.0% | not measured | not measured |
| compare dates, mixed formats | 20 | 100% | 100% | 100% |
| compare numbers, German and English formats | 20 | 95.0% | 100% | 100% |
| negation and double negation | 20 | 100% | 100% | 100% |
| arithmetic close to a threshold | 15 | 66.7% | 100% | 100% |
TypeSafe names these weaknesses itself and by name. Of five announced, two were confirmed. The recommended detour works: ask per list entry and add up in code, and counting rises from 65% to 95%.
Speed and cost
| Task | Requests | Tokens per request | Median runtime |
|---|---|---|---|
| Skill routing, two calls per request, run with the sharpened descriptions | 561 | 26,380 | 1.94 s |
| Mail triage, three questions in one call | 77 | 677 | 0.71 s |
| Claim against source | 74 | 674 | 0.69 s |
| Alerts | 68 | 361 | 0.68 s |
All tests together: 6,202 TypeSafe calls, 46.7 million input tokens, 1.96 US dollars. The price in the model overview is 0.042 dollars per million input tokens, output free. Across all calls the median is 0.78 seconds. The Claude side is not included in that, it ran as a subagent on the subscription and consumed a further 28.7 million tokens.
The uncomfortable part
Before the test we built a filter that removes customer names from everything that goes out. On the requests it worked. On the descriptions only partly: 14 mentions remained there, two of them customer and partner names. And it did not cover at all that the second pass additionally sends 700 characters from every skill file. Those excerpts contained customer names and two personal names, and one skill name carried a customer name anyway.
The author did not find this, an independent review round did, which we set on the internal report after writing it. Three review passes produced ten severe and more than forty medium findings, among them a wrong runtime figure, an inconsistent scoring and a misstated contractual position. This article went through the same review once more.
The lesson is uncomfortable and simple: anyone who puts a foreign model into a tool chain has to know the entire path of the payload, not only the request they wrote themselves. For us that means, specifically: the filter gets extended before any of this runs in production.
On the legal position, because the question comes up in every customer conversation: according to the privacy policy the service runs in the USA. The data processing agreement incorporates the EU standard contractual clauses, with Irish law and Irish supervision. No training on inputs. A waiver of retention exists only in the enterprise contract. Whether customer data goes there is decided by the customer, not by us.
When a model like this pays off
For us, not as an upstream router in front of a session with a strong model. The gain there is negative and the data path too long. It gets interesting where many small judgments have to be made fast and a smaller model decides: pre-sorting alerts, checking incoming texts for hidden instructions, checking records against their source. Always with a threshold calibrated on your own data, and calibrated again as soon as the descriptions change.
Limits of this measurement
- The test requests are synthetic, not the real requests from daily work. Real prompts would have handed customer content to a service in the USA.
- Opus reviewed the labels and was measured at the same time. The strict scoring reduces that advantage, it does not remove it.
- Claude answered in batches of 25 requests, Jev each request on its own. In a real session Claude decides per message and with more context.
- The Claude runs additionally had the session’s real skill list in context, even though the instruction was to use only the list that was handed over.
- Development half and test half are split at random, but are of the same type. Whoever sharpens the descriptions therefore knows the kind of errors, even without seeing the test requests.
- One model version was measured, four days after the release.
- The addendum measures calibration over all 490 requests with a skill, Result 4 measures accuracy over the test half. The two series are not directly comparable.
- Three of our own tasks were too easy. Only mail category, due date, counting and the external benchmark separate reliably.
That says nothing about sustained operation, for a decision it is enough.
In the end the cheapest gain was not in the new model. It was in 294 bytes of description that nobody had written before, because everyone knew what the skill does.
Addendum, 22 September 2026
Two questions under the LinkedIn post about this article hit a gap in the section on probabilities. How much does the top confidence band actually cover? And did we measure the calibration again after rewriting the descriptions? Both are fair. Here are the answers.
What the top band covers
The 184 requests from 99% upwards are 37.6% of the 490 requests that have a right skill. Not of the 1,732 examples, which is the material across all eleven tests together.
| Threshold | coverage | correct within | rest | correct in the rest |
|---|---|---|---|---|
| from 90% | 298 of 490, 60.8% | 99.7% | 192 | 71.9% |
| from 95% | 249 of 490, 50.8% | 99.6% | 241 | 77.6% |
| from 99% | 184 of 490, 37.6% | 100% | 306 | 82.0% |
So the last row of the calibration table does not carry the decision on its own. Cutting at 99% pushes 306 of 490 requests below the line, and the model would have answered 82.0% of them correctly, so the cut discards mostly right answers. At 90% it separates better: 298 requests pass and are 99.7% correct, while the remaining 192 sit at 71.9% and are better off with the larger model.
That table counts the 490 requests with a skill. In operation all 561 pass the threshold, including the 71 that no skill fits. At 90% exactly one of those slips through in the earlier run, which puts 299 requests through at 99.3% rather than 99.7%. From 99% upwards none slips through, and after the rewrite none does at 90% either.
The calibration after the rewrite
We had not measured it. The raw run was still there, so we did it now, with the same method as above. The table runs over all 490 requests with a skill, so the before column is the same one as above. The hit rates in Result 4 run over the test half only.
| Confidence for rank 1 | n before | n after | correct before | correct after |
|---|---|---|---|---|
| 0 to 50% | 39 | 26 | 41.0% | 61.5% |
| 50 to 70% | 64 | 47 | 68.8% | 74.5% |
| 70 to 90% | 89 | 96 | 87.6% | 94.8% |
| 90 to 99% | 114 | 116 | 99.1% | 100% |
| 99 to 100% | 184 | 205 | 100% | 100% |
The occupancy moves. From 99% upwards the coverage rises from 37.6% to 41.8%, from 90% upwards from 60.8% to 65.5%.
Those figures stand on all 490 requests, so that they stay comparable with the table further up. That set includes the half on which we sharpened the descriptions. On the test half, which nobody saw while rewriting, 257 requests remain, and there the gain in coverage looks different.
| Threshold | coverage before | coverage after | correct below, before | correct below, after |
|---|---|---|---|---|
| from 90% | 61.5% | 64.2% | 68.7% | 83.7% |
| from 99% | 40.1% | 40.9% | 79.9% | 90.1% |
On the clean half the occupancy barely moves: from 99% upwards it is 0.8 points instead of 4.3. What clearly rises there is something else, the share of correct answers below the threshold, from 79.9% to 90.1%. The threshold does not become more permissive, it discards more right answers.
The bottom end is where it gets awkward. Before, the model stated 34.0% there on average and landed at 41.0. After, it states 37.6% confidence there and lands at 61.5. The mean distance between stated confidence and accuracy rises from 0.041 to 0.069, so the calibration gets worse. The Brier score, which measures hits and confidence together, falls from 0.073 to 0.047.
So the accuracy gets better and the calibration gets worse. No sign flips along the way: the model states less than it hits in every band, before and after the rewrite. The gap only grows. It looks largest in the bottom band, from 7 to 24 points, but that band holds 39 requests before and 26 after. The difference is not established, the exact test gives p = 0.13. We name it because it points the same way as the rest, not as evidence.
What is established is the finding that matters. Below the recommended threshold of 90%, the test half held 68 correct answers out of 99 before and 77 out of 92 after, so 68.7% against 83.7% at p = 0.018. In practice: a threshold calibrated before the rewrite discards suggestions that would have been right.
That turns into a rule we did not have before. If you change the descriptions, you measure the threshold again. The descriptions are part of the input, not part of the environment.
Addendum, 23 September 2026: Jev reads e-invoices
This article measures Jev as a skill router. The BKS lab now has a second test bench, closer to our work with e-invoices. An e-invoice carries many details twice, as a structured field and as free text. If a detail appears only in the text, such as an early payment discount, software that reads only the fields does not see it.
So Jev gets only the free text of an invoice and answers ten yes/no questions in one request. The code reads the same details from the structured fields and compares.
On 34 public test cases published by KoSIT this produced 340 verdicts in 11.3 seconds. Only 4.1 percent of them fall between 0.3 and 0.7, so Jev almost always commits. In 9 cells a detail appeared only in the text. Six of them held up under the cross-check, three did not, and the page shows both.
You can replay the measurement on the lab page and move the thresholds yourself: Jev reads e-invoices.
Sources
Everything public, retrieved on 20 September 2026.
- TypeSafe, product page and announcement: typesafe.ai and typesafe.ai/blog
- Documentation: docs.typesafe.ai, question types, HTTP API, models, prices and limits
- Known weaknesses of the model: docs.typesafe.ai/model-jaggedness/jev-1.13
- Skill suggestion cookbook, the template for our hint arm: docs.typesafe.ai/cookbooks/skill_suggestion
- Agent skill and plugin from TypeSafe: github.com/typesafe-ai/skills
- Legal: data processing agreement, privacy policy
- External benchmark: huggingface.co/datasets/deepset/prompt-injections
- Press and criticism: The Register, ts2.tech on the self-measured comparison figures
- Projects from the first days: jev-ultrafast, jev-trader, openjev, LangChain blog
The measurements in this article come from our own runs against the API, as of 19 and 20 September 2026, model version jev-1.13.0.