Decision models: typed questions, answers with probabilities

A decision model does not write text. It answers a question of a fixed type and returns a probability for each option. Code decides what follows from that. On this page you see it in recorded answers from Jev, a decision model by TypeSafe, and drag the threshold yourself.

Draw the line yourself →Each dot is a request for which Jev picks the right tool (a skill). Right means sure, left means unsure. Filled: right pick, outlined: wrong. 490 judgments from jev-1.13.0, measured on 2026-09-19. Where would you draw the line?

At a glance

How a decision is made

A decision model like Jev from TypeSafe works with three parts: a state in named fields, a question with a fixed type, and code you can read, test and change.

Our example is the project match: a CV is checked line by line against a project posting. The cases come from our test bench and are invented. One requirement line from the test bench goes step by step through the model to the verdict in code. TypeSafe does not publish Jev's internals, so the opening figure starts with Winnow-12B, an open model that answered the same question. Switch to Jev to see what stays visible. Pick a case and a model, step forward and back, and compare the same step across three models.

Opening figure · How a decision is made

Recorded answers from 2026-10-01, no live request.

1 Input

One text: template, state, question, options as letters

Template<|turn>system You judge a project posting and a candidate CV. Answer each multiple-choice question with the letter of exactly one option.<turn|> <|turn>user
State: linerequirement “Build end-to-end test automation with Playwright and TypeScript for our customer portal”
State: CVcv 7 entries, one line each
QuestionQuestion: How well does `cv` show that the candidate has done what `requirement` asks for?
Options as letters
- A) Not at all
- B) Adjacent or transferable experience
- C) Done in at least one role or project
- D) Done repeatedly, across several roles or projects
Start of the answerempty, thinking mode off: no reasoning first. The model writes the first letter itself.

Template: the fixed frame in which this model expects questions. Markers like <|turn> belong to it.

731 tokens measured

Full prompt, word for word

<|turn>system
You judge a project posting and a candidate CV. Answer each multiple-choice question with the letter of exactly one option.<turn|>
<|turn>user
`requirement`:
Build end-to-end test automation with Playwright and TypeScript for our customer portal

`cv`:
cv.x-profile: Senior Test Automation Engineer (SDET) for web, API and CI/CD. Eleven years in quality assurance, eight of them in test automation. Builds maintainable end-to-end and API test suites, wires them into pipelines and teaches teams to own their tests.. B.Sc. Computer Science, Hochschule Bremen, 2012 - 2015. German (native), English (fluent)
cv.s-e-commerce-sdet: Senior SDET (freelance), Tidewave Commerce GmbH (03/2022 - present): - Built an end-to-end suite with Playwright and TypeScript for a web shop: 1,400 tests, run in Azure DevOps pipelines on every pull request API tests for the REST services (orders, payment, inventory) with Playwright and contract tests with Pact Cut the flaky test rate from 9% to under 1% through test isolation and stable test data Coached three development teams on the test pyramid and code review for tests Test reports for product owners in Azure DevOps dashboards
cv.s-insurer-automation: Test Automation Engineer (freelance), Hanse Versicherungsdienste AG (06/2019 - 02/2022): - Migrated 600 Selenium tests (Java) to Playwright, runtime from 3 hours to 25 minutes BDD scenarios with Cucumber, written together with the business unit API tests with Postman and Newman in the Azure DevOps pipeline Set up test data management for contract and claims scenarios
cv.s-telecom-qa: QA Engineer, Kombitel Services GmbH (09/2016 - 05/2019): - Selenium WebDriver tests in Java with TestNG for the customer portal Mobile tests with Appium for the Android and iOS apps Jenkins pipelines for nightly regression runs Defect triage together with development in Jira
cv.s-agency-tester: Software Tester, Pixelfabrik Medien GmbH (10/2014 - 08/2016): - Manual and exploratory testing of websites and web applications for agency clients First Selenium IDE and WebDriver scripts for smoke tests Test documentation in Confluence, bug tracking in Jira
cv.p-llm-testgen: Test case drafts with an LLM (2024): Internal tool that turns user stories into Playwright test drafts through an LLM API. A tester reviews every draft before it is merged. Used by two teams as a time saver for new features. Stack: TypeScript, Playwright, Azure OpenAI.
cv.p-flaky-radar: Failure classification dashboard (2023): Collects test results from pipelines and groups failures by cause (test data, timing, environment, real defect). Helps the team decide which failures to look at first. Stack: Python, pytest, Grafana.

Question: How well does `cv` show that the candidate has done what `requirement` asks for?
A) Not at all
B) Adjacent or transferable experience
C) Done in at least one role or project
D) Done repeatedly, across several roles or projects<turn|>
<|turn>model
<|channel>thought
<channel|>

2, 3 Model
- Template
- State: line
- State: CV
- Question
- Options as letters
Each token becomes a list of numbers
Schematic: Split and numbers are schematic, count is measured
Layer by layer the numbers mix
Schematic: Schematic: roughly how information flows, not the real weights
Winnow-12B, open language model fine-tuned for decisions
hf.co/EldanRing/Winnow-12B:Q8_0, local on an RTX 4090 through Ollama, 442 ms · Model card under further reading
4 Next token
CD
- A 0.000
- B 0.000
- C 0.737
- D 0.261
- other tokens (1 − mass) 0.002
Mass 0.9983sum over all option letters
C first token: “C”
5 Divide
1. A → 0 · not at all0.0000.000 → 0.000
2. B → 1 · adjacent, transferable0.0000.000 → 0.000
3. C → 2 · done in one role0.7380.737 → 0.738
4. D → 3 · done repeatedly0.2620.261 → 0.262
C: 0.7369 ÷ 0.9983 = 0.738
divide by the mass
6 Code
Code decides, not the model
bench/match.ts, readRequirement()
1. const level = Math.max(0, Math.min(3, x.level?.score ?? 0));← 2.26
2. cover: level / 3,→ 0.75
Locally the test bench puts the distribution after dividing here (bench/analyze.mjs, line 54).
0123
Expected value 2.26: level times probability, added up
coverage 0.75
Same rule, three models
- Jev coverage 0.90
- Winnow-12B coverage 0.75
- Qwen3.8-27B coverage 0.79

For comparison, our judge: Claude Opus, a general language model (Experiment 3). It answers with a choice and a reason: evidence Senior SDET (freelance), Tidewave Commerce GmbH, level 3, reason “Playwright+TS e2e suite of 1,400 tests; Playwright also at insurer”.

Code reads the expected value 2.26 and divides by 3. Result: coverage 0.75. Coverage means how well the CV backs this line, from 0 to 1. Below, the same rule for all three models.

No model writes the verdict. Each model only returns numbers, code reads them.Why the answer looks different for each question type →What does a threshold do with 490 judgments? →How one score is built from them →

All nine combinations of case and model
Case	Model	Distribution after dividing	Mass	first token	Code reads
A clear line	Jev	0 · not at all 0.00, 1 · adjacent, transferable 0.00, 2 · done in one role 0.30, 3 · done repeatedly 0.70	none, Jev returns the distribution itself	none, Jev returns the distribution itself	coverage 0.90
A clear line	Winnow-12B	0 · not at all 0.00, 1 · adjacent, transferable 0.00, 2 · done in one role 0.74, 3 · done repeatedly 0.26	0.9983	“C”	coverage 0.75
A clear line	Qwen3.8-27B	0 · not at all 0.00, 1 · adjacent, transferable 0.02, 2 · done in one role 0.60, 3 · done repeatedly 0.38	0.9987	“C”	coverage 0.79
A borderline case	Jev	no 0.52, yes 0.48	none, Jev returns the distribution itself	none, Jev returns the distribution itself	does not count
A borderline case	Winnow-12B	no 0.86, yes 0.15	0.9979	“B”	does not count
A borderline case	Qwen3.8-27B	no 0.36, yes 0.64	0.9953	“A”	counts
The model wants to write something else	Jev	no 0.06, yes 0.94	none, Jev returns the distribution itself	none, Jev returns the distribution itself	counts
The model wants to write something else	Winnow-12B	no 0.02, yes 0.98	0.0714	“The”	counts
The model wants to write something else	Qwen3.8-27B	no 0.09, yes 0.91	0.9983	“A”	counts

Source: test bench lab-bench, run 1 per model on 2026-10-01. Prompt rebuilt from bench/run-local.mjs, code word for word from bench/match.ts. Local models through Ollama on an RTX 4090, Jev through the TypeSafe API. Postings and CVs are invented. Reference: claude-opus-5-5, judged blind (Experiment 3).

Try it

Which experiment explains which step

Every step of the opening figure has an experiment that shows it in more detail. Tap a step. All experiments replay recorded answers. None makes a live request.

Tap a step: the experiments that explain it light up. Schematic: Schematic: the six steps of the opening figure as a path, no measured values

How a decision is madeThe opening figure: one line, three models, six steps to the verdict in code.Input · Split into tokens · Layers · Next token · Divide · Code
01One question, three shapesYes or no, choice, level: three shapes of answer, and the code reads one field.Input · Code
02Where does the code draw the line?One line, 490 judgments from skill routing and a second dataset: what goes through automatically and what a person sees.Code
03Who matches the judge?Jev and local models in the case player and in the table, against the English Opus judge.Next token · Divide
04How sure is sure?How often a model says 0.99, and whether it is right then.Divide · Code
05Shuffle the optionsThe same question, the options shuffled. Does the choice stay?Input · Next token
06From single judgments to one total scoreThree knobs of the calculation, ten pairs reorder.Code
Where does it hold, where not?Guess first, then the measurement: uses and weaknesses, each with a source.Layers · Code

Which experiment explains which step
Step	explained in
Input	How a decision is made, One question, three shapes, Shuffle the options
Split into tokens	How a decision is made
Layers	How a decision is made, Where does it hold, where not?
Next token	How a decision is made, Who matches the judge?, Shuffle the options
Divide	How a decision is made, Who matches the judge?, How sure is sure?
Code	How a decision is made, One question, three shapes, Where does the code draw the line?, How sure is sure?, From single judgments to one total score, Where does it hold, where not?

Experiment 2Own measurement · public and own dataset

Where does the code draw the line?

A probability alone decides nothing. Only a threshold in code turns it into yes, no or “please check”. Where it sits is up to the business side.

Experiment 2 · Where does the code draw the line?

Skill routing: 490 judgments from the decision model jev-1.13.0 through the TypeSafe API, measured on 2026-09-19. No live request.

Dataset

Skill routingPrompt injection

Which skill fits this request? A choice question. Code takes the first choice at or above a threshold, below that a person looks.

Request
a routing request, its text is not published
Jev (jev-1.13.0)
not published: architecture, size, base model, training data
According to TypeSafe, a model class of its own, trained with a method TypeSafe calls RLCD.
First choice
p = 0.92
The number Jev returns is where the dot sits in the field below.
modell.ts, entscheide()
1. export function entscheide(p: number, unten: number, oben: number): Entscheid {
2. if (p >= oben) return 'ja';
3. if (p < unten) return 'nein';
4. return 'mensch';
5. }
Result
automatic: take the first choice

first choice right
first choice wrong
wrong without a person
goes to a person

Threshold: at or above it, code takes the first choice0.90

The dots stay where they are. The thresholds only change what code makes of them. That needs no new request.

Free play: drag the threshold and watch how the errors rise and fall with the share decided automatically. The dots stay where they are.

Accuracy without a person

99.7%

297 of 298 right

Decided automatically

60.8%

298 of 490 decided automatically

To a person

192

39.2% of the judgments

Wrong without a person

1

of 298 decided automatically

What the code does with 490 judgments

automatic, right297
automatic, wrong1
to a person192

Trade-off: one measured point per threshold

Skill routing: threshold, decided automatically, errors and handed to a person
Threshold	Decided automatically	Wrong without a person	To a person
0.80	70.4%	6	145
0.85	66.1%	2	166
0.90	60.8%	1	192
0.95	50.8%	1	241
0.97	46.1%	0	264
0.99	37.6%	0	306

Source: our own test requests for skill routing, only those with a fitting skill, written by Claude Sonnet and checked by Claude Opus. Jev jev-1.13.0 through the TypeSafe API, one run on 2026-09-19. The same judgments appear in the article “Jev in practice”.

How the setup works At the top one judgment on its way through the code. Below, each dot is a recorded judgment, placed at Jev's probability, the axis squeezed below 0.80. Filled means the dataset says yes, outlined means no. The band goes to a person. Red marks what code gets wrong without a person. Next to it the trade-off per line.

Scope and limitations

Not part of this demo: The step “Tuned too well” for the dataset deepset/prompt-injections shows why a threshold set here does worse on new data.

deepset/prompt-injections is a public dataset labeled by people, with German and English texts. We use the part of the dataset held back for testing, 116 texts. Claude Sonnet wrote the 490 routing requests from skill descriptions, and Claude Opus checked every assignment. Both runs date from 2026-09-19. The full account is in the article “Jev in practice”.

Experiment 3Test bench pv1 · reference: Opus judges

Who matches the judge?

Jev runs at TypeSafe in the USA. Our focus is sovereign AI, so we want to know which models we can run ourselves and how close they get to Jev today. The test case is the project match, because it has many facets. We ask all models the same questions and compare with a reference judgment.

Measured on the test bench →

Experiment 3 · Who matches the judge?

13 selected cases, run 1 per model on 2026-10-01. No live request.

Important for everything called “right” here: the reference comes from three blind judges running Claude Opus (claude-opus-5-5, judged on 2026-10-01), not from people. None saw a model answer. Two judged the German lines independently, the third judged the English lines shown on this page on its own, so these lines have no cross-check. The same model family can share a blind spot. On 2 October, two more blind judges from other Claude families, Sonnet and Fable, cross-checked the German lines, and separate agents decided the 61 disputes with sources (results/pv1/crosscheck.md in the test bench). The English lines were not part of that, and the numbers on this page still count against the Opus judges as labelled. A human judgment is still to come.

This is the line from the opening figure. Now four models answer.

all match the reference

“Build end-to-end test automation with Playwright and TypeScript for our customer portal”

CV: Senior Test Automation Engineer (SDET) for web, API and CI/CD

Which entry of the CV shows this line?

Option	Jevdecision model	Winnow-12Bfine-tuned	Qwen3.8-27Bgeneral	Shisa DE-1fine-tuned
Senior SDET (freelance), Tidewave Commerce GmbHjudge's choice	1.00	1.00	1.00	0.99
Mass on the letters, first token in the ring	not applicable, Jev returns the distribution directly	0.99 B	1.00 B	1.00 B

7 more options, at most 0.01 together per model

Judge's reason “Playwright+TS e2e suite of 1,400 tests; Playwright also at insurer”

For all three local models nearly everything was on letters already, dividing changes little. How this comes about: opening figure, step 5

Result: all match the reference. 4 of 4 models pick what the judge picked.

All lines against the English Opus judge (reference)

Sort by a column head. A cell opens the lines it is counted from.

Model	n = 119	n = 44	n = 119	n = 119	n = 119	n = 119
Jevdecision model via API
Winnow-12Bopen language model fine-tuned for decisions
Qwen3.8-27Bgeneral language model
Shisa DE-1open language model fine-tuned for decisions
Qwen3-32B (earlier comparison run)general language model
GLiNER2.5 Decide-1Bclassifier, not a language model

Evidence, level and area only count lines the reference sees as a requirement. “Evidence invented” is the share of lines without evidence for which the model still names an entry. Lower is better. The count is shown under each value. * differs from Jev at p < 0.05 (exact McNemar test, paired, run 1). Gaps without a star may be chance.

So far one measurement per model, on 2026-10-01 and 2026-10-08. Each further measurement of the test bench is added here.

The net: two models side by side

Seven axes from 0 to 100 percent, further out is better. A ring marks the top value on each axis.

Speed is relative: 100 % is the fastest time. That is why speed shapes the area so strongly.

Point at or tap an axis, or reach one with the Tab key. The values then appear here.

All values as a table

German lines, adjudicated reference
Model	Speed (relative)	Level exact	Area	Requirement recognised	No invented evidence	Evidence	Mandatory	Time per line
Jev	100.0%	79.3%	93.1%	88.2%	75.0%	81.0%	93.1%	0.25 s
Winnow-12B	10.0%	77.6%	87.1%	86.8%	81.2%	85.3%	94.0%	2.49 s
Qwen3.8-27B	14.2%	69.0%	86.2%	85.3%	89.6%	87.9%	97.4%	1.76 s
Shisa DE-1	10.6%	66.4%	85.3%	85.3%	64.6%	79.3%	75.9%	2.35 s
Qwen3-32B	15.2%	47.4%	76.7%	85.3%	66.7%	77.6%	100.0%	1.65 s

English lines, reference Claude Opus
Model	Speed (relative)	Level exact	Area	Requirement recognised	No invented evidence	Evidence	Mandatory	Time per line
Jev	100.0%	74.8%	91.6%	88.5%	72.7%	79.0%	89.1%	0.25 s
Winnow-12B	10.1%	70.6%	88.2%	89.2%	79.5%	81.5%	97.5%	2.47 s
Qwen3.8-27B	14.5%	63.9%	81.5%	85.6%	90.9%	82.4%	97.5%	1.73 s
Shisa DE-1	10.7%	63.0%	86.6%	85.6%	68.2%	81.5%	63.0%	2.33 s
Qwen3-32B	15.6%	47.1%	71.4%	85.6%	68.2%	78.2%	100.0%	1.60 s

Test bench lab-bench, version pv1, first run on 1 October 2026. German lines against the adjudicated reference of 2 October 2026: 136 lines, 116 of them requirements, 48 of those without evidence. Jev 1.13.0 through the TypeSafe API, the other four models locally on one RTX 4090. “No invented evidence” is 100 minus the share of these 48 for which a model still names an entry. Speed is relative: the fastest time per line (Jev, 0.25 s) divided by the model's time, median over the 136 lines. Jev answers the five questions of a line in one request, the local models need five calls in a row. So time per line does not measure quite the same thing. Cost is not an axis. Locally we only estimated electricity, which is about Jev's list price, and the hardware is not counted, see the section “Speed and cost” below. Unlike the table above, the net counts German lines against the adjudicated reference, like the article.

Test bench lab-bench, version pv1, first run on 1 October 2026. English lines against the judgment of Claude Opus of 1 October 2026, without a cross-check: 139 lines, 119 of them requirements, 44 of those without evidence. Jev 1.13.0 through the TypeSafe API, the other four models locally on one RTX 4090. “No invented evidence” is 100 minus the share of these 44 for which a model still names an entry. Speed is relative: the fastest time per line (Jev, 0.25 s) divided by the model's time, median over the 139 lines. Jev answers the five questions of a line in one request, the local models need five calls in a row. So time per line does not measure quite the same thing. Cost is not an axis. Locally we only estimated electricity, which is about Jev's list price, and the hardware is not counted, see the section “Speed and cost” below.

Which models, where they ran

Model	Type	Version	Where	Template, start of answer	Date	Runs
Jev	decision model via API	`jev-1.13.0`	TypeSafe API, USA	none, request as JSON	2026-10-01	5, one of them shuffled
Winnow-12B	open language model fine-tuned for decisions	`hf.co/EldanRing/Winnow-12B:Q8_0`	local, RTX 4090 through Ollama	`gemma4`, no start of answer	2026-10-01	1
Qwen3.8-27B	general language model	`qwen3.8:27b`	local, RTX 4090 through Ollama	`chatml`, `“Answer: **”`	2026-10-01	1
Shisa DE-1	open language model fine-tuned for decisions	`hf.co/mradermacher/shisa-de-1-GGUF:Q4_K_M`	local, RTX 4090 through Ollama	`gemma4`, no start of answer	2026-10-01	1
Qwen3-32B (earlier comparison run)	general language model	`qwen3:32b`	local, RTX 4090 through Ollama	`chatml`, `“Answer: **”`	2026-10-01	2, one of them shuffled
GLiNER2.5 Decide-1B	classifier, not a language model	`fastino/GLiNER2.5-Decide-1B`	local, RTX 4090 with the gliner2 library	none, one task per question, options as labels	2026-10-08	1
Reference, English Opus judge	general language model as judge	`claude-opus-5-5`	blind, without model answers	choice and short reason	2026-10-01	one judgment per line

Winnow-12B and Shisa DE-1 ran in our setup, not in their own format. A different start of the answer alone changed 6 of 15 answers for Shisa DE-1. Their figures hold for this setup.

All prompts word for word, on the line from the opening figure (p01-en-l03), question about the level

Jev · POST /v1/systemone

{
  "model": "jev-latest",
  "state": {
    "requirement": "Build end-to-end test automation with Playwright and TypeScript for our customer portal",
    "cv": {
      "x-profile": "Senior Test Automation Engineer (SDET) for web, API and CI/CD. Eleven years in quality assurance, eight of them in test automation. Builds maintainable end-to-end and API test suites, wires them into pipelines and teaches teams to own their tests.. B.Sc. Computer Science, Hochschule Bremen, 2012 - 2015. German (native), English (fluent)",
      "s-e-commerce-sdet": "Senior SDET (freelance), Tidewave Commerce GmbH (03/2022 - present): - Built an end-to-end suite with Playwright and TypeScript for a web shop: 1,400 tests, run in Azure DevOps pipelines on every pull request API tests for the REST services (orders, payment, inventory) with Playwright and contract tests with Pact Cut the flaky test rate from 9% to under 1% through test isolation and stable test data Coached three development teams on the test pyramid and code review for tests Test reports for product owners in Azure DevOps dashboards",
      "s-insurer-automation": "Test Automation Engineer (freelance), Hanse Versicherungsdienste AG (06/2019 - 02/2022): - Migrated 600 Selenium tests (Java) to Playwright, runtime from 3 hours to 25 minutes BDD scenarios with Cucumber, written together with the business unit API tests with Postman and Newman in the Azure DevOps pipeline Set up test data management for contract and claims scenarios",
      "s-telecom-qa": "QA Engineer, Kombitel Services GmbH (09/2016 - 05/2019): - Selenium WebDriver tests in Java with TestNG for the customer portal Mobile tests with Appium for the Android and iOS apps Jenkins pipelines for nightly regression runs Defect triage together with development in Jira",
      "s-agency-tester": "Software Tester, Pixelfabrik Medien GmbH (10/2014 - 08/2016): - Manual and exploratory testing of websites and web applications for agency clients First Selenium IDE and WebDriver scripts for smoke tests Test documentation in Confluence, bug tracking in Jira",
      "p-llm-testgen": "Test case drafts with an LLM (2024): Internal tool that turns user stories into Playwright test drafts through an LLM API. A tester reviews every draft before it is merged. Used by two teams as a time saver for new features. Stack: TypeScript, Playwright, Azure OpenAI.",
      "p-flaky-radar": "Failure classification dashboard (2023): Collects test results from pipelines and groups failures by cause (test data, timing, environment, real defect). Helps the team decide which failures to look at first. Stack: Python, pytest, Grafana."
    }
  },
  "questions": {
    "is_req": {
      "type": "noul",
      "instructions": "Does `requirement` describe something the candidate must know, have done or will do (a skill, experience, qualification or task), rather than describing the company, its benefits or the application process?"
    },
    "must": {
      "type": "noul",
      "instructions": "Is `requirement` phrased as mandatory rather than optional or nice to have?"
    },
    "axis": {
      "type": "choice",
      "instructions": "Which area does `requirement` belong to?",
      "criteria": {
        "testauto": "test automation: automated UI, API and end-to-end tests with frameworks such as Playwright, Selenium, pytest or BDD",
        "perf": "load and performance testing: JMeter, Gatling, LoadRunner, performance analysis",
        "qm": "quality strategy and test management: test strategy, test process, test planning, acceptance, defect management",
        "ai": "AI and LLM engineering: building with large language models, RAG, prompt design, AI-assisted testing or evaluation",
        "agents": "AI agents and orchestration: multi-agent systems, agentic coding tools, MCP, A2A, tool use",
        "cloud": "cloud, DevOps and CI/CD: Azure or AWS, containers, pipelines, infrastructure, monitoring",
        "dev": "software development and integration: production code in Python, TypeScript, Java or Kotlin, APIs and system interfaces",
        "lead": "consulting and leadership: advising clients, workshops, coaching, leading or building teams, working with stakeholders",
        "other": "None of these areas"
      }
    },
    "evidence": {
      "type": "choice",
      "instructions": "Which entry of `cv` best shows that the candidate meets `requirement`?",
      "criteria": {
        "x-profile": "`cv.x-profile`",
        "s-e-commerce-sdet": "`cv.s-e-commerce-sdet`",
        "s-insurer-automation": "`cv.s-insurer-automation`",
        "s-telecom-qa": "`cv.s-telecom-qa`",
        "s-agency-tester": "`cv.s-agency-tester`",
        "p-llm-testgen": "`cv.p-llm-testgen`",
        "p-flaky-radar": "`cv.p-flaky-radar`",
        "none": "No entry of `cv` shows it"
      }
    },
    "level": {
      "type": "score",
      "instructions": "How well does `cv` show that the candidate has done what `requirement` asks for?",
      "criteria": [
        "Not at all",
        "Adjacent or transferable experience",
        "Done in at least one role or project",
        "Done repeatedly, across several roles or projects"
      ]
    }
  }
}

Winnow-12B and Shisa DE-1 · template gemma4

<|turn>system
You judge a project posting and a candidate CV. Answer each multiple-choice question with the letter of exactly one option.<turn|>
<|turn>user
`requirement`:
Build end-to-end test automation with Playwright and TypeScript for our customer portal

`cv`:
cv.x-profile: Senior Test Automation Engineer (SDET) for web, API and CI/CD. Eleven years in quality assurance, eight of them in test automation. Builds maintainable end-to-end and API test suites, wires them into pipelines and teaches teams to own their tests.. B.Sc. Computer Science, Hochschule Bremen, 2012 - 2015. German (native), English (fluent)
cv.s-e-commerce-sdet: Senior SDET (freelance), Tidewave Commerce GmbH (03/2022 - present): - Built an end-to-end suite with Playwright and TypeScript for a web shop: 1,400 tests, run in Azure DevOps pipelines on every pull request API tests for the REST services (orders, payment, inventory) with Playwright and contract tests with Pact Cut the flaky test rate from 9% to under 1% through test isolation and stable test data Coached three development teams on the test pyramid and code review for tests Test reports for product owners in Azure DevOps dashboards
cv.s-insurer-automation: Test Automation Engineer (freelance), Hanse Versicherungsdienste AG (06/2019 - 02/2022): - Migrated 600 Selenium tests (Java) to Playwright, runtime from 3 hours to 25 minutes BDD scenarios with Cucumber, written together with the business unit API tests with Postman and Newman in the Azure DevOps pipeline Set up test data management for contract and claims scenarios
cv.s-telecom-qa: QA Engineer, Kombitel Services GmbH (09/2016 - 05/2019): - Selenium WebDriver tests in Java with TestNG for the customer portal Mobile tests with Appium for the Android and iOS apps Jenkins pipelines for nightly regression runs Defect triage together with development in Jira
cv.s-agency-tester: Software Tester, Pixelfabrik Medien GmbH (10/2014 - 08/2016): - Manual and exploratory testing of websites and web applications for agency clients First Selenium IDE and WebDriver scripts for smoke tests Test documentation in Confluence, bug tracking in Jira
cv.p-llm-testgen: Test case drafts with an LLM (2024): Internal tool that turns user stories into Playwright test drafts through an LLM API. A tester reviews every draft before it is merged. Used by two teams as a time saver for new features. Stack: TypeScript, Playwright, Azure OpenAI.
cv.p-flaky-radar: Failure classification dashboard (2023): Collects test results from pipelines and groups failures by cause (test data, timing, environment, real defect). Helps the team decide which failures to look at first. Stack: Python, pytest, Grafana.

Question: How well does `cv` show that the candidate has done what `requirement` asks for?
A) Not at all
B) Adjacent or transferable experience
C) Done in at least one role or project
D) Done repeatedly, across several roles or projects<turn|>
<|turn>model
<|channel>thought
<channel|>

Qwen3.8-27B and Qwen3-32B · template chatml

<|im_start|>system
You judge a project posting and a candidate CV. Answer each multiple-choice question with the letter of exactly one option.<|im_end|>
<|im_start|>user
`requirement`:
Build end-to-end test automation with Playwright and TypeScript for our customer portal

`cv`:
cv.x-profile: Senior Test Automation Engineer (SDET) for web, API and CI/CD. Eleven years in quality assurance, eight of them in test automation. Builds maintainable end-to-end and API test suites, wires them into pipelines and teaches teams to own their tests.. B.Sc. Computer Science, Hochschule Bremen, 2012 - 2015. German (native), English (fluent)
cv.s-e-commerce-sdet: Senior SDET (freelance), Tidewave Commerce GmbH (03/2022 - present): - Built an end-to-end suite with Playwright and TypeScript for a web shop: 1,400 tests, run in Azure DevOps pipelines on every pull request API tests for the REST services (orders, payment, inventory) with Playwright and contract tests with Pact Cut the flaky test rate from 9% to under 1% through test isolation and stable test data Coached three development teams on the test pyramid and code review for tests Test reports for product owners in Azure DevOps dashboards
cv.s-insurer-automation: Test Automation Engineer (freelance), Hanse Versicherungsdienste AG (06/2019 - 02/2022): - Migrated 600 Selenium tests (Java) to Playwright, runtime from 3 hours to 25 minutes BDD scenarios with Cucumber, written together with the business unit API tests with Postman and Newman in the Azure DevOps pipeline Set up test data management for contract and claims scenarios
cv.s-telecom-qa: QA Engineer, Kombitel Services GmbH (09/2016 - 05/2019): - Selenium WebDriver tests in Java with TestNG for the customer portal Mobile tests with Appium for the Android and iOS apps Jenkins pipelines for nightly regression runs Defect triage together with development in Jira
cv.s-agency-tester: Software Tester, Pixelfabrik Medien GmbH (10/2014 - 08/2016): - Manual and exploratory testing of websites and web applications for agency clients First Selenium IDE and WebDriver scripts for smoke tests Test documentation in Confluence, bug tracking in Jira
cv.p-llm-testgen: Test case drafts with an LLM (2024): Internal tool that turns user stories into Playwright test drafts through an LLM API. A tester reviews every draft before it is merged. Used by two teams as a time saver for new features. Stack: TypeScript, Playwright, Azure OpenAI.
cv.p-flaky-radar: Failure classification dashboard (2023): Collects test results from pipelines and groups failures by cause (test data, timing, environment, real defect). Helps the team decide which failures to look at first. Stack: Python, pytest, Grafana.

Question: How well does `cv` show that the candidate has done what `requirement` asks for?
A) Not at all
B) Adjacent or transferable experience
C) Done in at least one role or project
D) Done repeatedly, across several roles or projects<|im_end|>
<|im_start|>assistant
<think>

</think>

Answer: **

Source: test bench lab-bench, run 1 per model on 2026-10-01 and 2026-10-08, scored by the test bench itself. Local language models through Ollama, GLiNER2.5-Decide-1B through gliner2, all on an RTX 4090. Cases and CVs are invented.

How the setup works The case player shows each model's distribution over the options. For the local models it also shows how much probability sat on the letters before dividing. The table counts all lines against the English Opus judge. The net below puts two models of your choice side by side on seven axes, plus speed. It counts the English lines against the same judge, and the German lines against the adjudicated reference, like the article “Jev against local models”.

Scope and limitations

Not part of this demo: The reference is the English Opus judge, see the note in the experiment. The cases are invented. The counts use run 1 of each model. Jev ran five times, Qwen3-32B twice, the other models once. The local language models ran in our setup through Ollama, Winnow-12B and Shisa DE-1 too, although both come with a format of their own. GLiNER2.5-Decide-1B ran on 2026-10-08 through the gliner2 library. Their figures depend on that setup. How the probabilities come about is shown in the opening figure.

A script picks the cases by five patterns from all lines of the test bench, not by hand. The table matches the test bench report, and the export script checks that.

Experiment 6Test bench pv1 · the project match's arithmetic

From single judgments to one total score

A model that writes “80 percent fit” cannot be checked. Ask instead per line about requirement, mandatory, area and level, and code you can read computes the total.

Experiment 6 · From single judgments to one total score

10 pairs, answers from the decision model Jev jev-1.13.0 through the TypeSafe API, run 1 on 2026-10-01. No live request.

One pair's path to the total

Pair p02: Test Automation Engineer, career changer from teaching, posting Freelance: Senior Test Automation Engineer with an AI focus

1 Lines and Jev's answers
14 lines, five questions each to Jev
Show lines with Jev's number for yes
1. 0.95 Freelance: Senior Test Automation Engineer with an AI focus
2. 0.49 Utility company, customer portal, start now, 6 months, mostly remote
3. 0.97 Build end-to-end test automation with Playwright and TypeScript for our customer portal
4. 0.97 Automated API tests for the REST interfaces to billing and contract management
5. 0.97 Integrate the test runs into Azure DevOps pipelines, including reports for the team
6. 0.96 Use AI to generate and maintain test cases from requirements
7. 0.96 Analyse test failures and incidents, also with the help of LLMs
8. 0.97 Coach the development team on test strategy and the test pyramid
9. 0.98 At least 5 years of experience automating tests for web applications
10. 0.98 Very good knowledge of Playwright or comparable frameworks
11. 0.98 Experience with CI/CD, ideally Azure DevOps
12. 0.97 Hands-on experience with LLMs in quality assurance
13. 0.94 Load testing skills are a plus
14. 0.96 Very good German
From here on real code, every number can be recomputed
2 Gate: does the line count?
Counting threshold (fixed at 0.50 in the project match)0.50
13 of 14 lines count as requirements
The same wording as case B in the opening figure, here with a different CV: 0.49 for yes instead of 0.48. Below the counting threshold the line drops out.
3 Mandatory counts more
Weight of a mandatory line2.0
11 of 13 counted lines are mandatory and count 2.0 times
4 Coverage and topic fit
coverage 0.564 (weighted mean of levels divided by 3), topic fit 0.480
5 Total
Share of requirements in the total0.70
The rest is topic fit: how well the areas the posting stresses are covered.
(0.70 × 0.564 + 0.30 × 0.480) × 100 = 54

Ranking by total

share from requirements
share from topic fit
contradicts the author's intent

Order as intended

33 of 33

comparisons between pairs with different intent

Can you break the order? So far the order is as intended.

Two solutions:

Intent of the author of the invented cases for p02, not a reference judgment: Playwright, TypeScript and API tests are there, but only 1.5 years instead of 5, no LLM use in QA and little CI/CD depth.

Total 54. On the right all ten pairs. Turn the knobs and see whether the order follows the intent.

The intent comes from the author of the invented cases and is not a reference judgment. It only shows whether the calculation roughly hits the intended order.

Source: test bench lab-bench, Jev run 1 on 2026-10-01. Calculation: the same formula as the project match (bench/match.ts scoreMatch). As in the test bench report, levels here are the expected value of Jev's rounded bars. The project match uses Jev's own score, which can differ by about 0.01.

How the setup works On the left the path of many small judgments to the total, with three knobs: share of requirements, weight of a mandatory line, counting threshold. On the right the 10 pairs by total, each bar split into requirements and topic fit.

Scope and limitations

Not part of this demo: The author's intent is not a reference judgment. The calculation uses Jev's answers from one run. Over Jev's four runs in the same order, a pair's total varies by less than one point according to the test bench report (standard deviation).

The calculation is the same as in the project match. Requirements count at a probability of 0.5 or higher, mandatory lines with the chosen weight, the level as the expected value over the four levels. The rest is topic fit by area.

What it is good for

Where does it hold, where not? Guess first, then measure

Every figure comes from our own measurement. We measured Jev jev-1.13.0, a decision model via API, on 2026-09-19. We ran Claude Haiku and Claude Opus, both general language models, as subagents on 2026-09-19: each got a task in plain language and answered in text. Their exact version was not logged. Laya 0.3.5, an open decision model: it knows the same three question types as Jev, writes no text and returns a probability per option. Technically it is a model that only reads text, with a last layer that outputs the probabilities directly (an encoder with a decision head). It ran locally on an RTX 4090 and was measured on 2026-09-22. The figures for the project match come from the test bench, measured on 2026-10-08. The methods are in the sources under further reading.

A decision model chooses between options. Counting, arithmetic or extraction belong in code or in a different question shape. TypeSafe itself names counting and arithmetic as weak spots, the other points come from our own measurements.

Counting
ChoiceJevcode reads the number
Your guess: does it hold here, or is it weaker?
measured, weaker
Jev65.0%
ask per entry, code counts
direct65.0%
with code95.0%
Asked directly how many entries a list has, Jev is right in 65.0% of 20 cases. Ask per entry and count in code, and it is 95.0%. Source: “Jev in practice”.
Routing requests to the right tool
ChoiceJevcode takes the first choice
Your guess: does it hold here, or is it weaker?
measured, holds
Jev84.3%
Claude Haiku 82.7%Claude Opus 94.3%
We did not measure a variant with code here.
Over all 561 requests, including those without a fitting skill, Jev picks the fitting skill in 84.3% of cases. Claude Haiku reaches 82.7%, Claude Opus 94.3%. The gap to Haiku is not significant (p = 0.38), but the gap to Opus is. For us, Jev does not pay off as a router in front of a strong model. It fits pre-sorting with a tuned threshold better. Source: “Jev in practice”.
Arithmetic close to a threshold
ChoiceJevcode reads the answer
Your guess: does it hold here, or is it weaker?
measured, weaker
Jev66.7%
We did not measure a variant with code here.
When an amount sits just above or below a threshold, Jev decides correctly in 66.7% of 15 cases. Arithmetic belongs in code. Source: “Jev in practice”.
Pre-screening attacks on language models
Yes or noJevcode draws a line
Your guess: does it hold here, or is it weaker?
measured, holds
Jev75.0%
set the threshold on one half, check on the other
direct75.0%
with code89.7%
On 116 texts labeled by people, Jev is right in 75.0% of cases at a threshold of 0.5. With a threshold set on one half of the data and tested on the other, it reaches 89.7%. Experiment 2 shows why a threshold set on the same data looks too good. The threshold belongs to the application, not to the model.
Due date among several dates
ChoiceJevcode reads the date
Your guess: does it hold here, or is it weaker?
measured, weaker
Jev83.3%
Claude Haiku 100.0%Claude Opus 100.0%Laya 15.0%
We did not measure a variant with code here.
Jev finds the amount payable in 100.0% of cases, but the due date among several dates only in 83.3%. Claude Haiku and Claude Opus reached 100.0% and 100.0% here, Laya as a router 15.0%. Sources: “Jev in practice” and “Laya against Jev”.
Classifying emails and claims
ChoiceJevcode sorts
Your guess: does it hold here, or is it weaker?
measured, holds
Jev92.2%
Claude Haiku 87.0%Claude Opus 100.0%
We did not measure a variant with code here.
Emails into seven categories on 77 cases: Jev 92.2% right, Claude Haiku 87.0%, Claude Opus 100.0%. Checking a claim against its source on 74 cases: Jev 97.3%, Haiku 98.6%, Opus 100.0%. According to the article, this second task does not separate the models reliably. Both tasks have fixed options, the shape a decision model is built for. Source: “Jev in practice”.
Many options for small local models
ChoiceJevcode takes the choice
Your guess: does it hold here, or is it weaker?
measured, weaker
Jev88.8%
Laya 5.5%
We did not measure a variant with code here.
Over the 490 requests with a fitting skill, Jev picks the right one from all 117 skills at once in 88.8% of cases, Laya only in 5.5%. This figure counts only the first pass, over the 490 requests with a fitting skill. The article “Jev in practice” scores the same requests differently and gives Jev the right skill in 82.9% of them, and 84.3% over all 561 requests. The figures are not directly comparable. The weak spot is Laya's: small local models need a preselection, and that has to be good itself. Source: “Laya against Jev”.
CV against posting
many small questions per lineJevcode computes the total
Your guess: does it hold here, or is it weaker?
measured, holds
Here code already does the calculation: Experiment 6 shows how the total comes about. →
Many small questions per line, the arithmetic in code. In assigning the area Jev is ahead: 91.6% against 71.4% to 88.2% for the local language models, significantly against 2 of 4. Jev picks the same evidence as the English Opus judge in 79.0% of lines, and the local language models are close with no significant gap. On lines without evidence Jev still names an entry in 27.3%, Qwen3.8-27B significantly less often. Jev judges whether a line is mandatory the same way as the English Opus judge in 89.1% of lines. Significantly ahead of Jev: Qwen3-32B (100.0%), Qwen3.8-27B (97.5%) and Winnow-12B (97.5%). Significantly behind: Shisa DE-1 (63.0%). More in Experiment 3.

Where the model chooses between fixed options, it holds. Counting, arithmetic and extraction belong in code or in another question shape. The question shape decides.

Limits

What we measured and what we did not

This page shows recorded measurements. Here we list what they support and what they do not.

01
Jev runs in the USA
Jev is a TypeSafe service and, according to its privacy policy, runs in the USA. The data processing agreement incorporates the EU standard contractual clauses. According to the privacy policy, TypeSafe does not train on inputs. TypeSafe waives retention only under the enterprise contract. Whether customer data may go there is the customer's decision. The documents are listed under further reading.
02
Replay, not a live request
The page plays back stored answers. No request to a model leaves your browser. You cannot ask your own questions here yet.
03
Invented cases, the judges are models
Postings and CVs in the test bench lab-bench are invented. The test bench lives in one public repository, bks-lab/lab-bench, including Jev's answers per question, published with TypeSafe's permission of 10 October 2026. The reference is the Opus judges from Experiment 3.
04
One run per measurement
Almost every answer shown comes from a single run. On the test bench Jev ran several times, and individual answers flip between runs. Experiment 5 shows how often.
05
Open models you run yourself
We measured Laya. The result is in the article “Laya against Jev”. On 2026-10-01 four open language models ran on the test bench, locally on an RTX 4090 through Ollama. Third parties fine-tuned Winnow-12B and Shisa DE-1 for decisions. Qwen3.8-27B and Qwen3-32B are general language models. Their figures are in Experiment 3, measured against the English Opus judge.
06
A classifier against Jev
On 2026-10-08 GLiNER2.5-Decide-1B by Fastino joined. Fastino puts Jev into a comparison table of its own on its model card. GLiNER is not a language model but a classifier: it reads a text and spreads the probability over fixed labels, without writing text. Of nine variants the best one is in the table of Experiment 3. Averaged over the five line questions it reaches 52.3%, Jev 84.6%. Simply giving every line the judge's most frequent answer reaches 53.4%. GLiNER never chooses “no evidence”, so on lines without evidence it always names an entry (100.0%). The area is the question GLiNER is built for, sorting a short text into a topic. There it reaches 62.2%, Jev 91.6%. The other eight variants and the method are in the test bench under gliner.md.
07
Speed and cost: Jev leads
Jev needs about 0.25 seconds per line, the local language models 1.6 to 2.5 seconds, one after the other on an RTX 4090. At list price 1,000 lines cost about 7 US cents with Jev. Locally the electricity alone costs roughly 7 to 11 euro cents, estimated from the measured compute time and the card's maximum power draw, without hardware. So the experiments compare the quality of the single judgments, because that is where a local model can keep up or not. The calculation is in the test bench under speed-cost.md.
08
What we did not measure
Real documents from our own work, load tests, cost in continuous operation and the correctness of the project match. Calibration of the local models rests on one run in Experiment 4. Locally the probabilities vary between two runs: a second run of Qwen3-32B on two pairs of the test bench made the same choices, but with shifted probabilities. Reliable calibration needs repeated runs. Hosted endpoints in the EU for the open models were not measured. The test bench has ten invented pairs, and according to a paired McNemar test many small gaps in Experiment 3 do not hold up. The e-invoice measurement runs on public test data.
09
Origin and license
The values in Experiment 2 come from deepset/prompt-injections (Apache-2.0 license) and from our own routing requests. We do not publish the texts of the routing requests or skill names. The page carries only the probability and the dataset label for each judgment.

What we measured and wrote about this

The figures on this page come from measurements we describe in detail elsewhere.

01
Jev in practice
The report behind Experiments 2 and 4: skill routing, calibration and a series of small tasks, with both addenda, on calibration (22 September) and on e-invoices (23 September).
Read the article →
02
Laya against Jev
An open model you run yourself, on the same tasks. Replacing Jev with a local model means rebuilding the workflow.
Read the article →
03
Jev reads e-invoices
Our measurement on public KoSIT test invoices: several yes or no questions per invoice in one request, every judgment open to inspection. Listed under “Other areas of work” at the end of the page.
04
Test bench lab-bench
Our test bench for the project match: invented postings and CVs, Jev next to local models, reference from Opus judges. It lives in one public repository, https://github.com/bks-lab/lab-bench, including Jev's answers per question, published with TypeSafe's permission of 10 October 2026. Usage notice for them: https://github.com/bks-lab/lab-bench/blob/main/LICENSE-JEV-OUTPUTS
05
TypeSafe
TypeSafe documents models, question types, the interface, prices and the known weak spots, and outlines its training method RLCD (Reinforcement Learning for Calibrated Decisions). It also publishes its privacy policy, its data processing agreement and an open adapter that puts language models in Jev's place.
TypeSafe model overview ↗Interface /v1/systemone ↗Introduction with RLCD ↗Announcement of Jev ↗Adapter for language models ↗Jev's weak spots according to TypeSafe ↗Privacy policy ↗Data processing ↗
06
The models next to Jev
The model cards of the four open language models from the test bench. Shisa DE-1 describes its training and answer format there, Winnow-12B its training. The code of its server winnow-inference (native/protocol.h) shows how Winnow-12B expects to be asked.
Winnow-12B ↗Server winnow-inference ↗Shisa DE-1 ↗Qwen3.8-27B ↗Qwen3-32B ↗
07
deepset/prompt-injections
The public dataset behind Experiment 2, German and English texts, labeled by people.
Dataset on Hugging Face ↗

Why we show this

Anyone who builds a language model with a prompt and a parser into a workflow often also decides with a threshold. It sits hidden in the parser. A decision model puts that spot in the open. Here you can try it, on measured data.

Other areas of work

BKS open-bridge · Agent contextIn daily own use · Open source (MIT)
Jev reads e-invoices · Field and textOwn measurement setup · public test data
Test bench · Tools comparedOwn measurements · public repository
Automating processesTwo delivered reference workflows · one comparison model
Making operations understandableTested in our own operations
k2a · Knowledge toolIn-house development
Making knowledge usableProduct concept · k2a in closed trial
Orchestrating agentsStructural diagram · not a specific deployment

Back to the Labor overview →

Where does a hidden threshold sit in your workflows?

Bring a workflow in which a person or a prompt decides today. Together we look at which questions can be asked with a fixed type and where the threshold should sit.

Schedule a conversation

Decision models: typed questions, answers with probabilities

One question, three shapes: the question type decides the shape of the answer

Where does the code draw the line?

Who matches the judge?

How sure is sure?

Shuffle the options

From single judgments to one total score

Counting

Routing requests to the right tool

Arithmetic close to a threshold

Pre-screening attacks on language models

Due date among several dates

Classifying emails and claims

Many options for small local models

CV against posting

Jev runs in the USA

Replay, not a live request

Invented cases, the judges are models

One run per measurement

Open models you run yourself

A classifier against Jev

Speed and cost: Jev leads

What we did not measure

Origin and license

Jev in practice

Laya against Jev

Jev reads e-invoices

Test bench lab-bench

TypeSafe

The models next to Jev

deepset/prompt-injections

Where does a hidden threshold sit in your workflows?