When a client asks us whether an open model can take over one of their tasks, we want to answer with a measurement. Not with an impression from a demo, and not with the vendor’s own table. That is why we built a test bench, bks-lab/lab-bench on GitHub. As of today, 10 October 2026, its results are also on Hugging Face.

The question behind the bench

Our focus is sovereign AI: the data stays on hardware the company controls. That promise depends on one plain question. Which tasks can an open model handle on such hardware?

We kept the question deliberately narrow. One machine, one graphics card, an RTX 4090 with 24 GB of memory. That is a workstation a mid-sized company without a data centre can put under a desk. We can promise a client whatever runs on this card. If something only runs with the model offloaded to system memory, or does not finish at all, we say so just as clearly.

How we measure

For the capability tests this post is mostly about, the rules are in plan/ROADMAP.md and in one test plan per task. The main ones:

  • The plan is fixed before the run. Data, models, metric, baseline and thresholds are committed to the repository before the first model runs. The test cannot be fitted to the results afterwards.
  • Public datasets with a checked licence. We check every model’s licence up front as well. Models restricted to non-commercial use were dropped, and every row of the result table states the licence class.
  • Automatic metrics. Nobody grades answers by feel. Everything is counted against fixed gold values, the correct answers stored in the datasets.
  • An interval with every number. Each value carries a 95 % bootstrap interval. That is the range in which the value would likely fall on another sample from the same data. If the intervals of two models overlap and the paired test does not separate them either, we do not call them different.

One rule matters a great deal to me. I call it the reference value rule: no model is scored before the bench has reproduced a published number. For German retrieval, bge-m3 first had to hit its value from the public MTEB table. We got 0.5767, the published value is 0.5759. Only then did the other models run. If the reference does not match, the setup is wrong and every further number is worthless. For invoices there is no published value, so the bench first checks that every gold value appears in the rendered document. All 391 of 391 did.

What came out

So far we have measured the three tasks at the top of our plan for clients: search over a company’s own documents, speech recognition for meeting minutes, and invoice fields. All three passed their check under the reference value rule.

Task Best model Value Comparison
German retrieval (nDCG@10, MIRACL de) Qwen3-Embedding-8B with bge-reranker-v2-m3 0.640 BM25 0.199
German speech to text (word error rate, FLEURS de) canary-1b-v2 and whisper-large-v3 4.52 % and 4.70 % not separated
Invoice fields from page images (34 invoices) gemma4-26b and gemma4-12b 98.0 % and 97.8 % regex over the PDF text 27.0 %

Retrieval. nDCG@10 measures how well the first ten hits are ranked, 1 would be perfect. Qwen3-Embedding-8B alone reaches 0.613 and is separated from every other embedding model. A reranker that reorders the top 100 hits lifts that to 0.640. The cost is time. The large model indexes 44 passages per second, bge-m3 and e5-large about 1,200 at an nDCG@10 of 0.56 to 0.58. For a large collection that changes daily, a smaller model such as bge-m3 is the practical choice. BM25, classic keyword search, trails at 0.199. That is a property of the dataset, which deliberately contains wrong passages that share the query’s words. It says nothing about keyword search over company documents.

Speech to text. The word error rate is the share of words recognised wrongly. canary-1b-v2 is at 4.52 %, whisper-large-v3 at 4.70 %, and the test does not separate them. In that case the overview shows the model with less GPU memory, here whisper-large-v3 at 8.9 GB. One finding outside the plan matters more for meetings than the headline. On the 21 recordings longer than 30 seconds, whisper-large-v3 rises to 17.6 % while canary-1b-v2 stays at 3.2 %. The group is small, but meetings are long.

Invoice fields. We rendered 34 test invoices from KoSIT, the German coordination office for IT standards, as page images. For each invoice the model reads twelve fields: invoice number, issue and due date, seller and buyer, seller VAT id, IBAN, four amounts and the currency. The gold values come from each invoice’s XML, so nobody had to type them in. gemma4-12b reads 399 of 408 fields correctly, gemma4-26b 400. The smaller model needs 9.8 GB and 2.1 seconds per invoice. A regular expression over the PDF’s text gets 27.0 %.

One number surprised us. A text model, Qwen3.8-27B, got the clean text layer of the PDF instead of the image and reached only about 66 %. The raw rows show why. On all 34 invoices it answered null for all four amount fields, while it got 268 of its other 272 fields right. The instruction asks for amounts as numbers, the schema only allows a string or null, and the model picked null. Without the raw rows we would have drawn the wrong conclusion from 66 %. That is why we publish them.

Measured since: four more capabilities

Updated on 12 October 2026. Between 10 and 12 October 2026 four more tests ran on the same card: tool calls, reading comprehension, long context and throughput. All values come from the findings in the bench at commit 08331c4. The rows are also in the lab-bench-results dataset on Hugging Face, config capabilities.

Task Best model Value Comparison
Tool calls (BFCL v3, 1,000 tasks) qwen3-14b and qwen3.8-27b 91.9 % and 90.6 % not separated
Reading comprehension (F1, GermanQuAD, 1,000 questions) gelectra-large 0.872 best language model 0.849
Long context (accuracy at 32k tokens, 180 tasks) qwen3-14b, qwen3.8-27b, gpt-oss-20b 100 % each all seven models at least 90 %
Throughput (decisions per hour, 16 users) granite4-3b 260,731 qwen3-8b 190,414

Tool calls. A model gets a description of functions and has to call the right one with the right arguments, through Ollama’s built-in tool interface. qwen3-14b gets 91.9 % of the 1,000 tasks right, qwen3.8-27b 90.6 %. The test does not separate the two (p 0.22), but qwen3-14b needs 10.3 instead of 18.1 to 22.3 GB. The small granite4-3b reaches 86.9 % with 3.1 GB. Two findings matter more for real use than the ranking. With the functions as JSON in the system prompt, five of seven models do better than through the tool interface (qwen3.8-27b was not measured in prompt mode), qwen3-14b for example with 94.6 %. And gpt-oss-20b fails through the Ollama integration: it returns at most one call per task and ends at 50.8 %. When Ollama enforces a JSON schema during generation, all eight models keep to it in 89.5 % to 93.4 % of cases, without enforcement only in 73.3 % to 90.5 %. A check on the model’s output stays necessary. We first checked the setup against the BFCL leaderboard: 93.5 % on simple calls against 95.25 % for Qwen3-14B, with five points of tolerance fixed in advance. The leaderboard ran with thinking on and full precision, we ran with thinking off and Q4_K_M. Details in the C4 findings.

Reading comprehension. The model gets a German Wikipedia passage and a question and has to give the answer from the text. The winner is not a language model but gelectra-large, a small reader model with 335 million parameters: F1 0.872 against 0.849 for gemma4-12b, the best language model (p 0.024). It needs 1.7 GB and 28 milliseconds per question, gemma4-12b 9.2 GB and 302 milliseconds. The comparison is uneven, though. The reader was trained on the training split of the same dataset, the language models saw no example. Size did not help: gemma4-26b reaches 0.778, and qwen3.8-27b wrote “keine Antwort” (no answer) for 104 questions whose answer was in the passage. Details in the C6 findings.

Long context. We hide a fact in a long German text and ask for it, at 4k, 16k and 32k tokens (about 4,000, 16,000 and 32,000). All seven models are at 90 % or above at every length. qwen3-14b, qwen3.8-27b and gpt-oss-20b answer all 180 tasks at 32k tokens correctly. gpt-oss-20b reads such a text in 2.7 seconds, qwen3-14b needs 8.1. The Gemma and Mistral misses show no weakness at finding. Ollama cut 97 of 3,780 inputs to about half without an error, because the text in the model’s own tokens was longer than the configured context window. They answer every uncut input correctly. In production the context window must be larger than the longest input, counted in the model’s tokens. We did not test texts longer than 32k tokens. Details in the C7 findings.

Throughput. How much can the card handle when several users ask at once? With 16 parallel requests granite4-3b makes 260,731 short decisions per hour, qwen3-8b 190,414. For a single user gemma4-26b is among the fastest at 178.3 tokens per second, because as a mixture of experts it uses only about 4 billion parameters per token. At 16 users the picture turns. Each request needs a cache for the text so far, and for some models 16 of them no longer fit next to the weights. gemma4-12b then drops from 397.0 tokens per second at 8 users to 146.2 at 16, because Ollama moves parts to the CPU. qwen3.8-27b ran with a single slot and serves 16 users one after another. Against vLLM, a server built for many parallel requests, Ollama loses at 16 users: qwen3-14b reaches 789.5 instead of 564.0 tokens per second under vLLM, gemma4-12b 708.8 instead of 146.2. With one user Ollama is faster, 85.6 against 79.6 for qwen3-14b. Each ran its own 4-bit build of the weights (for gemma4-12b under vLLM a community conversion), so the comparison covers server and build together. On energy use granite4-3b leads with 165 joules per 1,000 generated tokens, measured at the graphics card with 16 users. This test measures speed and power only, not quality. Details in the C8 findings.

Measured since: translation and three task tests

Updated on 12 October 2026. On the same day translation (C5) and three tests on the decision models were added: stability under shuffled options (T2), robustness against typos (T9) and throughput with batching (T10). All values come from the findings in the bench at commit cfd07a2.

Translation. We measure chrF++ on 1,012 sentences from FLORES+, in both directions, on a scale from 0 to 100. German to English, gemma4-26b reaches 68.1 and gemma4-12b 67.9, English to German 65.5 and 65.1. The paired test does not separate the two (p 0.060 and 0.053). Our pick is therefore the smaller gemma4-12b, with 8.7 instead of 18.5 GB of GPU memory. eurollm-9b follows with 67.3 and 64.8. opus-mt, built for translation only, trails by three to four points at 64.9 and 61.1, but needs only 1.9 GB and manages about 84 sentences per second in batches of 32 (mean of both directions), against 4.2 for gemma4-12b one at a time. Whether three to four points matter for a large stack depends on the task. The limits: one sentence per call, German and English only, news and encyclopedic text. FLORES+ may have been in the training data of some models. Details in the finding C5.

Stability. The same 5,948 questions on 2,974 German commands run twice, once with the options in a fixed order and once shuffled. For qwen3.8-27b the answer changes on 5.7% of the questions, for winnow-12b on 7.8%, for gemma4-12b on 9.1%. The changes sit mostly in the question with 18 options (10.4%, 13.9% and 14.4%), in the second question they are 1.0%, 1.7% and 3.9%. For winnow-12b and qwen3.8-27b the shuffled version also scores about one point higher, so the order of the options affects accuracy too. GLiNER Decide changes no answer at all (0 of 5,948). We did not check why. Details in the finding T2.

Robustness. 1,000 German commands run in four versions: clean, with typos, without umlauts (ae, oe, ue, ss) and colloquial. Typos cost each of the four tools 3.0 to 4.2 points of accuracy, missing umlauts and colloquial wording at most 1.8. GLiNER Decide changes the most answers under typos (18.2% against 7.1 to 9.2% for the language models) but loses a similar amount net. The disturbances are generated by fixed rules and do not imitate real speech recognition errors, so the numbers are more of a floor. Details in the finding T9.

Batching. GLiNER Decide answers 399.5 decisions per second when requests run in batches of 8, against 69.8 one at a time. That is 5.7 times as many, and the answers stay the same at every batch size. Larger batches are slower: 380.6 at 32 and 354.4 at 128. For context: the classifier is right on 56% of these commands, against 79 to 82% for the language models in T2. We did not re-measure the language models here, their throughput is above under “Throughput”. Details in the finding T10.

Measured since: personal data

Updated on 12 October 2026. Detection of personal data in German text (C9) is measured, with a limit up front: the reference spans of this data set were generated by a language model, and the check of the 40 sample texts was also done by a language model, not by a human. That was Claude Haiku 5.5, and Claude Opus 5.5 cross-checked every objection. Our test design asks for a check by a human. It is still open, and we record the deviation in the design.

The check found 43 wrong spans among 149, for instance “Damen”, “Herren” or “Jetzt” as a person. The precision of the reference is 0.685, below the threshold of 0.90 set beforehand. The test therefore counts as “reference noisy”, all numbers only describe, and we claim no ranking. On the checked reference of the 40 texts gemma4-12b reaches 0.839 and qwen3.8-27b 0.828 (not separated), gliner-pii-domains 0.738, gliner2.5-multi 0.649, gliner-multi-pii 0.444, Presidio 0.300 and the regex baseline 0.150. On all 476 texts with the original reference qwen3.8-27b is ahead with 0.644, gemma4-12b with 0.627. Both ways of scoring show the same groups: the two language models first, then the GLiNER models, Presidio and regex at the end. Presidio reports a lot (precision 0.238), the regex hits fixed formats such as e-mail addresses cleanly but only 8% of all spans. The data set is invented and from finance, the numbers say nothing about real documents. Details in the finding C9.

What a comparison can hide

We covered the decision models at length in Jev against local models. One result from the same series shows why every number here comes with its origin. Sorting short German commands into 18 topics (MASSIVE), Jev by TypeSafe reaches 72.8 % and the local Winnow-12B 71.8 %. The test does not separate them.

A classically trained classifier reaches 86.9 % on the same question. That looks like a clear winner. But it had seen 11,514 labelled examples of exactly these topics beforehand, the other models not one. With ten examples per topic it drops to 70.8 %. So the number measures what labelled data buys, and it is not comparable with the others. Details are in the route1 findings and on the lab page Decision models.

Why we publish it

A recommendation nobody can check is a claim. So GitHub holds everything needed to recompute our numbers: the code, the dated test plans, the cases and the raw rows of every run. Where a dataset is too large or its licence does not allow passing it on, the repository holds the script that fetches it and verifies it by checksum.

On every push, a second script that does not use the scoring code recomputes the headline numbers for speech recognition and invoice fields from the raw rows, with its own normalisation. If anything differs, the check fails. Anyone who doubts the numbers can recompute them.

Why Hugging Face

Hugging Face is where developers and data teams look for models and datasets. The models we measured live there too. A result placed next to the model is more likely to be found than one in a repository nobody knows. As of today, the bks-lab organisation there holds:

I expect the invoice dataset to be the most useful part for others. If you want to test your own model on invoices, take the same page images and compare its output with ours, invoice by invoice. The gold values are exact because they come from the XML.

What we deliberately keep off Hugging Face

Hugging Face is a US company. That is fine for public test data and results, and not for anything else. Three decisions follow from that.

No live demo with uploads. The invoice viewer runs no model and accepts no file. It only shows stored outputs from our runs. No real invoice ends up on a server outside our control, not even by accident.

No per-question answers from Jev. Jev’s answers to the individual questions are on GitHub only. On 10 October 2026 TypeSafe allowed us to publish these answers so others can reproduce the measurements. The conditions: no training or distilling a model on them, and no building a product that competes with TypeSafe. The wording is in the permission record and the usage notice. A dataset on Hugging Face is a training source, so those rows stay out. Jev’s aggregate scores are benchmark results and are in the result table.

No cases from other people’s datasets. The commands from MASSIVE and the tasks from fast-decisions come from Amazon and Fastino and are already on Hugging Face there, as AmazonScience/massive and fastino/fast-decisions. A second copy with us would only add a second place that can go stale.

How the numbers stay current

To keep the copy on Hugging Face from going stale, a workflow in the repository rebuilds the datasets and Spaces and uploads them whenever the results, the invoice data or the export code change. Every dataset card names the bench commit it was built from. A number found on Hugging Face is one click away from the state of the repository that produced it.

The limits

A test bench that hides its limits is worth no more than a vendor table. These apply to every number above:

  • One run per model. Repeated runs are still missing, and speed was measured once instead of three times. Only the throughput test C8 repeats every measurement three times.
  • 34 invoices give wide intervals. For gemma4-12b, anything between 96.3 % and 99.0 % is consistent with the data.
  • Test invoices are not real scans. They all share one clean layout, without stamps, logos or skew. Real scans will probably score lower, we have not measured that yet.
  • The best model is picked after the fact. Measure many variants and you are more likely to find one that looks good. The p values are not corrected for that.
  • Reading aloud is not a meeting. FLEURS contains sentences read aloud in good recording quality, with no crosstalk.

What comes next

For us the bench is the basis of every model recommendation. Next in the plan we measure long documents, and the check of the personal data reference by a human. Each test gets its plan in the repository first, then its measurement.

Updated on 12 October 2026: tool calls and structured output (C4), reading comprehension (C6), long context (C7) and throughput and energy (C8) are now measured, the results are above under “Measured since”. Translation (C5) and the task tests stability (T2), robustness (T9) and throughput with batching (T10) are added, also above under “Measured since”. Detection of personal data (C9) is added, with the reference checked by a language model, not by a human, see above under “Measured since: personal data”.

If you have a task and want to know whether an open model can handle it on your own hardware, write to us. Ideally with a public dataset we can add to the bench. Then anyone can recompute the answer, you included.

Everything to check

On Hugging Face:

On GitHub:

On bks-lab.com:

All values come from the findings in the bench at commit 1cb5dab of 10 October 2026. Measured on 9 and 10 October 2026 on one RTX 4090 with 24 GB, the decision models on 9 October 2026. The values under “Measured since” come from commit 08331c4 of 12 October 2026, measured from 10 to 12 October 2026 on the same card with Ollama 0.35.0 and vLLM 0.31.0.