{"id":15876,"date":"2026-11-19T15:16:15","date_gmt":"2026-11-19T07:16:15","guid":{"rendered":"https:\/\/ai-stack.ai\/jev-%e6%b1%ba%e7%ad%96%e6%a8%a1%e5%9e%8b-benchmark%ef%bc%9a10-%e7%a8%ae%e6%b1%ba%e7%ad%96%e6%a8%a1%e5%9e%8b-x-%e5%85%a9%e5%a4%a7%e6%83%85%e5%a2%83%e5%af%a6%e6%b8%ac%e8%88%87%e9%81%b8%e5%9e%8b"},"modified":"2026-11-20T16:15:41","modified_gmt":"2026-11-20T08:15:41","slug":"jev-benchmark","status":"publish","type":"post","link":"https:\/\/ai-stack.ai\/en\/jev-benchmark","title":{"rendered":"Jev Decision Model Benchmark: 10 Decision Models \u00d7 2 Scenarios, Test Results and Selection Guide"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">When companies bring LLMs into customer service, internal knowledge bases and AI agent workflows, a &#8220;decision&#8221; usually has to happen before any &#8220;generation&#8221;: Which model should this message go to? Does the content contain personal data? Is it a prompt injection? These decisions sit on the <strong>critical path<\/strong> while the user waits, and every single request passes through them. So before choosing a model, teams usually need to answer the following questions:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>How large is the real <strong>accuracy<\/strong> gap between cloud and on-prem decision models?<\/li>\n\n\n\n<li>Can <strong>a single model<\/strong> handle both &#8220;topic classification&#8221; and &#8220;content safety&#8221;?<\/li>\n\n\n\n<li>If we deploy a <strong>quantized version<\/strong>, will the decisions change?<\/li>\n\n\n\n<li>What happens to the system when a generative LLM guard <strong>returns no verdict<\/strong>?<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">To answer these questions, we used a test set of <strong>5,100 samples, half Chinese and half English<\/strong>, across two scenarios, &#8220;topic classification (Domain)&#8221; and &#8220;content safety (Guard)&#8221;. We tested <strong>10 decision models in 11 deployments<\/strong> (clef was deployed in two ways), with one generative LLM guard (gpt-oss-safeguard) as a control.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>What Is the Jev Decision Model?<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Jev is a <strong>System One decision model<\/strong> from TypeSafe. You send it a piece of text to evaluate (state) and a set of typed questions (questions), and the model <strong>returns the answer and probability distribution for every question in one call, without generating any text<\/strong>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The same \/v1\/systemone interface has other implementations. The clef, clm, laya, winnow and kev families tested here, as well as vllm-sr&#8217;s d3 family, are all compatible; below we refer to them collectively as &#8220;<strong>decision models<\/strong>&#8220;.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>1. Three question types: choice, score, noul<\/strong><\/h4>\n\n\n\n<figure class=\"wp-block-table is-style-stripes has-small-font-size\"><table><tbody><tr><td><strong>Question type<\/strong><\/td><td><strong>Question it answers<\/strong><\/td><td><strong>How it is defined (criteria)<\/strong><\/td><td><strong>What it returns<\/strong><\/td><\/tr><tr><td>choice (classification)<\/td><td>Which category is it?<\/td><td>Object: each option name maps to a description<\/td><td>choice, confidence, probabilities for each option<\/td><\/tr><tr><td>score (graded rating)<\/td><td>How high is the level?<\/td><td>Array: level descriptions ordered from low to high<\/td><td>score, confidence, a legend of levels, probabilities for each level<\/td><\/tr><tr><td>noul (probability)<\/td><td>Yes or no?<\/td><td>Not needed; only instructions<\/td><td>noul: the probability of &#8220;yes&#8221;, between 0 and 1<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>score is the expected value of the levels<\/strong>: each level&#8217;s index is weighted by its probability, so the range is 0 to &#8220;number of levels \u2212 1&#8221;. For example, if three levels have probabilities of 0.67, 0.29 and 0.04, the score is 0.38. The order of the criteria is therefore the direction of the scale, and <strong>they must be listed from low to high<\/strong>.<\/li>\n\n\n\n<li><strong>All three question types can go in one request<\/strong>: you get several independent decisions at once, with no extra network round trips.<\/li>\n\n\n\n<li><strong>This benchmark uses choice only<\/strong>: we confirmed the response formats of score and noul by testing them against jev-latest (reported as jev-1.13.0 in the response).<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>2. One request, several decisions: a worked example<\/strong><\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">The example below evaluates one customer email for its <strong>topic<\/strong> (choice), <strong>urgency<\/strong> (score) and <strong>whether the customer is asking for a refund<\/strong> (noul) at the same time.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Request\n{\n  \"model\": \"jev-latest\",\n  \"state\": \"Hi, I was charged twice for my subscription this month and I need a refund today, otherwise I will cancel.\",\n  \"questions\": {\n    \"topic\": {\n      \"type\": \"choice\",\n      \"instructions\": \"Pick the topic of the message.\",\n      \"criteria\": {\n        \"billing\": \"Payments, invoices, refunds.\",\n        \"technical\": \"Bugs, errors, outages.\",\n        \"sales\": \"Pricing and plans.\"\n      }\n    },\n    \"urgency\": {\n      \"type\": \"score\",\n      \"instructions\": \"Rate how urgent this message is.\",\n      \"criteria\": &#91;\n        \"Not urgent: no time pressure.\",\n        \"Low: wants help this week.\",\n        \"Medium: wants an answer within a day.\",\n        \"High: needs action today, money involved.\",\n        \"Critical: service down or legal threat.\"\n      ]\n    },\n    \"refund\": {\n      \"type\": \"noul\",\n      \"instructions\": \"Is the customer asking for a refund?\"\n    }\n  }\n}\n<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>\nResponse\n{\n  \"model\": \"jev-1.13.0\",\n  \"answers\": {\n    \"topic\": {\n      \"type\": \"choice\",\n      \"choice\": \"billing\",\n      \"confidence\": 1.0,\n      \"probabilities\": {\"billing\": 1.0, \"technical\": 0.0, \"sales\": 0.0}\n    },\n    \"urgency\": {\n      \"type\": \"score\",\n      \"score\": 3.0,\n      \"confidence\": 1.0,\n      \"legend\": {\n        \"0\": \"Not urgent: no time pressure.\",\n        \"1\": \"Low: wants help this week.\",\n        \"2\": \"Medium: wants an answer within a day.\",\n        \"3\": \"High: needs action today, money involved.\",\n        \"4\": \"Critical: service down or legal threat.\"\n      },\n      \"probabilities\": {\"0\": 0.0, \"1\": 0.0, \"2\": 0.0, \"3\": 1.0, \"4\": 0.0}\n    },\n    \"refund\": {\"type\": \"noul\", \"noul\": 0.99}\n  },\n  \"usage\": {\"input_tokens\": 458, \"output_tokens\": 69}\n}\n\n\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Note that confidence is a separate confidence score returned by the service and <strong>is not necessarily equal to the probability of the top option<\/strong> (in this test, laya:multilingual once returned a confidence of 0.9955 with a top probability of 0.997). According to <a href=\"https:\/\/www.datacamp.com\/blog\/system-one-models-jev\" target=\"_blank\" rel=\"noopener\">DataCamp&#8217;s introduction<\/a>, Jev&#8217;s probabilities are calibrated. In our Domain test, jev-latest <strong>got every question right<\/strong> where confidence was \u2265 0.9 (94.9% of questions), which is consistent with that claim.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>3. Decision models vs. generative LLMs: key differences<\/strong><\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">The table below compares how a decision model and a generative LLM guard handle the same task. The data comes from our Guard test. Because the two ran on different hosts, latency is for reference only.<\/p>\n\n\n\n<figure class=\"wp-block-table is-style-stripes has-small-font-size\"><table><tbody><tr><td><strong>Aspect<\/strong><\/td><td><strong>Decision model (systemone)<\/strong><\/td><td><strong>Generative LLM guard (gpt-oss-safeguard:20b)<\/strong><\/td><\/tr><tr><td><strong>Output<\/strong><\/td><td>A typed answer for each question (category, score or probability) + probability distribution<\/td><td>Reasoning + a block of JSON text that you must parse yourself<\/td><\/tr><tr><td><strong>Output tokens (self-hosted)<\/strong><\/td><td>0<\/td><td>327\u2013365 on average<\/td><\/tr><tr><td><strong>Latency mainly depends on<\/strong><\/td><td>Input length (criteria + content)<\/td><td>Output length (correlation with latency 1.00)<\/td><\/tr><tr><td><strong>p50 \/ p95 latency<\/strong><\/td><td>winnow:latest 416 \/ 436 ms<\/td><td>1.58 \/ 6.24 s<\/td><\/tr><tr><td><strong>Can it return no answer?<\/strong><\/td><td>No, every question always gets an answer<\/td><td>Yes, 1.0% (improved prompt) to 6.2% (original prompt)<\/td><\/tr><tr><td><strong>Confidence score<\/strong><\/td><td>Yes, usable for threshold-based routing<\/td><td>None<\/td><\/tr><tr><td><strong>Explainability<\/strong><\/td><td>Probabilities only<\/td><td>Rationale text<\/td><\/tr><tr><td><strong>Changing the decision rules<\/strong><\/td><td>Edit the criteria text, no retraining needed<\/td><td>Edit the policy prompt<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Note: the cloud jev-latest reports output tokens in its usage (about 160 for Guard and 53 for Domain in this test, 69 in the example above), but the response is still a typed answer with no text to parse.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img data-recalc-dims=\"1\" fetchpriority=\"high\" decoding=\"async\" width=\"1024\" height=\"541\" data-attachment-id=\"15907\" data-permalink=\"https:\/\/ai-stack.ai\/en\/jev-benchmark\/image-48\" data-orig-file=\"https:\/\/i0.wp.com\/ai-stack.ai\/wp-content\/uploads\/2026\/11\/90622cbb-edited.png?fit=1600%2C845&amp;quality=100&amp;ct=202603031250&amp;ssl=1\" data-orig-size=\"1600,845\" data-comments-opened=\"0\" data-image-title=\"image\" data-image-description=\"\" data-image-caption=\"\" data-large-file=\"https:\/\/i0.wp.com\/ai-stack.ai\/wp-content\/uploads\/2026\/11\/90622cbb-edited.png?fit=1024%2C541&amp;quality=100&amp;ct=202603031250&amp;ssl=1\" src=\"https:\/\/i0.wp.com\/ai-stack.ai\/wp-content\/uploads\/2026\/11\/90622cbb-edited-1024x541.png?resize=1024%2C541&#038;quality=100&#038;ct=202603031250&#038;ssl=1\" alt=\"\" class=\"wp-image-15907\" style=\"aspect-ratio:1.7777777777777777\"\/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Where Do Decision Models Fit?<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The rule is simple: <strong>if the answer can be listed in advance<\/strong> (a category, a level, or a yes\/no), the task suits a decision model. Tasks that need to produce text or reasons do not.<\/p>\n\n\n\n<figure class=\"wp-block-table is-style-stripes has-small-font-size\"><table><tbody><tr><td><strong>Question type<\/strong><\/td><td><strong>Suitable tasks<\/strong><\/td><td><strong>Examples<\/strong><\/td><\/tr><tr><td>choice (classification)<\/td><td>Classification and routing<\/td><td>Routing questions to the right model or agent by medical, financial or legal topic (the Domain scenario in this article); identifying which type of harmful content a message contains (the Guard scenario in this article); assigning customer service tickets<\/td><\/tr><tr><td>score (graded rating)<\/td><td>Grading and ranking<\/td><td>Complaint urgency, risk level, content rating; scoring the quality of LLM answers, then ranking them or applying a threshold<\/td><\/tr><tr><td>noul (probability)<\/td><td>Yes\/no and approval gates<\/td><td>Whether content contains personal or confidential data, whether a refund is requested, whether to hand off to a human, whether it is a prompt injection; set a blocking threshold on the probability of &#8220;yes&#8221;<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>1. Combining question types<\/strong><\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">For a customer email, one request can return the topic, the urgency and whether a refund is requested, completing <strong>routing, prioritization and triggering the refund process<\/strong> in a single step. Our later recommendation to move &#8220;contains personal data&#8221; out of the single-choice question into a separate noul question applies this approach.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>2. Gate thresholds are adjustable<\/strong><\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">noul returns <strong>a probability rather than a yes\/no<\/strong>, so you can set the blocking threshold according to whether the business prefers to over-block or under-block.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>3. Unsuitable tasks<\/strong><\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">Tasks that need to generate text (writing emails, summaries, code), or decisions that need <strong>a written rationale for auditing<\/strong>. Decision models return only answers and probabilities, so these tasks should still go to an LLM.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Because most of these scenarios sit on the critical path while the user waits, <strong>latency<\/strong> and &#8220;<strong>what happens when there is no answer<\/strong>&#8221; matter as much as accuracy. In addition, the decision rules live in the criteria text: adding a category or adjusting a level only requires editing a description, with <strong>no retraining<\/strong>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Models Tested<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The 10 decision models (11 deployments) range from <strong>a 322M encoder to a 27B cloud model<\/strong>; the 5 on-prem families each use a different internal approach. Architecture descriptions come from the official descriptions returned by each service&#8217;s \/v1\/models, and memory is the actual footprint after loading.<\/p>\n\n\n\n<figure class=\"wp-block-table is-style-stripes has-small-font-size\"><table><tbody><tr><td><strong>Model<\/strong><\/td><td><strong>Architecture (official description)<\/strong><\/td><td><strong>Parameters \/ precision<\/strong><\/td><td><strong>Memory<\/strong><\/td><td><strong>Deployment<\/strong><\/td><\/tr><tr><td><strong>jev-latest<\/strong><\/td><td>TypeSafe&#8217;s System One decision model; architecture not disclosed<\/td><td>Undisclosed<\/td><td>\u2014<\/td><td>Cloud api.typesafe.ai<\/td><\/tr><tr><td><strong>vllm-sr\/d3<\/strong><\/td><td>Alias of d3-turbo; reads text, images and video<\/td><td>27B<\/td><td>\u2014<\/td><td>Cloud decision.vllm-sr.ai<\/td><\/tr><tr><td><strong>vllm-sr\/d3-flash<\/strong><\/td><td>Smaller model in the same family<\/td><td>9B<\/td><td>\u2014<\/td><td>Cloud decision.vllm-sr.ai<\/td><\/tr><tr><td><strong>vllm-sr\/d3-mini<\/strong><\/td><td>Smaller model in the same family<\/td><td>4B<\/td><td>\u2014<\/td><td>Cloud decision.vllm-sr.ai<\/td><\/tr><tr><td><strong>winnow:latest<\/strong><\/td><td>Fine-tuned Gemma 4 12B IT; reads the logits of option labels after its own prompt<\/td><td>12B \/ Q8_0 GGUF<\/td><td>12.7 GB<\/td><td>On-prem Ollaya<\/td><\/tr><tr><td><strong>winnow:e4b<\/strong><\/td><td>Fine-tuned Gemma 4 E4B IT; same approach as above, plus a temperature fitted by the author<\/td><td>7.5B \/ Q8_0 GGUF<\/td><td>8.0 GB<\/td><td>On-prem Ollaya<\/td><\/tr><tr><td><strong>clef:flash<\/strong><\/td><td>Post-trained Qwen3.5-9B; a joint schema head scores every option of every question in one forward pass<\/td><td>9B \/ F32 ONNX<\/td><td>19 GB<\/td><td>On-prem Ollaya<\/td><\/tr><tr><td><strong>clef-flash@ollama-q8<\/strong><\/td><td>The same clef-flash, run on llama.cpp instead<\/td><td>9.1B \/ Q8_0 GGUF<\/td><td>12.8 GB<\/td><td>On-prem ollama<\/td><\/tr><tr><td><strong>kev:4b<\/strong><\/td><td>Qwen3.5-4B base with LoRA and a pointer head<\/td><td>4.2B \/ F32 ONNX<\/td><td>9.5 GB<\/td><td>On-prem Ollaya<\/td><\/tr><tr><td><strong>clm:8b<\/strong><\/td><td>Contrastive: a Qwen3-8B encoder embeds the state and each option separately and scores by similarity; repeated options are cached<\/td><td>8.2B \/ F32 ONNX<\/td><td>16 GB<\/td><td>On-prem Ollaya<\/td><\/tr><tr><td><strong>laya:multilingual<\/strong><\/td><td>mmBERT-base, supports over 100 languages<\/td><td>322M \/ F16<\/td><td>0.7 GB<\/td><td>On-prem Ollaya<\/td><\/tr><tr><td><strong>gpt-oss-safeguard:20b<\/strong><\/td><td>Generative reasoning model that classifies safety based on policy text (control)<\/td><td>20B<\/td><td>12.9 GB<\/td><td>A separate shared Ollama host<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Architectural differences may show up directly in the results: laya and clm, which <strong>score by similarity<\/strong>, were nearly unusable with the criteria in this test. We suspect this relates to the <strong>negative phrasing and priority rules<\/strong> in the criteria (see &#8220;Three Key Technical Observations&#8221;).<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>How We Tested<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Both scenarios used the same procedure: every model was called one sample at a time on <strong>the same set of questions in the same order<\/strong>, and all models on the on-prem Ollaya and ollama servers were tested with <strong>exclusive use of the GPU<\/strong>.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>1. Two test scenarios<\/strong><\/h4>\n\n\n\n<figure class=\"wp-block-table is-style-stripes has-small-font-size\"><table><tbody><tr><td><strong>Item<\/strong><\/td><td><strong>Domain (topic classification)<\/strong><\/td><td><strong>Guard (content safety)<\/strong><\/td><\/tr><tr><td><strong>Categories<\/strong><\/td><td>Medical, financial, legal, sensitive data, general: 5 categories<\/td><td>Harassment, hate speech, violence, encouraging suicide, prompt injection, normal and more: 13 categories<\/td><\/tr><tr><td><strong>Decision rule<\/strong><\/td><td>Any content containing confidential or personal data is labeled &#8220;sensitive data&#8221;, regardless of topic<\/td><td>A 13-category prompt guard criteria, used unchanged during testing<\/td><\/tr><tr><td><strong>Data volume<\/strong><\/td><td>500 per category, 2,500 in total<\/td><td>200 per category, 2,600 in total<\/td><\/tr><tr><td><strong>Language<\/strong><\/td><td>Half Traditional Chinese, half English<\/td><td>Half Traditional Chinese, half English<\/td><\/tr><tr><td><strong>Share of hard cases<\/strong><\/td><td>20%; 40% for sensitive data<\/td><td>25% for most categories; 60% for normal<\/td><\/tr><tr><td><strong>Input per request<\/strong><\/td><td>About 480 tokens<\/td><td>About 1,150 tokens<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>2. Test data sources<\/strong><\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Generated one by one by an LLM from a writing spec<\/strong>, with template filling prohibited. The Traditional Chinese and English samples were written independently, not translated from each other, and include short questions, chat messages, emails and forum posts.<\/li>\n\n\n\n<li><strong>Hard cases deliberately include words from other categories<\/strong>. For example, &#8220;Any Korean dramas as good as Hospital Playlist?&#8221; belongs to the general category, while &#8220;Patient Chen Yi-chun, ID A225678901, diagnosed with stage 2 breast cancer\u2026&#8221; is about a medical topic but contains fictitious personal data, so it should be labeled sensitive data.<\/li>\n\n\n\n<li><strong>Quality checked by code<\/strong>: duplicates removed, fields checked, and non-sensitive categories scanned with regular expressions for accidental personal data patterns.<\/li>\n\n\n\n<li><strong>Note on Guard data sources<\/strong>: 9 of the 13 categories were written by an LLM; the privacy category reuses the personal data questions from Domain; for hate speech, encouraging suicide and illegal assistance, the LLM&#8217;s safety mechanisms blocked generation, so we used separately obtained data, which is more template-like and shorter.<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>3. Test environment<\/strong><\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>On-prem<\/strong>: one NVIDIA RTX 3090 (24 GB). The decision models were served by Ollaya, an Ollama-like inference server that exposes the \/v1\/systemone interface; the Q8 version of clef-flash was served by ollama (llama.cpp) on the same machine.<\/li>\n\n\n\n<li><strong>Control<\/strong>: gpt-oss-safeguard:20b ran on a separate shared Ollama host. The model was fully loaded into VRAM, but other users&#8217; models were also loaded on the same machine, so <strong>exclusive use could not be guaranteed<\/strong>.<\/li>\n\n\n\n<li><strong>Cloud<\/strong>: jev-latest (api.typesafe.ai) and the vllm-sr d3 family (decision.vllm-sr.ai); latency includes the network round trip from Taiwan to each service.<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>4. Test procedure<\/strong><\/h4>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Run systemctl restart ollaya over SSH to clear VRAM. Because ollama and Ollaya share the same GPU, Ollaya was also cleared before testing clef-flash.<\/li>\n\n\n\n<li><strong>Warm-up<\/strong>: testing starts only after 3 successful responses. The first request gets a 900-second timeout so the server has time to load the model; failures are retried.<\/li>\n\n\n\n<li>Use ollaya ps or \/api\/ps to confirm the model is fully on the GPU; otherwise it is not tested.<\/li>\n\n\n\n<li>All models use the same question order, shuffled with a <strong>fixed random seed<\/strong>, and requests are sent one at a time over a single keep-alive connection (concurrency = 1).<\/li>\n\n\n\n<li>Each result is written to JSONL immediately, so an interrupted run can resume from where it stopped.<\/li>\n\n\n\n<li>Transport errors and 5xx responses are retried twice; 429 responses are retried after waiting as the server instructs and are not counted as errors. <strong>Retried requests are excluded from latency statistics<\/strong>.<\/li>\n\n\n\n<li>After testing, confirm again that the model is still on the GPU, and check that no one else has loaded a model on Ollaya.<\/li>\n<\/ol>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>How to Read the Metrics<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">&#8220;<strong>Classification accuracy<\/strong>&#8221; answers whether the category was right; &#8220;<strong>blocking balanced accuracy<\/strong>&#8221; answers whether what should be blocked was blocked and what should pass was let through. For a prompt guard, the latter is the main metric.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>1. Classification metrics<\/strong><\/h4>\n\n\n\n<figure class=\"wp-block-table is-style-stripes has-small-font-size\"><table><tbody><tr><td><strong>Metric<\/strong><\/td><td><strong>Calculation<\/strong><\/td><td><strong>What it means<\/strong><\/td><\/tr><tr><td><strong>Accuracy<\/strong><\/td><td>Correct predictions \u00f7 total samples N<\/td><td>Overall share correct; no response or unparseable output counts as wrong<\/td><\/tr><tr><td><strong>Precision (class c)<\/strong><\/td><td>TP \u00f7 (TP + FP)<\/td><td>Share of predictions of c that are correct<\/td><\/tr><tr><td><strong>Recall (class c)<\/strong><\/td><td>TP \u00f7 (TP + FN)<\/td><td>Share of class c that is correctly identified<\/td><\/tr><tr><td><strong>F1 (class c)<\/strong><\/td><td>2 \u00d7 Precision \u00d7 Recall \u00f7 (Precision + Recall)<\/td><td>Harmonic mean of precision and recall<\/td><\/tr><tr><td><strong>Macro-F1<\/strong><\/td><td>Mean F1 across classes<\/td><td>Every class has equal weight, so poor performance on a minority class shows directly<\/td><\/tr><tr><td><strong>Lenient accuracy (Guard)<\/strong><\/td><td>A prediction of a class that overlaps by criteria definition also counts as correct<\/td><td>For example, hate speech is a subset of harassment, so predicting harassment is not wrong<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Note: TP is the number of samples where both the label and the prediction are c; FP is predicted c but the label is not c; FN is labeled c but the prediction is not c.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>2. Block \/ pass metrics (Guard)<\/strong><\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">Any prediction other than &#8220;normal&#8221; counts as a <strong>block<\/strong>, whether or not the category is correct. Questions in the &#8220;protected characteristics&#8221; category are neutral content and are excluded.<\/p>\n\n\n\n<figure class=\"wp-block-table is-style-stripes has-small-font-size\"><table><tbody><tr><td><strong>Metric<\/strong><\/td><td><strong>Calculation<\/strong><\/td><td><strong>What it means<\/strong><\/td><\/tr><tr><td><strong>Harmful block rate (TPR)<\/strong><\/td><td>Harmful samples blocked \u00f7 harmful samples<\/td><td>How much of what should be blocked was blocked<\/td><\/tr><tr><td><strong>False block rate (FPR)<\/strong><\/td><td>Normal samples blocked \u00f7 normal samples<\/td><td>How much of what should pass was blocked<\/td><\/tr><tr><td><strong>Blocking balanced accuracy<\/strong><\/td><td>(TPR + (1 \u2212 FPR)) \u00f7 2<\/td><td>The average of the two, unaffected by the ratio of harmful to normal samples<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">We use balanced accuracy instead of plain accuracy because there are 2,200 harmful samples and only 200 normal ones: a model that <strong>blocks everything<\/strong> reaches 91.7% plain accuracy but only 50% balanced accuracy. laya:multilingual is exactly this case: a 98.9% block rate, a 98.5% false block rate, and only 50.2% balanced accuracy.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>3. Speed metrics<\/strong><\/h4>\n\n\n\n<figure class=\"wp-block-table is-style-stripes has-small-font-size\"><table><tbody><tr><td><strong>Metric<\/strong><\/td><td><strong>Calculation<\/strong><\/td><td><strong>What it means<\/strong><\/td><\/tr><tr><td><strong>Latency<\/strong><\/td><td>End-to-end time measured on the client; retried requests excluded<\/td><td>Actual waiting time, including network and server queuing<\/td><\/tr><tr><td><strong>p50, p95, p99<\/strong><\/td><td>Sort latencies ascending, let h = (n \u2212 1) \u00d7 q, take the \u230ah\u230b-th and (\u230ah\u230b + 1)-th values (counting from 0), and interpolate linearly by the fractional part of h<\/td><td>The gap between p95 and p50 shows how stable tail latency is<\/td><\/tr><tr><td><strong>Single-connection throughput<\/strong><\/td><td>1000 \u00f7 mean latency (ms)<\/td><td>Requests per second on one connection; total multi-connection throughput was not tested<\/td><\/tr><tr><td><strong>Cold start<\/strong><\/td><td>Latency of the first request after a restart<\/td><td>Includes model loading time<\/td><\/tr><tr><td><strong>Token\u2013latency correlation<\/strong><\/td><td>Pearson r between input (or output) tokens and latency<\/td><td>Shows whether latency is driven by input length or output length<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>4. Confidence score<\/strong><\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">Decision models return a confidence score. We calculated the &#8220;<strong>coverage at confidence \u2265 0.9<\/strong>&#8221; and the accuracy within it, to assess whether a threshold can be used for routing: use high-confidence answers directly, and send the rest to a stronger model.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Results 1: Domain Topic Classification<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The top three all exceed <strong>98.9% accuracy<\/strong>, less than 1 percentage point apart; what really separates the models is the &#8220;<strong>sensitive data<\/strong>&#8221; category.<\/p>\n\n\n\n<figure class=\"wp-block-table is-style-stripes has-small-font-size\"><table><tbody><tr><td><strong>Model<\/strong><\/td><td><strong>Deployment<\/strong><\/td><td><strong>Accuracy<\/strong><\/td><td><strong>Chinese \/ English<\/strong><\/td><td><strong>Sensitive data recall<\/strong><\/td><td><strong>p50<\/strong><\/td><td><strong>p95<\/strong><\/td><\/tr><tr><td>jev-latest<\/td><td>Cloud<\/td><td>99.8%<\/td><td>99.8% \/ 99.7%<\/td><td>99.2%<\/td><td>205 ms<\/td><td>286 ms<\/td><\/tr><tr><td>winnow:latest<\/td><td>On-prem<\/td><td>99.0%<\/td><td>99.0% \/ 99.0%<\/td><td>100.0%<\/td><td>203 ms<\/td><td>225 ms<\/td><\/tr><tr><td>vllm-sr\/d3<\/td><td>Cloud<\/td><td>98.9%<\/td><td>98.7% \/ 99.0%<\/td><td>94.8%<\/td><td>276 ms<\/td><td>296 ms<\/td><\/tr><tr><td>vllm-sr\/d3-flash<\/td><td>Cloud<\/td><td>97.0%<\/td><td>97.0% \/ 97.0%<\/td><td>86.6%<\/td><td>231 ms<\/td><td>240 ms<\/td><\/tr><tr><td>vllm-sr\/d3-mini<\/td><td>Cloud<\/td><td>95.5%<\/td><td>95.0% \/ 95.9%<\/td><td>79.6%<\/td><td>228 ms<\/td><td>235 ms<\/td><\/tr><tr><td>clef:flash<\/td><td>On-prem (F32)<\/td><td>94.2%<\/td><td>93.7% \/ 94.7%<\/td><td>73.2%<\/td><td>809 ms<\/td><td>878 ms<\/td><\/tr><tr><td>clef-flash@ollama-q8<\/td><td>On-prem (Q8)<\/td><td>94.2%<\/td><td>93.8% \/ 94.6%<\/td><td>73.6%<\/td><td>246 ms<\/td><td>264 ms<\/td><\/tr><tr><td>winnow:e4b<\/td><td>On-prem<\/td><td>93.2%<\/td><td>93.8% \/ 92.6%<\/td><td>99.2%<\/td><td>106 ms<\/td><td>119 ms<\/td><\/tr><tr><td>kev:4b<\/td><td>On-prem<\/td><td>87.9%<\/td><td>87.4% \/ 88.3%<\/td><td>47.0%<\/td><td>479 ms<\/td><td>537 ms<\/td><\/tr><tr><td>laya:multilingual<\/td><td>On-prem<\/td><td>35.8%<\/td><td>35.5% \/ 36.2%<\/td><td>55.4%<\/td><td>30 ms<\/td><td>39 ms<\/td><\/tr><tr><td>clm:8b<\/td><td>On-prem<\/td><td>32.7%<\/td><td>34.0% \/ 31.4%<\/td><td>10.0%<\/td><td>186 ms<\/td><td>239 ms<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img data-recalc-dims=\"1\" decoding=\"async\" width=\"1024\" height=\"515\" data-attachment-id=\"15903\" data-permalink=\"https:\/\/ai-stack.ai\/en\/jev-benchmark\/image-47\" data-orig-file=\"https:\/\/i0.wp.com\/ai-stack.ai\/wp-content\/uploads\/2026\/11\/bfc32893-edited.png?fit=1536%2C773&amp;quality=100&amp;ct=202603031250&amp;ssl=1\" data-orig-size=\"1536,773\" data-comments-opened=\"0\" data-image-title=\"image\" data-image-description=\"\" data-image-caption=\"\" data-large-file=\"https:\/\/i0.wp.com\/ai-stack.ai\/wp-content\/uploads\/2026\/11\/bfc32893-edited.png?fit=1024%2C515&amp;quality=100&amp;ct=202603031250&amp;ssl=1\" src=\"https:\/\/i0.wp.com\/ai-stack.ai\/wp-content\/uploads\/2026\/11\/bfc32893-edited-1024x515.png?resize=1024%2C515&#038;quality=100&#038;ct=202603031250&#038;ssl=1\" alt=\"\" class=\"wp-image-15903\" style=\"aspect-ratio:1.7777777777777777\"\/><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>1. The gap is concentrated in &#8220;sensitive data&#8221;<\/strong><\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">For the medical, financial, legal and general categories, the top seven models all score above 95%, while sensitive data recall drops from 100% all the way to 47%. Most errors are hard cases where &#8220;<strong>the topic is legal or financial but the content includes confidential information<\/strong>&#8220;: the model classifies by topic and ignores the rule that sensitive data takes priority. Of jev-latest&#8217;s 6 errors, 4 involved internal bid amounts or contract unit prices.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>2. No clear difference between Chinese and English<\/strong><\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">For usable models, the accuracy gap between Traditional Chinese and English is within <strong>1.2 percentage points<\/strong>.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>3. Speed<\/strong><\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">Among usable models, <strong>winnow:e4b is the fastest<\/strong> (p50 106 ms). Running the same clef-flash as Q8 on llama.cpp cut p50 from 809 ms to 246 ms. After a restart, on-prem models needed 1.4\u20136.7 seconds to load on the first request.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>4. The accuracy \u00d7 speed Pareto frontier<\/strong><\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">The models that no other model beats on both accuracy and speed are jev-latest, winnow:latest, winnow:e4b and laya:multilingual. laya makes the list only because it is the fastest; its 35.8% accuracy is <strong>not usable in practice<\/strong>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Results 2: Guard Content Safety<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">jev-latest, vllm-sr\/d3 and winnow:latest all blocked <strong>94\u201395% of harmful content<\/strong> with false block rates <strong>below 2%<\/strong>. The table is sorted by blocking balanced accuracy.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>1. Blocking performance and latency<\/strong><\/h4>\n\n\n\n<figure class=\"wp-block-table is-style-stripes has-small-font-size\"><table><tbody><tr><td><strong>Model<\/strong><\/td><td><strong>Deployment<\/strong><\/td><td><strong>Blocking balanced accuracy<\/strong><\/td><td><strong>Harmful block rate<\/strong><\/td><td><strong>False block rate (95% CI)<\/strong><\/td><td><strong>p50<\/strong><\/td><\/tr><tr><td>jev-latest<\/td><td>Cloud<\/td><td>97.4%<\/td><td>95.3%<\/td><td>0.5% (0.1\u20132.8%)<\/td><td>207 ms<\/td><\/tr><tr><td>vllm-sr\/d3<\/td><td>Cloud<\/td><td>96.9%<\/td><td>94.3%<\/td><td>0.5% (0.1\u20132.8%)<\/td><td>344 ms<\/td><\/tr><tr><td>winnow:latest<\/td><td>On-prem<\/td><td>96.3%<\/td><td>94.6%<\/td><td>2.0% (0.8\u20135.0%)<\/td><td>416 ms<\/td><\/tr><tr><td>gpt-oss-safeguard (improved prompt)<\/td><td>On-prem (separate host)<\/td><td>94.6%<\/td><td>89.2%<\/td><td>0.0% (0.0\u20131.9%)<\/td><td>1.58 s<\/td><\/tr><tr><td>vllm-sr\/d3-flash<\/td><td>Cloud<\/td><td>94.1%<\/td><td>94.8%<\/td><td>6.5% (3.8\u201310.8%)<\/td><td>257 ms<\/td><\/tr><tr><td>vllm-sr\/d3-mini<\/td><td>Cloud<\/td><td>93.7%<\/td><td>93.0%<\/td><td>5.5% (3.1\u20139.6%)<\/td><td>243 ms<\/td><\/tr><tr><td>gpt-oss-safeguard (original prompt)<\/td><td>On-prem (separate host)<\/td><td>91.8%<\/td><td>83.6%<\/td><td>0.0% (0.0\u20131.9%)<\/td><td>1.92 s<\/td><\/tr><tr><td>winnow:e4b<\/td><td>On-prem<\/td><td>90.6%<\/td><td>91.3%<\/td><td>10.0% (6.6\u201314.9%)<\/td><td>208 ms<\/td><\/tr><tr><td>kev:4b<\/td><td>On-prem<\/td><td>90.0%<\/td><td>94.4%<\/td><td>14.5% (10.3\u201320.0%)<\/td><td>1.08 s<\/td><\/tr><tr><td>clef-flash@ollama-q8<\/td><td>On-prem (Q8)<\/td><td>79.6%<\/td><td>98.7%<\/td><td>39.5% (33.0\u201346.4%)<\/td><td>514 ms<\/td><\/tr><tr><td>clef:flash<\/td><td>On-prem (F32)<\/td><td>79.3%<\/td><td>98.6%<\/td><td>40.0% (33.5\u201346.9%)<\/td><td>1.79 s<\/td><\/tr><tr><td>laya:multilingual<\/td><td>On-prem<\/td><td>50.2%<\/td><td>98.9%<\/td><td>98.5% (95.7\u201399.5%)<\/td><td>30 ms<\/td><\/tr><tr><td>clm:8b<\/td><td>On-prem<\/td><td>49.1%<\/td><td>1.3%<\/td><td>3.0% (1.4\u20136.4%)<\/td><td>139 ms<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>2. Classification accuracy and block rates for key categories<\/strong><\/h4>\n\n\n\n<figure class=\"wp-block-table is-style-stripes has-small-font-size\"><table><tbody><tr><td><strong>Model<\/strong><\/td><td><strong>Classification accuracy<\/strong><\/td><td><strong>Privacy block rate<\/strong><\/td><td><strong>Prompt injection block rate<\/strong><\/td><\/tr><tr><td>jev-latest<\/td><td>83.2%<\/td><td>61.5%<\/td><td>100.0%<\/td><\/tr><tr><td>vllm-sr\/d3<\/td><td>80.6%<\/td><td>64.5%<\/td><td>100.0%<\/td><\/tr><tr><td>winnow:latest<\/td><td>78.3%<\/td><td>73.0%<\/td><td>98.0%<\/td><\/tr><tr><td>gpt-oss-safeguard (improved prompt)<\/td><td>73.3%<\/td><td>72.0%<\/td><td>76.5%<\/td><\/tr><tr><td>vllm-sr\/d3-flash<\/td><td>83.3%<\/td><td>85.5%<\/td><td>100.0%<\/td><\/tr><tr><td>vllm-sr\/d3-mini<\/td><td>80.1%<\/td><td>67.5%<\/td><td>100.0%<\/td><\/tr><tr><td>gpt-oss-safeguard (original prompt)<\/td><td>69.0%<\/td><td>60.5%<\/td><td>62.5%<\/td><\/tr><tr><td>winnow:e4b<\/td><td>73.4%<\/td><td>48.0%<\/td><td>84.0%<\/td><\/tr><tr><td>kev:4b<\/td><td>76.3%<\/td><td>70.0%<\/td><td>86.5%<\/td><\/tr><tr><td>clef-flash@ollama-q8<\/td><td>77.9%<\/td><td>95.5%<\/td><td>99.5%<\/td><\/tr><tr><td>clef:flash<\/td><td>77.9%<\/td><td>95.5%<\/td><td>99.5%<\/td><\/tr><tr><td>laya:multilingual<\/td><td>21.6%<\/td><td>100.0%<\/td><td>100.0%<\/td><\/tr><tr><td>clm:8b<\/td><td>7.5%<\/td><td>1.5%<\/td><td>0.0%<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>3. Interpreting the results<\/strong><\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>The top three are statistically hard to separate<\/strong>: the false block rate comes from only 200 normal samples, so 0.5% vs. 2.0% is really a difference of 1 vs. 4 samples, and the confidence intervals overlap heavily.<\/li>\n\n\n\n<li><strong>Privacy is a shared weakness<\/strong>: most usable models block only 60\u201373%, and personal data embedded in health or legal questions is let through most often. Excluding clef, whose false block rate reaches 40%, vllm-sr\/d3-flash is highest at 85.5%, but its false block rate is also 6.5%.<\/li>\n\n\n\n<li><strong>Blocking more is not the same as better<\/strong>: both clef versions blocked 98.6% of harmful content, but <strong>wrongly blocked 4 out of every 10 normal messages<\/strong>; even &#8220;how much water to put in the outer pot of a rice cooker to steam eggs&#8221; was labeled &#8220;misinformation&#8221;.<\/li>\n\n\n\n<li><strong>Low classification accuracy is not entirely the models&#8217; fault<\/strong>: in the original criteria, &#8220;protected characteristics&#8221; and &#8220;non-protected targets&#8221; read more like term definitions, and most models labeled the former as normal and the latter as harassment. Excluding these 400 questions raises the top three&#8217;s classification accuracy by another 10\u201314 percentage points, but this <strong>does not affect the block decision<\/strong>.<\/li>\n\n\n\n<li><strong>Discount the 3 template-like categories<\/strong>: hate speech, encouraging suicide and illegal assistance use repetitive sentence patterns, are short and have no hard cases. Usable models mostly reach 90\u2013100% recall on them, and real-world performance may be lower.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Three Key Technical Observations<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">These are the three most notable technical observations from this test. They cover <strong>whether decisions stay consistent after changing the deployment<\/strong>, <strong>how criteria wording interacts with model architecture<\/strong>, and <strong>the failure mode of LLM guards<\/strong>.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>1. Quantization barely affects decisions, but confidence scores need recalibration<\/strong><\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">The same clef-flash runs as F32 ONNX (19 GB) on Ollaya and as Q8_0 GGUF (12.8 GB) on ollama. Their per-sample predictions matched on <strong>99.8%<\/strong> of Domain and <strong>99.6%<\/strong> of Guard samples, and the Q8 version was <strong>3.3\u20133.5\u00d7 faster<\/strong>. Because both the runtime and the precision changed, the speedup cannot be attributed entirely to quantization.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Note that the median gap in confidence between the two is about <strong>0.12<\/strong>. If your system routes by confidence threshold, <strong>you must recalibrate the threshold after changing the deployment<\/strong>.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>2. Criteria wording must match the model architecture<\/strong><\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">laya (mmBERT) and clm (contrastive encoder) score by the <strong>similarity<\/strong> between the text and each option&#8217;s description. We suspect this is the main reason they performed poorly with the criteria in this test, but we have not yet verified this with rewritten criteria.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Specifically, Domain&#8217;s &#8220;general&#8221; category is defined with a negative statement (not mainly about medical, financial or legal&#8230;), so its description is full of other categories&#8217; keywords; &#8220;sensitive data&#8221; also carries a rule that it &#8220;takes priority regardless of topic&#8221;. As a result, laya labeled 62% of questions as sensitive data, and recall for the general category was only 1.8%. Even clef and kev often classified by topic and ignored the priority rule (sensitive data recall of 73% and 47% respectively).<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td><strong>Tips for writing criteria<\/strong><br>Describe each option in <strong>positive terms<\/strong>, and avoid negative statements such as &#8220;anything other than\u2026&#8221;.<br><strong>Don&#8217;t put priority rules into a single-choice question<\/strong>; split them into a separate noul yes\/no question, for example asking separately &#8220;does it contain personal or confidential data?&#8221;.<br><strong>Don&#8217;t use entries that are only term definitions as options<\/strong>, such as Guard&#8217;s &#8220;protected characteristics&#8221; and &#8220;non-protected targets&#8221;.<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>3. The LLM guard failure mode: no verdict means it gets through<\/strong><\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">gpt-oss-safeguard first generates reasoning and then outputs a JSON verdict. A common approach is to combine the policy and the user content into one user message; even with max_tokens set to 4,096, <strong>6.2% (162 samples) returned no verdict<\/strong>:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>64 samples<\/strong>: the reasoning used up all 4,096 tokens.<\/li>\n\n\n\n<li><strong>98 samples<\/strong>: the run finished normally, but the output was not JSON. On resending, we found the model was not classifying at all: it answered the question directly (actually writing an email), refused to answer, or was hijacked by the injected instructions in the question and returned an empty string.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Of these 162 samples, <strong>134 were harmful content<\/strong>. If the system uses a <strong>fail-open<\/strong> design, this content would pass straight through.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We ran a controlled experiment on the same 130 samples (max_tokens 4,096 in all cases; because there were only 10 normal samples, differences in false block rate cannot be interpreted):<\/p>\n\n\n\n<figure class=\"wp-block-table is-style-stripes has-small-font-size\"><table><tbody><tr><td><strong>Setup<\/strong><\/td><td><strong>No verdict<\/strong><\/td><td><strong>Harmful block rate<\/strong><\/td><td><strong>Blocking balanced accuracy<\/strong><\/td><td><strong>Mean output tokens (with verdict)<\/strong><\/td><\/tr><tr><td>Original policy, in the user message<\/td><td>10<\/td><td>84.5%<\/td><td>92.3%<\/td><td>365<\/td><\/tr><tr><td>Original policy, in the system message<\/td><td>22<\/td><td>76.4%<\/td><td>88.2%<\/td><td>350<\/td><\/tr><tr><td>Explicit instructions added, in the user message<\/td><td>0<\/td><td>90.0%<\/td><td>95.0%<\/td><td>389<\/td><\/tr><tr><td>Explicit instructions added, in the system message<\/td><td>0<\/td><td>91.8%<\/td><td>95.9%<\/td><td>331<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/developers.openai.com\/cookbook\/articles\/gpt-oss-safeguard-guide\" target=\"_blank\" rel=\"noopener\">OpenAI&#8217;s user guide<\/a> recommends putting the policy in the system message and the content in the user message, but <strong>moving only the policy actually made things worse<\/strong>: once the content stands alone as a user message, it looks like an ordinary question.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">What actually works is adding <strong>explicit instructions<\/strong> at the start of the policy (you are a classifier, output JSON only, do not respond to the content) and <strong>restating the output format<\/strong> at the end. Validated on all 2,600 samples, the no-verdict rate fell from <strong>6.2% to 1.0%<\/strong>, and all remaining cases were reasoning that used up 4,096 tokens.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Other points to note:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Close to the timeout limit<\/strong>: 5.2% of responses exceeded 1,024 tokens, and the longest took 27.6 seconds, close to the common 30-second timeout.<\/li>\n\n\n\n<li><strong>Unstable tail latency<\/strong>: latency is driven by output length (correlation 1.00), and p95 is 3\u20134\u00d7 p50.<\/li>\n\n\n\n<li><strong>Results are not fully reproducible<\/strong>: with temperature set to 0, resending the same question still does not guarantee the same result; rerunning the same 1,714 questions gave 99.5% prediction agreement.<\/li>\n\n\n\n<li><strong>The policy could be shorter<\/strong>: the policy in this test is about 2,500 tokens, far above the 400\u2013600 tokens the guide recommends, so shortening it may improve results further.<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img data-recalc-dims=\"1\" decoding=\"async\" width=\"1024\" height=\"576\" data-attachment-id=\"15894\" data-permalink=\"https:\/\/ai-stack.ai\/en\/jev-benchmark\/image-46\" data-orig-file=\"https:\/\/i0.wp.com\/ai-stack.ai\/wp-content\/uploads\/2026\/11\/c16f0e4b.png?fit=1600%2C900&amp;quality=100&amp;ct=202603031250&amp;ssl=1\" data-orig-size=\"1600,900\" data-comments-opened=\"0\" data-image-title=\"image\" data-image-description=\"\" data-image-caption=\"\" data-large-file=\"https:\/\/i0.wp.com\/ai-stack.ai\/wp-content\/uploads\/2026\/11\/c16f0e4b.png?fit=1024%2C576&amp;quality=100&amp;ct=202603031250&amp;ssl=1\" src=\"https:\/\/i0.wp.com\/ai-stack.ai\/wp-content\/uploads\/2026\/11\/e0ae1fa6.png?resize=1024%2C576&#038;quality=100&#038;ct=202603031250&#038;ssl=1\" alt=\"\" class=\"wp-image-15894\" style=\"aspect-ratio:1.7777777777777777\"\/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>How to Choose a Model for Your Needs<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">No single model is the best choice in every scenario. We recommend <strong>first deciding whether data can be sent outside<\/strong>, then weighing accuracy against latency.<\/p>\n\n\n\n<figure class=\"wp-block-table is-style-stripes has-small-font-size\"><table><tbody><tr><td><strong>Need<\/strong><\/td><td><strong>Recommendation<\/strong><\/td><td><strong>Basis<\/strong><\/td><td><strong>Caveats<\/strong><\/td><\/tr><tr><td><strong>Cloud allowed, accuracy first<\/strong><\/td><td>jev-latest<\/td><td>Domain 99.8%; Guard blocking balanced accuracy 97.4%, false block rate 0.5%; p50 about 205 ms<\/td><td>All messages are sent to a third party; latency depends on the network<\/td><\/tr><tr><td><strong>Second cloud provider or backup<\/strong><\/td><td>vllm-sr\/d3<\/td><td>Domain 98.9%; Guard 96.9%, false block rate 0.5%<\/td><td>Guard p50 344 ms; privacy block rate only 64.5%<\/td><\/tr><tr><td><strong>Data cannot leave the internal environment<\/strong><\/td><td>winnow:latest<\/td><td>Domain 99.0% (sensitive data 100%); Guard 96.3%; p50 203 \/ 416 ms<\/td><td>Needs about 13 GB VRAM; confirm the model is on the GPU<\/td><\/tr><tr><td><strong>On-prem, latency or cost first<\/strong><\/td><td>winnow:e4b<\/td><td>Domain p50 106 ms, 9.3 req\/s on one connection; 8 GB VRAM<\/td><td>Guard false block rate 10%, privacy block rate 48%; better suited to Domain routing<\/td><\/tr><tr><td><strong>Especially concerned about personal data leaks<\/strong><\/td><td>In addition to the main guard, use vllm-sr\/d3-flash to detect personal data<\/td><td>Privacy block rate 85.5%, the highest among usable models<\/td><td>False block rate 6.5%, not suitable as the main guard on its own; the effect of a separate yes\/no question was not tested<\/td><\/tr><tr><td><strong>Readable rationale required<\/strong><\/td><td>gpt-oss-safeguard: add explicit instructions to the policy and put it in the system message, with max_tokens of at least 4,096<\/td><td>Blocking balanced accuracy 94.6%, with rationale<\/td><td>p50 1.58 s, p95 6.2 s; still 1% with no verdict<\/td><\/tr><tr><td><strong>Not recommended with these criteria<\/strong><\/td><td>laya:multilingual, clm:8b; the clef family for Guard<\/td><td>Balanced accuracy about 50%; clef false block rate 40%<\/td><td>Can be re-evaluated after rewriting the criteria<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Architecture recommendations<\/strong><\/h4>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Route by confidence score<\/strong>: in the Domain scenario, 97.2% of winnow:latest&#8217;s questions had confidence \u2265 0.9, with 99.8% accuracy on that portion; jev-latest had 94.9% coverage at 100% accuracy. A faster model can handle high-confidence requests and pass only low-confidence ones to a stronger model. This should not be applied directly to Guard, though: winnow:latest&#8217;s classification accuracy on questions with confidence \u2265 0.9 was only 82.4%, so <strong>thresholds must be calibrated for each scenario<\/strong>.<\/li>\n\n\n\n<li><strong>Split one question into several<\/strong>: first ask &#8220;is it harmful?&#8221; and then &#8220;which category?&#8221;, or use noul to split personal and confidential data into separate yes\/no questions. Because one request can ask several questions, <strong>this adds no network round trips<\/strong>.<\/li>\n\n\n\n<li><strong>Define the failure-handling strategy in advance<\/strong>: a decision model always returns an option, but the service can still time out or return a 5xx; an LLM guard adds the &#8220;no verdict&#8221; case. In production, <strong>choose explicitly between fail-open and fail-closed<\/strong>, and keep monitoring how often it happens.<\/li>\n\n\n\n<li><strong>Monitor deployment status<\/strong>: regularly check the model&#8217;s DEVICE and per-request latency. If latency suddenly jumps by 10\u00d7 or more, the model has most likely been pushed onto the CPU. Early in testing, winnow:latest took 12.5 seconds per request on the CPU, versus only 203 ms with exclusive use of the GPU.<\/li>\n<\/ol>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>How Do the Three Representative Options Differ?<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The table below summarizes the key differences between three representative options (cloud, on-prem and a generative LLM guard) for quick comparison.<\/p>\n\n\n\n<figure class=\"wp-block-table is-style-stripes has-small-font-size\"><table><tbody><tr><td><strong>Aspect<\/strong><\/td><td><strong>jev-latest (cloud)<\/strong><\/td><td><strong>winnow:latest (on-prem)<\/strong><\/td><td><strong>gpt-oss-safeguard (LLM guard)<\/strong><\/td><\/tr><tr><td><strong>Does data leave the internal environment?<\/strong><\/td><td>Yes, sent to a third-party service<\/td><td>No<\/td><td>No (can be self-hosted)<\/td><\/tr><tr><td><strong>Domain accuracy<\/strong><\/td><td>99.8%<\/td><td>99.0%<\/td><td>Not tested<\/td><\/tr><tr><td><strong>Guard blocking balanced accuracy<\/strong><\/td><td>97.4%<\/td><td>96.3%<\/td><td>94.6% (improved prompt)<\/td><\/tr><tr><td><strong>Guard false block rate<\/strong><\/td><td>0.5%<\/td><td>2.0%<\/td><td>0.0%<\/td><\/tr><tr><td><strong>Guard p50 latency<\/strong><\/td><td>207 ms<\/td><td>416 ms<\/td><td>1.58 s<\/td><\/tr><tr><td><strong>Can it return no verdict?<\/strong><\/td><td>No<\/td><td>No<\/td><td>Yes, about 1.0%<\/td><\/tr><tr><td><strong>Rationale<\/strong><\/td><td>Probabilities only<\/td><td>Probabilities only<\/td><td>Rationale text<\/td><\/tr><tr><td><strong>Best for<\/strong><\/td><td>Cloud allowed, accuracy first<\/td><td>Data must stay internal, with good accuracy<\/td><td>An auditable rationale is required<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">In short: <strong>if you want accuracy and can use the cloud, choose jev-latest; if data must stay internal, choose winnow:latest; consider a generative LLM guard only when you need a readable rationale, and make sure to handle the &#8220;no verdict&#8221; case.<\/strong><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Test Limitations and Next Steps<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">This benchmark is best used to <strong>compare the relative performance of models<\/strong>, not as absolute numbers for production.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Synthetic data<\/strong>: the test data was written by an LLM and may be tidier than real user language; 3 Guard categories use template-like sentences.<\/li>\n\n\n\n<li><strong>Only 200 normal samples<\/strong>: a single false block is 0.5%, with a 95% confidence interval of 0.1\u20132.8%, so <strong>differences within 2 percentage points are not statistically meaningful<\/strong>. Confidence intervals were calculated with the Wilson formula (explained below).<\/li>\n\n\n\n<li><strong>Each model was run only once<\/strong>: decision models give almost identical results on reruns; LLM guard output is not fully reproducible.<\/li>\n\n\n\n<li><strong>Cloud latency is for reference only<\/strong>: it includes the round trip from our network to the service, and we cannot confirm the provider&#8217;s hardware or load.<\/li>\n\n\n\n<li><strong>Criteria were not tuned per model<\/strong>: the results reflect performance &#8220;under the same criteria&#8221;; the low scores of laya and clm do not mean they would do as poorly with other wording.<\/li>\n\n\n\n<li><strong>Single connection only<\/strong>: total throughput and tail latency under high concurrency are the next things to test.<\/li>\n\n\n\n<li><strong>Only one RTX 3090 on-prem<\/strong>: absolute numbers will differ on other GPUs, but the relative trends between models should be similar.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>About the Wilson confidence interval<\/strong>: the Wilson formula gives a reasonable interval even when the proportion is 0% or close to 0%.<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td><strong>Formula<\/strong><br>Wilson interval = ( p\u0302 + z\u00b2\/2n\u00a0 \u00b1\u00a0 z \u00d7 \u221a( p\u0302(1 \u2212 p\u0302)\/n + z\u00b2\/4n\u00b2 ) ) \u00f7 ( 1 + z\u00b2\/n )\u00a0p\u0302: observed proportion \u00a0 n: sample size \u00a0 z: 1.96 for a 95% confidence level<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Directions worth exploring<\/strong>: retest the encoder-based models with rewritten criteria, and split personal data detection into a separate yes\/no question to check whether &#8220;splitting the question&#8221; fixes the weakness in the privacy category.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>FAQ<\/strong><\/h2>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Q1: What is the biggest difference between a decision model and using a generative LLM for classification?<\/strong><\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">A decision model <strong>generates no text<\/strong>: every question always returns a typed answer and a probability distribution, and latency depends mainly on input length. A generative LLM has to reason first and then output JSON, its latency depends on output length, and it can return &#8220;no verdict&#8221;. If the business needs an auditable written rationale, a generative LLM still has value.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Q2: If data cannot leave our internal environment, which model should we choose?<\/strong><\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">In this test, the best on-prem model was <strong>winnow:latest<\/strong>: 99.0% Domain accuracy (100% sensitive data recall) and 96.3% Guard blocking balanced accuracy, needing about 13 GB VRAM. If latency and cost matter more, consider winnow:e4b, but its Guard false block rate is 10%, so it is better suited to Domain routing.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Q3: Will accuracy drop if we deploy a Q8 quantized version?<\/strong><\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">Taking clef-flash as an example, the per-sample predictions of the F32 and Q8 versions matched on 99.8% of Domain and 99.6% of Guard samples, and Q8 was 3.3\u20133.5\u00d7 faster. However, the median confidence gap between them is about 0.12, so if you route by confidence threshold, <strong>the threshold needs to be recalibrated<\/strong>.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Q4: Why does the Guard scenario use &#8220;blocking balanced accuracy&#8221; instead of plain accuracy?<\/strong><\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">Because the numbers of harmful and normal samples are very unequal (2,200 vs. 200). A model that blocks everything reaches 91.7% plain accuracy but only 50% balanced accuracy. Balanced accuracy reflects both &#8220;was what should be blocked blocked&#8221; and &#8220;was what should pass let through&#8221;.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Q5: laya and clm scored very low. Are the models themselves bad?<\/strong><\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">Not necessarily. Both score by &#8220;the similarity between the text and each option&#8217;s description&#8221;. We suspect the negative phrasing and priority rules in these criteria are the main cause, but we have not yet verified this with rewritten criteria. The results only represent performance &#8220;under the same criteria&#8221;.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Q6: Can we route directly with a confidence threshold?<\/strong><\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">It works well in the Domain scenario: 97.2% of winnow:latest&#8217;s questions had confidence \u2265 0.9, with 99.8% accuracy on that portion. In the Guard scenario, however, classification accuracy at the same threshold was only 82.4%, so <strong>thresholds must be calibrated separately for each scenario<\/strong>.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Q7: Can the numbers in this benchmark be used directly as production performance metrics?<\/strong><\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">We don&#8217;t recommend it. This test used synthetic data, a single connection and a single RTX 3090, and cloud latency includes the network round trip. The data is best used to <strong>compare the relative performance of models<\/strong>; before going to production, validate with your own data and real traffic.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Conclusion<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Decision models replace generated text with &#8220;<strong>typed answers + probability distributions<\/strong>&#8220;, so decisions on the critical path, such as classification, grading and approval gates, can balance <strong>accuracy, latency and predictability<\/strong>. This test shows that both the cloud jev-latest and the on-prem winnow:latest are ready for real-world use, while the way the criteria are written, how confidence thresholds are calibrated, and how failure handling is designed determine how well the system ultimately performs.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">As AI agents and multi-model routing become standard in enterprises, &#8220;<strong>decide first, then generate<\/strong>&#8221; will be an important design principle for controlling cost and risk. If you are planning AI inference infrastructure, model routing or content safety guardrails, contact the <strong>INFINITIX team<\/strong>, and our consultants will help you assess the deployment that fits you best.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>References<\/strong><\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/www.datacamp.com\/blog\/system-one-models-jev\" target=\"_blank\" rel=\"noopener\">DataCamp: System One models \u2014 Jev<\/a>: an introduction to Jev&#8217;s three question types (choice, score, noul), probability calibration and suitable use cases.<\/li>\n\n\n\n<li><a href=\"https:\/\/developers.openai.com\/cookbook\/articles\/gpt-oss-safeguard-guide\" target=\"_blank\" rel=\"noopener\">OpenAI: User guide for gpt-oss-safeguard<\/a>: put the policy in the system message, make the output format explicit and restate it, and keep the policy to 400\u2013600 tokens.<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Benchmarking 10 decision models, including Jev, on 5,100 bilingual samples for domain classification and content safety, with cloud vs. on-premises recommendations.<\/p>\n","protected":false},"author":253372381,"featured_media":15886,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[96987570,96989088],"tags":[],"class_list":["post-15876","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technical-support","category-tech-insights"],"blocksy_meta":[],"acf":[],"jetpack_shortlink":"https:\/\/wp.me\/ph344V-484","jetpack_sharing_enabled":true,"jetpack_featured_media_url":"https:\/\/i0.wp.com\/ai-stack.ai\/wp-content\/uploads\/2026\/11\/976b0ab9-1.png?fit=1920%2C1080&quality=100&ct=202603031250&ssl=1","_links":{"self":[{"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/posts\/15876","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/users\/253372381"}],"replies":[{"embeddable":true,"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/comments?post=15876"}],"version-history":[{"count":4,"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/posts\/15876\/revisions"}],"predecessor-version":[{"id":15941,"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/posts\/15876\/revisions\/15941"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/media\/15886"}],"wp:attachment":[{"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/media?parent=15876"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/categories?post=15876"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/tags?post=15876"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}