When companies bring LLMs into customer service, internal knowledge bases and AI agent workflows, a “decision” usually has to happen before any “generation”: Which model should this message go to? Does the content contain personal data? Is it a prompt injection? These decisions sit on the critical path while the user waits, and every single request passes through them. So before choosing a model, teams usually need to answer the following questions:

  • How large is the real accuracy gap between cloud and on-prem decision models?
  • Can a single model handle both “topic classification” and “content safety”?
  • If we deploy a quantized version, will the decisions change?
  • What happens to the system when a generative LLM guard returns no verdict?

To answer these questions, we used a test set of 5,100 samples, half Chinese and half English, across two scenarios, “topic classification (Domain)” and “content safety (Guard)”. We tested 10 decision models in 11 deployments (clef was deployed in two ways), with one generative LLM guard (gpt-oss-safeguard) as a control.

What Is the Jev Decision Model?

Jev is a System One decision model from TypeSafe. You send it a piece of text to evaluate (state) and a set of typed questions (questions), and the model returns the answer and probability distribution for every question in one call, without generating any text.

The same /v1/systemone interface has other implementations. The clef, clm, laya, winnow and kev families tested here, as well as vllm-sr’s d3 family, are all compatible; below we refer to them collectively as “decision models“.

1. Three question types: choice, score, noul

Question typeQuestion it answersHow it is defined (criteria)What it returns
choice (classification)Which category is it?Object: each option name maps to a descriptionchoice, confidence, probabilities for each option
score (graded rating)How high is the level?Array: level descriptions ordered from low to highscore, confidence, a legend of levels, probabilities for each level
noul (probability)Yes or no?Not needed; only instructionsnoul: the probability of “yes”, between 0 and 1
  • score is the expected value of the levels: each level’s index is weighted by its probability, so the range is 0 to “number of levels − 1”. For example, if three levels have probabilities of 0.67, 0.29 and 0.04, the score is 0.38. The order of the criteria is therefore the direction of the scale, and they must be listed from low to high.
  • All three question types can go in one request: you get several independent decisions at once, with no extra network round trips.
  • This benchmark uses choice only: we confirmed the response formats of score and noul by testing them against jev-latest (reported as jev-1.13.0 in the response).

2. One request, several decisions: a worked example

The example below evaluates one customer email for its topic (choice), urgency (score) and whether the customer is asking for a refund (noul) at the same time.

Request
{
  "model": "jev-latest",
  "state": "Hi, I was charged twice for my subscription this month and I need a refund today, otherwise I will cancel.",
  "questions": {
    "topic": {
      "type": "choice",
      "instructions": "Pick the topic of the message.",
      "criteria": {
        "billing": "Payments, invoices, refunds.",
        "technical": "Bugs, errors, outages.",
        "sales": "Pricing and plans."
      }
    },
    "urgency": {
      "type": "score",
      "instructions": "Rate how urgent this message is.",
      "criteria": [
        "Not urgent: no time pressure.",
        "Low: wants help this week.",
        "Medium: wants an answer within a day.",
        "High: needs action today, money involved.",
        "Critical: service down or legal threat."
      ]
    },
    "refund": {
      "type": "noul",
      "instructions": "Is the customer asking for a refund?"
    }
  }
}

Response
{
  "model": "jev-1.13.0",
  "answers": {
    "topic": {
      "type": "choice",
      "choice": "billing",
      "confidence": 1.0,
      "probabilities": {"billing": 1.0, "technical": 0.0, "sales": 0.0}
    },
    "urgency": {
      "type": "score",
      "score": 3.0,
      "confidence": 1.0,
      "legend": {
        "0": "Not urgent: no time pressure.",
        "1": "Low: wants help this week.",
        "2": "Medium: wants an answer within a day.",
        "3": "High: needs action today, money involved.",
        "4": "Critical: service down or legal threat."
      },
      "probabilities": {"0": 0.0, "1": 0.0, "2": 0.0, "3": 1.0, "4": 0.0}
    },
    "refund": {"type": "noul", "noul": 0.99}
  },
  "usage": {"input_tokens": 458, "output_tokens": 69}
}


Note that confidence is a separate confidence score returned by the service and is not necessarily equal to the probability of the top option (in this test, laya:multilingual once returned a confidence of 0.9955 with a top probability of 0.997). According to DataCamp’s introduction, Jev’s probabilities are calibrated. In our Domain test, jev-latest got every question right where confidence was ≥ 0.9 (94.9% of questions), which is consistent with that claim.

3. Decision models vs. generative LLMs: key differences

The table below compares how a decision model and a generative LLM guard handle the same task. The data comes from our Guard test. Because the two ran on different hosts, latency is for reference only.

AspectDecision model (systemone)Generative LLM guard (gpt-oss-safeguard:20b)
OutputA typed answer for each question (category, score or probability) + probability distributionReasoning + a block of JSON text that you must parse yourself
Output tokens (self-hosted)0327–365 on average
Latency mainly depends onInput length (criteria + content)Output length (correlation with latency 1.00)
p50 / p95 latencywinnow:latest 416 / 436 ms1.58 / 6.24 s
Can it return no answer?No, every question always gets an answerYes, 1.0% (improved prompt) to 6.2% (original prompt)
Confidence scoreYes, usable for threshold-based routingNone
ExplainabilityProbabilities onlyRationale text
Changing the decision rulesEdit the criteria text, no retraining neededEdit the policy prompt

Note: the cloud jev-latest reports output tokens in its usage (about 160 for Guard and 53 for Domain in this test, 69 in the example above), but the response is still a typed answer with no text to parse.

Where Do Decision Models Fit?

The rule is simple: if the answer can be listed in advance (a category, a level, or a yes/no), the task suits a decision model. Tasks that need to produce text or reasons do not.

Question typeSuitable tasksExamples
choice (classification)Classification and routingRouting questions to the right model or agent by medical, financial or legal topic (the Domain scenario in this article); identifying which type of harmful content a message contains (the Guard scenario in this article); assigning customer service tickets
score (graded rating)Grading and rankingComplaint urgency, risk level, content rating; scoring the quality of LLM answers, then ranking them or applying a threshold
noul (probability)Yes/no and approval gatesWhether content contains personal or confidential data, whether a refund is requested, whether to hand off to a human, whether it is a prompt injection; set a blocking threshold on the probability of “yes”

1. Combining question types

For a customer email, one request can return the topic, the urgency and whether a refund is requested, completing routing, prioritization and triggering the refund process in a single step. Our later recommendation to move “contains personal data” out of the single-choice question into a separate noul question applies this approach.

2. Gate thresholds are adjustable

noul returns a probability rather than a yes/no, so you can set the blocking threshold according to whether the business prefers to over-block or under-block.

3. Unsuitable tasks

Tasks that need to generate text (writing emails, summaries, code), or decisions that need a written rationale for auditing. Decision models return only answers and probabilities, so these tasks should still go to an LLM.

Because most of these scenarios sit on the critical path while the user waits, latency and “what happens when there is no answer” matter as much as accuracy. In addition, the decision rules live in the criteria text: adding a category or adjusting a level only requires editing a description, with no retraining.

Models Tested

The 10 decision models (11 deployments) range from a 322M encoder to a 27B cloud model; the 5 on-prem families each use a different internal approach. Architecture descriptions come from the official descriptions returned by each service’s /v1/models, and memory is the actual footprint after loading.

ModelArchitecture (official description)Parameters / precisionMemoryDeployment
jev-latestTypeSafe’s System One decision model; architecture not disclosedUndisclosed—Cloud api.typesafe.ai
vllm-sr/d3Alias of d3-turbo; reads text, images and video27B—Cloud decision.vllm-sr.ai
vllm-sr/d3-flashSmaller model in the same family9B—Cloud decision.vllm-sr.ai
vllm-sr/d3-miniSmaller model in the same family4B—Cloud decision.vllm-sr.ai
winnow:latestFine-tuned Gemma 4 12B IT; reads the logits of option labels after its own prompt12B / Q8_0 GGUF12.7 GBOn-prem Ollaya
winnow:e4bFine-tuned Gemma 4 E4B IT; same approach as above, plus a temperature fitted by the author7.5B / Q8_0 GGUF8.0 GBOn-prem Ollaya
clef:flashPost-trained Qwen3.5-9B; a joint schema head scores every option of every question in one forward pass9B / F32 ONNX19 GBOn-prem Ollaya
clef-flash@ollama-q8The same clef-flash, run on llama.cpp instead9.1B / Q8_0 GGUF12.8 GBOn-prem ollama
kev:4bQwen3.5-4B base with LoRA and a pointer head4.2B / F32 ONNX9.5 GBOn-prem Ollaya
clm:8bContrastive: a Qwen3-8B encoder embeds the state and each option separately and scores by similarity; repeated options are cached8.2B / F32 ONNX16 GBOn-prem Ollaya
laya:multilingualmmBERT-base, supports over 100 languages322M / F160.7 GBOn-prem Ollaya
gpt-oss-safeguard:20bGenerative reasoning model that classifies safety based on policy text (control)20B12.9 GBA separate shared Ollama host

Architectural differences may show up directly in the results: laya and clm, which score by similarity, were nearly unusable with the criteria in this test. We suspect this relates to the negative phrasing and priority rules in the criteria (see “Three Key Technical Observations”).

How We Tested

Both scenarios used the same procedure: every model was called one sample at a time on the same set of questions in the same order, and all models on the on-prem Ollaya and ollama servers were tested with exclusive use of the GPU.

1. Two test scenarios

ItemDomain (topic classification)Guard (content safety)
CategoriesMedical, financial, legal, sensitive data, general: 5 categoriesHarassment, hate speech, violence, encouraging suicide, prompt injection, normal and more: 13 categories
Decision ruleAny content containing confidential or personal data is labeled “sensitive data”, regardless of topicA 13-category prompt guard criteria, used unchanged during testing
Data volume500 per category, 2,500 in total200 per category, 2,600 in total
LanguageHalf Traditional Chinese, half EnglishHalf Traditional Chinese, half English
Share of hard cases20%; 40% for sensitive data25% for most categories; 60% for normal
Input per requestAbout 480 tokensAbout 1,150 tokens

2. Test data sources

  • Generated one by one by an LLM from a writing spec, with template filling prohibited. The Traditional Chinese and English samples were written independently, not translated from each other, and include short questions, chat messages, emails and forum posts.
  • Hard cases deliberately include words from other categories. For example, “Any Korean dramas as good as Hospital Playlist?” belongs to the general category, while “Patient Chen Yi-chun, ID A225678901, diagnosed with stage 2 breast cancer…” is about a medical topic but contains fictitious personal data, so it should be labeled sensitive data.
  • Quality checked by code: duplicates removed, fields checked, and non-sensitive categories scanned with regular expressions for accidental personal data patterns.
  • Note on Guard data sources: 9 of the 13 categories were written by an LLM; the privacy category reuses the personal data questions from Domain; for hate speech, encouraging suicide and illegal assistance, the LLM’s safety mechanisms blocked generation, so we used separately obtained data, which is more template-like and shorter.

3. Test environment

  • On-prem: one NVIDIA RTX 3090 (24 GB). The decision models were served by Ollaya, an Ollama-like inference server that exposes the /v1/systemone interface; the Q8 version of clef-flash was served by ollama (llama.cpp) on the same machine.
  • Control: gpt-oss-safeguard:20b ran on a separate shared Ollama host. The model was fully loaded into VRAM, but other users’ models were also loaded on the same machine, so exclusive use could not be guaranteed.
  • Cloud: jev-latest (api.typesafe.ai) and the vllm-sr d3 family (decision.vllm-sr.ai); latency includes the network round trip from Taiwan to each service.

4. Test procedure

  1. Run systemctl restart ollaya over SSH to clear VRAM. Because ollama and Ollaya share the same GPU, Ollaya was also cleared before testing clef-flash.
  2. Warm-up: testing starts only after 3 successful responses. The first request gets a 900-second timeout so the server has time to load the model; failures are retried.
  3. Use ollaya ps or /api/ps to confirm the model is fully on the GPU; otherwise it is not tested.
  4. All models use the same question order, shuffled with a fixed random seed, and requests are sent one at a time over a single keep-alive connection (concurrency = 1).
  5. Each result is written to JSONL immediately, so an interrupted run can resume from where it stopped.
  6. Transport errors and 5xx responses are retried twice; 429 responses are retried after waiting as the server instructs and are not counted as errors. Retried requests are excluded from latency statistics.
  7. After testing, confirm again that the model is still on the GPU, and check that no one else has loaded a model on Ollaya.

How to Read the Metrics

“Classification accuracy” answers whether the category was right; “blocking balanced accuracy” answers whether what should be blocked was blocked and what should pass was let through. For a prompt guard, the latter is the main metric.

1. Classification metrics

MetricCalculationWhat it means
AccuracyCorrect predictions ÷ total samples NOverall share correct; no response or unparseable output counts as wrong
Precision (class c)TP ÷ (TP + FP)Share of predictions of c that are correct
Recall (class c)TP ÷ (TP + FN)Share of class c that is correctly identified
F1 (class c)2 × Precision × Recall ÷ (Precision + Recall)Harmonic mean of precision and recall
Macro-F1Mean F1 across classesEvery class has equal weight, so poor performance on a minority class shows directly
Lenient accuracy (Guard)A prediction of a class that overlaps by criteria definition also counts as correctFor example, hate speech is a subset of harassment, so predicting harassment is not wrong

Note: TP is the number of samples where both the label and the prediction are c; FP is predicted c but the label is not c; FN is labeled c but the prediction is not c.

2. Block / pass metrics (Guard)

Any prediction other than “normal” counts as a block, whether or not the category is correct. Questions in the “protected characteristics” category are neutral content and are excluded.

MetricCalculationWhat it means
Harmful block rate (TPR)Harmful samples blocked ÷ harmful samplesHow much of what should be blocked was blocked
False block rate (FPR)Normal samples blocked ÷ normal samplesHow much of what should pass was blocked
Blocking balanced accuracy(TPR + (1 − FPR)) ÷ 2The average of the two, unaffected by the ratio of harmful to normal samples

We use balanced accuracy instead of plain accuracy because there are 2,200 harmful samples and only 200 normal ones: a model that blocks everything reaches 91.7% plain accuracy but only 50% balanced accuracy. laya:multilingual is exactly this case: a 98.9% block rate, a 98.5% false block rate, and only 50.2% balanced accuracy.

3. Speed metrics

MetricCalculationWhat it means
LatencyEnd-to-end time measured on the client; retried requests excludedActual waiting time, including network and server queuing
p50, p95, p99Sort latencies ascending, let h = (n − 1) × q, take the ⌊h⌋-th and (⌊h⌋ + 1)-th values (counting from 0), and interpolate linearly by the fractional part of hThe gap between p95 and p50 shows how stable tail latency is
Single-connection throughput1000 ÷ mean latency (ms)Requests per second on one connection; total multi-connection throughput was not tested
Cold startLatency of the first request after a restartIncludes model loading time
Token–latency correlationPearson r between input (or output) tokens and latencyShows whether latency is driven by input length or output length

4. Confidence score

Decision models return a confidence score. We calculated the “coverage at confidence ≥ 0.9” and the accuracy within it, to assess whether a threshold can be used for routing: use high-confidence answers directly, and send the rest to a stronger model.

Results 1: Domain Topic Classification

The top three all exceed 98.9% accuracy, less than 1 percentage point apart; what really separates the models is the “sensitive data” category.

ModelDeploymentAccuracyChinese / EnglishSensitive data recallp50p95
jev-latestCloud99.8%99.8% / 99.7%99.2%205 ms286 ms
winnow:latestOn-prem99.0%99.0% / 99.0%100.0%203 ms225 ms
vllm-sr/d3Cloud98.9%98.7% / 99.0%94.8%276 ms296 ms
vllm-sr/d3-flashCloud97.0%97.0% / 97.0%86.6%231 ms240 ms
vllm-sr/d3-miniCloud95.5%95.0% / 95.9%79.6%228 ms235 ms
clef:flashOn-prem (F32)94.2%93.7% / 94.7%73.2%809 ms878 ms
clef-flash@ollama-q8On-prem (Q8)94.2%93.8% / 94.6%73.6%246 ms264 ms
winnow:e4bOn-prem93.2%93.8% / 92.6%99.2%106 ms119 ms
kev:4bOn-prem87.9%87.4% / 88.3%47.0%479 ms537 ms
laya:multilingualOn-prem35.8%35.5% / 36.2%55.4%30 ms39 ms
clm:8bOn-prem32.7%34.0% / 31.4%10.0%186 ms239 ms

1. The gap is concentrated in “sensitive data”

For the medical, financial, legal and general categories, the top seven models all score above 95%, while sensitive data recall drops from 100% all the way to 47%. Most errors are hard cases where “the topic is legal or financial but the content includes confidential information“: the model classifies by topic and ignores the rule that sensitive data takes priority. Of jev-latest’s 6 errors, 4 involved internal bid amounts or contract unit prices.

2. No clear difference between Chinese and English

For usable models, the accuracy gap between Traditional Chinese and English is within 1.2 percentage points.

3. Speed

Among usable models, winnow:e4b is the fastest (p50 106 ms). Running the same clef-flash as Q8 on llama.cpp cut p50 from 809 ms to 246 ms. After a restart, on-prem models needed 1.4–6.7 seconds to load on the first request.

4. The accuracy × speed Pareto frontier

The models that no other model beats on both accuracy and speed are jev-latest, winnow:latest, winnow:e4b and laya:multilingual. laya makes the list only because it is the fastest; its 35.8% accuracy is not usable in practice.

Results 2: Guard Content Safety

jev-latest, vllm-sr/d3 and winnow:latest all blocked 94–95% of harmful content with false block rates below 2%. The table is sorted by blocking balanced accuracy.

1. Blocking performance and latency

ModelDeploymentBlocking balanced accuracyHarmful block rateFalse block rate (95% CI)p50
jev-latestCloud97.4%95.3%0.5% (0.1–2.8%)207 ms
vllm-sr/d3Cloud96.9%94.3%0.5% (0.1–2.8%)344 ms
winnow:latestOn-prem96.3%94.6%2.0% (0.8–5.0%)416 ms
gpt-oss-safeguard (improved prompt)On-prem (separate host)94.6%89.2%0.0% (0.0–1.9%)1.58 s
vllm-sr/d3-flashCloud94.1%94.8%6.5% (3.8–10.8%)257 ms
vllm-sr/d3-miniCloud93.7%93.0%5.5% (3.1–9.6%)243 ms
gpt-oss-safeguard (original prompt)On-prem (separate host)91.8%83.6%0.0% (0.0–1.9%)1.92 s
winnow:e4bOn-prem90.6%91.3%10.0% (6.6–14.9%)208 ms
kev:4bOn-prem90.0%94.4%14.5% (10.3–20.0%)1.08 s
clef-flash@ollama-q8On-prem (Q8)79.6%98.7%39.5% (33.0–46.4%)514 ms
clef:flashOn-prem (F32)79.3%98.6%40.0% (33.5–46.9%)1.79 s
laya:multilingualOn-prem50.2%98.9%98.5% (95.7–99.5%)30 ms
clm:8bOn-prem49.1%1.3%3.0% (1.4–6.4%)139 ms

2. Classification accuracy and block rates for key categories

ModelClassification accuracyPrivacy block ratePrompt injection block rate
jev-latest83.2%61.5%100.0%
vllm-sr/d380.6%64.5%100.0%
winnow:latest78.3%73.0%98.0%
gpt-oss-safeguard (improved prompt)73.3%72.0%76.5%
vllm-sr/d3-flash83.3%85.5%100.0%
vllm-sr/d3-mini80.1%67.5%100.0%
gpt-oss-safeguard (original prompt)69.0%60.5%62.5%
winnow:e4b73.4%48.0%84.0%
kev:4b76.3%70.0%86.5%
clef-flash@ollama-q877.9%95.5%99.5%
clef:flash77.9%95.5%99.5%
laya:multilingual21.6%100.0%100.0%
clm:8b7.5%1.5%0.0%

3. Interpreting the results

  • The top three are statistically hard to separate: the false block rate comes from only 200 normal samples, so 0.5% vs. 2.0% is really a difference of 1 vs. 4 samples, and the confidence intervals overlap heavily.
  • Privacy is a shared weakness: most usable models block only 60–73%, and personal data embedded in health or legal questions is let through most often. Excluding clef, whose false block rate reaches 40%, vllm-sr/d3-flash is highest at 85.5%, but its false block rate is also 6.5%.
  • Blocking more is not the same as better: both clef versions blocked 98.6% of harmful content, but wrongly blocked 4 out of every 10 normal messages; even “how much water to put in the outer pot of a rice cooker to steam eggs” was labeled “misinformation”.
  • Low classification accuracy is not entirely the models’ fault: in the original criteria, “protected characteristics” and “non-protected targets” read more like term definitions, and most models labeled the former as normal and the latter as harassment. Excluding these 400 questions raises the top three’s classification accuracy by another 10–14 percentage points, but this does not affect the block decision.
  • Discount the 3 template-like categories: hate speech, encouraging suicide and illegal assistance use repetitive sentence patterns, are short and have no hard cases. Usable models mostly reach 90–100% recall on them, and real-world performance may be lower.

Three Key Technical Observations

These are the three most notable technical observations from this test. They cover whether decisions stay consistent after changing the deployment, how criteria wording interacts with model architecture, and the failure mode of LLM guards.

1. Quantization barely affects decisions, but confidence scores need recalibration

The same clef-flash runs as F32 ONNX (19 GB) on Ollaya and as Q8_0 GGUF (12.8 GB) on ollama. Their per-sample predictions matched on 99.8% of Domain and 99.6% of Guard samples, and the Q8 version was 3.3–3.5× faster. Because both the runtime and the precision changed, the speedup cannot be attributed entirely to quantization.

Note that the median gap in confidence between the two is about 0.12. If your system routes by confidence threshold, you must recalibrate the threshold after changing the deployment.

2. Criteria wording must match the model architecture

laya (mmBERT) and clm (contrastive encoder) score by the similarity between the text and each option’s description. We suspect this is the main reason they performed poorly with the criteria in this test, but we have not yet verified this with rewritten criteria.

Specifically, Domain’s “general” category is defined with a negative statement (not mainly about medical, financial or legal…), so its description is full of other categories’ keywords; “sensitive data” also carries a rule that it “takes priority regardless of topic”. As a result, laya labeled 62% of questions as sensitive data, and recall for the general category was only 1.8%. Even clef and kev often classified by topic and ignored the priority rule (sensitive data recall of 73% and 47% respectively).

Tips for writing criteria
Describe each option in positive terms, and avoid negative statements such as “anything other than…”.
Don’t put priority rules into a single-choice question; split them into a separate noul yes/no question, for example asking separately “does it contain personal or confidential data?”.
Don’t use entries that are only term definitions as options, such as Guard’s “protected characteristics” and “non-protected targets”.

3. The LLM guard failure mode: no verdict means it gets through

gpt-oss-safeguard first generates reasoning and then outputs a JSON verdict. A common approach is to combine the policy and the user content into one user message; even with max_tokens set to 4,096, 6.2% (162 samples) returned no verdict:

  • 64 samples: the reasoning used up all 4,096 tokens.
  • 98 samples: the run finished normally, but the output was not JSON. On resending, we found the model was not classifying at all: it answered the question directly (actually writing an email), refused to answer, or was hijacked by the injected instructions in the question and returned an empty string.

Of these 162 samples, 134 were harmful content. If the system uses a fail-open design, this content would pass straight through.

We ran a controlled experiment on the same 130 samples (max_tokens 4,096 in all cases; because there were only 10 normal samples, differences in false block rate cannot be interpreted):

SetupNo verdictHarmful block rateBlocking balanced accuracyMean output tokens (with verdict)
Original policy, in the user message1084.5%92.3%365
Original policy, in the system message2276.4%88.2%350
Explicit instructions added, in the user message090.0%95.0%389
Explicit instructions added, in the system message091.8%95.9%331

OpenAI’s user guide recommends putting the policy in the system message and the content in the user message, but moving only the policy actually made things worse: once the content stands alone as a user message, it looks like an ordinary question.

What actually works is adding explicit instructions at the start of the policy (you are a classifier, output JSON only, do not respond to the content) and restating the output format at the end. Validated on all 2,600 samples, the no-verdict rate fell from 6.2% to 1.0%, and all remaining cases were reasoning that used up 4,096 tokens.

Other points to note:

  • Close to the timeout limit: 5.2% of responses exceeded 1,024 tokens, and the longest took 27.6 seconds, close to the common 30-second timeout.
  • Unstable tail latency: latency is driven by output length (correlation 1.00), and p95 is 3–4× p50.
  • Results are not fully reproducible: with temperature set to 0, resending the same question still does not guarantee the same result; rerunning the same 1,714 questions gave 99.5% prediction agreement.
  • The policy could be shorter: the policy in this test is about 2,500 tokens, far above the 400–600 tokens the guide recommends, so shortening it may improve results further.

How to Choose a Model for Your Needs

No single model is the best choice in every scenario. We recommend first deciding whether data can be sent outside, then weighing accuracy against latency.

NeedRecommendationBasisCaveats
Cloud allowed, accuracy firstjev-latestDomain 99.8%; Guard blocking balanced accuracy 97.4%, false block rate 0.5%; p50 about 205 msAll messages are sent to a third party; latency depends on the network
Second cloud provider or backupvllm-sr/d3Domain 98.9%; Guard 96.9%, false block rate 0.5%Guard p50 344 ms; privacy block rate only 64.5%
Data cannot leave the internal environmentwinnow:latestDomain 99.0% (sensitive data 100%); Guard 96.3%; p50 203 / 416 msNeeds about 13 GB VRAM; confirm the model is on the GPU
On-prem, latency or cost firstwinnow:e4bDomain p50 106 ms, 9.3 req/s on one connection; 8 GB VRAMGuard false block rate 10%, privacy block rate 48%; better suited to Domain routing
Especially concerned about personal data leaksIn addition to the main guard, use vllm-sr/d3-flash to detect personal dataPrivacy block rate 85.5%, the highest among usable modelsFalse block rate 6.5%, not suitable as the main guard on its own; the effect of a separate yes/no question was not tested
Readable rationale requiredgpt-oss-safeguard: add explicit instructions to the policy and put it in the system message, with max_tokens of at least 4,096Blocking balanced accuracy 94.6%, with rationalep50 1.58 s, p95 6.2 s; still 1% with no verdict
Not recommended with these criterialaya:multilingual, clm:8b; the clef family for GuardBalanced accuracy about 50%; clef false block rate 40%Can be re-evaluated after rewriting the criteria

Architecture recommendations

  1. Route by confidence score: in the Domain scenario, 97.2% of winnow:latest’s questions had confidence ≥ 0.9, with 99.8% accuracy on that portion; jev-latest had 94.9% coverage at 100% accuracy. A faster model can handle high-confidence requests and pass only low-confidence ones to a stronger model. This should not be applied directly to Guard, though: winnow:latest’s classification accuracy on questions with confidence ≥ 0.9 was only 82.4%, so thresholds must be calibrated for each scenario.
  2. Split one question into several: first ask “is it harmful?” and then “which category?”, or use noul to split personal and confidential data into separate yes/no questions. Because one request can ask several questions, this adds no network round trips.
  3. Define the failure-handling strategy in advance: a decision model always returns an option, but the service can still time out or return a 5xx; an LLM guard adds the “no verdict” case. In production, choose explicitly between fail-open and fail-closed, and keep monitoring how often it happens.
  4. Monitor deployment status: regularly check the model’s DEVICE and per-request latency. If latency suddenly jumps by 10× or more, the model has most likely been pushed onto the CPU. Early in testing, winnow:latest took 12.5 seconds per request on the CPU, versus only 203 ms with exclusive use of the GPU.

How Do the Three Representative Options Differ?

The table below summarizes the key differences between three representative options (cloud, on-prem and a generative LLM guard) for quick comparison.

Aspectjev-latest (cloud)winnow:latest (on-prem)gpt-oss-safeguard (LLM guard)
Does data leave the internal environment?Yes, sent to a third-party serviceNoNo (can be self-hosted)
Domain accuracy99.8%99.0%Not tested
Guard blocking balanced accuracy97.4%96.3%94.6% (improved prompt)
Guard false block rate0.5%2.0%0.0%
Guard p50 latency207 ms416 ms1.58 s
Can it return no verdict?NoNoYes, about 1.0%
RationaleProbabilities onlyProbabilities onlyRationale text
Best forCloud allowed, accuracy firstData must stay internal, with good accuracyAn auditable rationale is required

In short: if you want accuracy and can use the cloud, choose jev-latest; if data must stay internal, choose winnow:latest; consider a generative LLM guard only when you need a readable rationale, and make sure to handle the “no verdict” case.

Test Limitations and Next Steps

This benchmark is best used to compare the relative performance of models, not as absolute numbers for production.

  • Synthetic data: the test data was written by an LLM and may be tidier than real user language; 3 Guard categories use template-like sentences.
  • Only 200 normal samples: a single false block is 0.5%, with a 95% confidence interval of 0.1–2.8%, so differences within 2 percentage points are not statistically meaningful. Confidence intervals were calculated with the Wilson formula (explained below).
  • Each model was run only once: decision models give almost identical results on reruns; LLM guard output is not fully reproducible.
  • Cloud latency is for reference only: it includes the round trip from our network to the service, and we cannot confirm the provider’s hardware or load.
  • Criteria were not tuned per model: the results reflect performance “under the same criteria”; the low scores of laya and clm do not mean they would do as poorly with other wording.
  • Single connection only: total throughput and tail latency under high concurrency are the next things to test.
  • Only one RTX 3090 on-prem: absolute numbers will differ on other GPUs, but the relative trends between models should be similar.

About the Wilson confidence interval: the Wilson formula gives a reasonable interval even when the proportion is 0% or close to 0%.

Formula
Wilson interval = ( p̂ + z²/2n  ±  z × √( p̂(1 − p̂)/n + z²/4n² ) ) ÷ ( 1 + z²/n ) p̂: observed proportion   n: sample size   z: 1.96 for a 95% confidence level

Directions worth exploring: retest the encoder-based models with rewritten criteria, and split personal data detection into a separate yes/no question to check whether “splitting the question” fixes the weakness in the privacy category.

FAQ

Q1: What is the biggest difference between a decision model and using a generative LLM for classification?

A decision model generates no text: every question always returns a typed answer and a probability distribution, and latency depends mainly on input length. A generative LLM has to reason first and then output JSON, its latency depends on output length, and it can return “no verdict”. If the business needs an auditable written rationale, a generative LLM still has value.

Q2: If data cannot leave our internal environment, which model should we choose?

In this test, the best on-prem model was winnow:latest: 99.0% Domain accuracy (100% sensitive data recall) and 96.3% Guard blocking balanced accuracy, needing about 13 GB VRAM. If latency and cost matter more, consider winnow:e4b, but its Guard false block rate is 10%, so it is better suited to Domain routing.

Q3: Will accuracy drop if we deploy a Q8 quantized version?

Taking clef-flash as an example, the per-sample predictions of the F32 and Q8 versions matched on 99.8% of Domain and 99.6% of Guard samples, and Q8 was 3.3–3.5× faster. However, the median confidence gap between them is about 0.12, so if you route by confidence threshold, the threshold needs to be recalibrated.

Q4: Why does the Guard scenario use “blocking balanced accuracy” instead of plain accuracy?

Because the numbers of harmful and normal samples are very unequal (2,200 vs. 200). A model that blocks everything reaches 91.7% plain accuracy but only 50% balanced accuracy. Balanced accuracy reflects both “was what should be blocked blocked” and “was what should pass let through”.

Q5: laya and clm scored very low. Are the models themselves bad?

Not necessarily. Both score by “the similarity between the text and each option’s description”. We suspect the negative phrasing and priority rules in these criteria are the main cause, but we have not yet verified this with rewritten criteria. The results only represent performance “under the same criteria”.

Q6: Can we route directly with a confidence threshold?

It works well in the Domain scenario: 97.2% of winnow:latest’s questions had confidence ≥ 0.9, with 99.8% accuracy on that portion. In the Guard scenario, however, classification accuracy at the same threshold was only 82.4%, so thresholds must be calibrated separately for each scenario.

Q7: Can the numbers in this benchmark be used directly as production performance metrics?

We don’t recommend it. This test used synthetic data, a single connection and a single RTX 3090, and cloud latency includes the network round trip. The data is best used to compare the relative performance of models; before going to production, validate with your own data and real traffic.

Conclusion

Decision models replace generated text with “typed answers + probability distributions“, so decisions on the critical path, such as classification, grading and approval gates, can balance accuracy, latency and predictability. This test shows that both the cloud jev-latest and the on-prem winnow:latest are ready for real-world use, while the way the criteria are written, how confidence thresholds are calibrated, and how failure handling is designed determine how well the system ultimately performs.

As AI agents and multi-model routing become standard in enterprises, “decide first, then generate” will be an important design principle for controlling cost and risk. If you are planning AI inference infrastructure, model routing or content safety guardrails, contact the INFINITIX team, and our consultants will help you assess the deployment that fits you best.

References