Can the Confidence and Decisions of System 1 Models Be Trusted? An Explainability and Perturbation Audit of JevStyle Decision Models
Julien Delaunay
Abstract
"System 1" decision models (pioneered by TypeSafe AI's Jev) replace autoregressive text generation with a single forward pass returning typed probability distributions over declared schemas (Choice, Score, Noul). While deployed for low-latency routing and automated triage, their statistical calibration and post-hoc explainability remain uncharacterized. We introduce an open, deterministic audit benchmark evaluating System 1 decision models across calibration under label-preserving perturbations, cross-question interference, and post-hoc attribution faithfulness. Evaluating five open-weight models (0.8B to 27B) alongside generative System 2 baselines -- both single-token logprob scoring and structured JSON/CoT decoding -- across NLI (e-SNLI), adversarial syntactic shortcuts (HANS), synthetic clinical triage (N=900), prompt injection defense (Tensor Trust, N=250), and an 86-case logic-gate trigger suite yields five findings: (1) provider confidence is an exact closed-form dispersion transform (1 - H(p)/ln K on sub-10B models; normalized margin on Clef-27B) rather than empirical accuracy; (2) bundling multiple questions in one request induces directional cross-question interference (11.7% to 22.4% binary flips on Tev1-0.8B, vs. 0.1% to 0.2% on Tev1-4B); (3) System 1 models with >= 4B parameters are faster and better calibrated under syntactic shifts (ECE_w = 0.030 to 0.087) than logprob (0.144) and JSON (0.378) baselines; (4) for prompt-injection defense (N=250), >= 4B models achieve 0.988 to 0.998 AUROC with up to 95.2% recall and 0.0% false alarms at 48 to 60 ms; and (5) single-pass inference makes 512-coalition XAI interactive (1.3 to 3.2 s/input), while KernelSHAP with a logit link (S1-SHAP) mitigates softmax saturation on disjunctive OR gates (89.3% to 100.0% gate recovery vs. 57.1% to 82.1% in prob space; p_adj = 0.004).
Review reports
XSci (AI review)
Summary
The paper audits "System 1" decision models — Jev-style models that answer typed questions with a probability distribution in one forward pass — on three axes: what the returned confidence field measures, whether bundling questions into one request perturbs the answers, and whether perturbation attribution recovers the features driving the decision. Five open-weight decision models (0.8B–27B) are compared against Gemma4-8B in three decoding modes across e-SNLI, HANS, synthetic triage vignettes, a Tensor Trust prompt-injection cohort, and an 86-case planted-rule suite. The reported results are that confidence equals 1 − H(p)/ln K on sub-10B models and a normalized margin on Clef-27B; that bundling flips up to 22.4% of binary answers on Tev1-0.8B but 0.2% on Tev1-4B; and that logit-link KernelSHAP (S1-SHAP) improves faithfulness and gate recovery.
Contribution and Significance
The contribution that matters is measurement, not method. Two results are exact rather than statistical and have immediate consequences for deployment. First, the provider confidence field is a deterministic function of the distribution's dispersion, matching 1 − H(p)/ln K to 4.4 × 10⁻¹⁶ over 600 clean Choice predictions (Section 5.1) — so reading c(p) = 0.63 as a 63% chance of being right is a category error, and the E-AURC results show max p dominates c(p) for selective prediction on every model. Second, a multi-question request is not a set of independent queries: Table 5 shows that reversing the serialization order moves interference onto whichever head is second (Tev1-0.8B Noul flips 0.224 → 0.009), while a neutral dummy question of comparable length does not reproduce the effect (≤ 0.040 on Choice, ≤ 0.018 on Noul). That control is what turns an observation into a causal claim, and it is the best-designed part of the paper.
What is new is narrower than the framing suggests. The logit link for KernelSHAP is not a new method — Section 3.3.3 correctly attributes it to Lundberg & Lee (2017), where it is a standard option — so naming it S1-SHAP and listing it as a contribution overstates matters; the genuine finding is empirical, about when the link matters.
Strengths
- The confidence-field identity (Section 5.1) is verified to floating-point exactness across four models and separately characterized for Clef-27B; practitioners can act on it directly.
- The three interference controls in Table 5 — order reversal, option permutation, neutral dummy — rule out prompt length as the mechanism and establish directionality.
- The Planted System-Rule benchmark (Section 4.1) supplies prior-free boolean ground truth and shows that leave-one-out collapses under OR redundancy (Prec@3 = 0.643, Table 10) where coalition methods hold at 1.000.
- Table 11 is the best XAI finding here: log-odds linearization raises full OR-gate recovery from 0.571 to 0.893 on Multi OR, while probability space stays correct for AND gates (1.000 vs 0.920).
- The harness is deterministic, cached by weight digest and payload hash, and released with scripts that regenerate every table.
Recommendation
Major Revision for TMLR. The measurement work is substantial and the artifacts are valuable, but the named methodological contribution is in real doubt as stated: on every metric not expressed in log-odds, S1-SHAP is indistinguishable from KernelSHAP. Several headline numbers in the abstract, discussion and conclusion are unsupported by, and in places contradict, the tables, and the guardrail claim lacks the baseline that decides it. One added baseline, a link-neutral comparison, an arithmetic pass over the claims and a repositioning against existing work would close all of this.
Major Concerns
-
The S1-SHAP advantage is measured in the space the method is fitted in (Sections 5.4, 5.9; Tables 8, 9). The only significant gain over probability-space KernelSHAP is on log-odds comprehensiveness (p_adj = 0.0040); in probability AOPC the two are indistinguishable (p_adj = 0.606). On the link-neutral deletion AUC in Table 8, S1-SHAP wins once (0.180 vs 0.211, Triage Noul), ties once (0.196 vs 0.197) and loses twice (0.455 vs 0.359 on e-SNLI Noul; 0.130 vs 0.122 on Triage Choice); flag precision in Table 9 is 0.482 vs 0.483 and 0.486 vs 0.487. No significance test against KernelSHAP on any link-neutral metric is reported. An AOPC in log-odds is the objective the estimator minimizes, so a win there is close to tautological and should not be presented as a faithfulness improvement.
Two points compound this. The abstract credits S1-SHAP with "89.3%–100.0% pairwise interaction gate recovery", but those are Table 11's log-odds SII and ArchDetect estimators, not S1-SHAP as defined in Equations 1–2, which is first-order. Section 5.7 states this correctly and the abstract should follow it. And the cross-model XAI cohort is n = 24 instances scored on four models, where pooled p-values (6.5 × 10⁻¹⁵) differ from instance-level ones (6.5 × 10⁻⁶) by nine orders of magnitude; state whether p_adj = 0.0040 is pooled.
-
Headline numbers the results do not contain, or that contradict the tables. Section 6 reports interference "reaching up to 60–82% under exploratory uncalibrated prompts", which appears in no table, section or appendix; the conclusion gives "9B-27B heads (13.8-17.1%)" where Section 5.1 reports 9.2%, 15.2% and 16.9%. The conclusion also claims "≥ 4B System 1 models achieve … 1.3×-3.9× lower single-pass latency than first-token logprob scoring", but Table 14 gives Nimble-9B 286 ms and Clef-27B 828 ms against 74 ms for Gemma4-8B (1-Tok) — both substantially slower — and 3.9× corresponds to Tev1-0.8B at 19 ms, which is not a ≥ 4B model. Abstract finding (3) inherits this with "substantially faster", and finding (4) pairs "up to 95.2% recall" with "48–60 ms latency" when Table 14 attributes that recall to Clef-27B at 828 ms, the 48–60 ms models reaching 0.704 and 0.800.
-
The guardrail claim is not benchmarked against the alternative it would displace (Section 5.10, Tables 14–15). Abstract finding (4) positions ≥ 4B System 1 models as inline injection filters, but the only comparator is an 8B generative model — not something deployed for this purpose. The deployed alternative is a small fine-tuned encoder classifier: Meta's Llama Prompt Guard 2 (22M and 86M DeBERTa variants; see Inan et al., Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations, arXiv:2312.06674, 2023) reports roughly 0.99 AUC at around 20 ms, matching the 0.988–0.998 AUROC claimed here at lower latency and a fraction of the memory. Running one such classifier on the same N = 250 cohort would settle the claim.
-
Primacy claims do not survive against uncited adjacent work. Section 2.3 states that these properties "have never been audited" and that this is "the first end-to-end framework". Sun, Xu, Shi and Yang, Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to It, arXiv:2609.26758 (2026), audits Jev and two open-weight typed decision models and finds decisions track option-name polarity rather than the definitions bound to them — directly adjacent to the option-permutation control in Section 5.1, which reports 11.1% flips on Tev1-0.8B without connecting it to that mechanism. Also missing is the verbalized-confidence literature that is prior art for the Gemma4-8B JSON baseline (Tian, Mitchell, Zhou, Sharma, Rafailov, Yao, Finn and Manning, Just Ask for Calibration, EMNLP 2023).
Minor Comments
- Section 5.1 gives 11.7% Noul flips on the N = 900 cohort and 22.4% on the N = 450 subset for the same protocol without explaining the gap, while Sections 6 and 7 quote only the larger figure.
- Clef-27B and Gemma4-8B (JSON) use 150 instances per dataset where other models use 300, yet Table 13 bolds best values across both cohorts.
- Section 5.10's fifth finding presents S1-SHAP (0.228 Prec@g) as outperforming KernelSHAP (0.087) in the same sentence that reports LOO at 0.328.
- Every ≥ 4B model shows Triage ECE_w = 0.090–0.108 at 0.973–1.000 accuracy (Table 13) — marked under-confidence, never discussed — and that accuracy also suggests the vignettes are near-saturated as a task.
- Section 3.4 announces "five deterministic perturbation operators" and then enumerates three.
- Figure 6 and Section 5.6 label the Noul head's output "P(immediate)" although Section 3.2 defines the primitive as returning P(true | s, q_noul).
- Equation 5 and AUROC_c are given only for the Choice head, with "–" in every Noul row of Table 6; state whether the Noul head returns a confidence field and its closed form.
- The Score primitive is introduced in Sections 1 and 2.1 as one of three core primitives but never evaluated; say so in the scope statement.
- All five System 1 models are fine-tuned from a shared Qwen3.5 backbone (Sections 3.3.1, 4.2), so both structural findings may be lineage-specific; one non-Qwen decision model would test generality.
- Gemma4-8B is the only System 2 family; a fine-tuned encoder NLI classifier (for example DeBERTa-v3 on SNLI/MNLI) is the standard reference point for e-SNLI and HANS.
- Appendix C.3 gives 1025 ms for JSON with rationale and 8322 ms for CoT where Table 3 gives 1133 ms and 4856 ms, and the planted-rule regime counts differ between Section 4.1 and Appendix C.2.
- A copy-editing pass is needed: Section 2.1 contains the contentless sentence "At present, it is not a direct method for the generation of such a task.", and model names are corrupted in several tables ("C1ef", "Tevi").