XSci

Lost in Rounding: bf16 Recurrent State Makes Hybrid Reasoning Models Think Longer

Yuchuan Weng, Xin Chen, Chi Liu

Submitted by Yuchuan Weng

Published November 13, 2026 · Version v1, November 13, 2026 · DOI 10.66977/xsci.2611.0020

Artificial Intelligence

Abstract

Hybrid reasoning models interleave attention with recurrent mixers such as Mamba-2 and Gated DeltaNet, whose state is cached between decoding steps. Prior work showed that storing this state in FP16 or INT8 with round-to-nearest (RTN) makes Mamba-2 and GDN hybrids more verbose, and that stochastic rounding (SR) fixes it. We study the other 16-bit format that serving engines expose, bf16, on six open hybrid reasoning models and 590 competition-math problems. With bf16 RTN state, five of six models reason longer (geometric-mean length 1.07x to 1.27x; also under greedy decoding in the two models tested) and hit the 32K token cap more often. On Qwen3.5-9B, cap hits rise from 118 to 223 and accuracy falls from 85.6% to 70.8%. On LiveCodeBench, both models tested also reason longer and lose 12.6 and 13.4 points of pass@1. Rerunning the capped items with a 128K budget halves the accuracy gap but does not close it (93.6% vs. 86.6%): bf16 runs that never finish are twice as common, and 85% of them are exact repetition loops. On Nemotron-Nano-9B-v2, by contrast, 30 of 31 extra caps finish at 128K, so the cost there is mostly extra tokens (exploratory). Same-width bf16 SR removes the length increase in four of five affected models, restores Qwen3.5-9B to 87.6% and brings code length and pass@1 near FP32 levels, so the failure is mostly rounding bias, not missing resolution. With an integer-hash random generator, SR on Qwen3.5-9B matches FP32 on cap hits and accuracy (113 vs. 118; 85.6% for both) at the memory and decoding speed of bf16 RTN. In the Nemotron-Nano-v2 lineage, SR on 25% of heads (an intermediate-decay band) removes the effect, while random heads and the top teacher-forced-KL octiles do not; in Nemotron-3 and Qwen3.5-9B the effect is spread across heads. We recommend that evaluations of hybrid models report the state dtype, cap rate and output length alongside accuracy.

View PDF

Review reports

xPeerd (AI review)

November 13, 2026

AI Review (xPeerd — DAReview)

Summary

Lost in Rounding: bf16 Recurrent State Makes Hybrid Reasoning Models Think Longer studies whether storing recurrent state in bf16 with round-to-nearest (RTN) alters the behavior of hybrid reasoning models relative to FP32. The manuscript’s main claim is that bf16 RTN lengthens reasoning traces in five of six tested hybrid models, increases token-cap hits, and can reduce task accuracy, while same-width stochastic rounding (SR) largely removes the effect in four of the five affected models. The strongest empirical case is Qwen3.5-9B, where cap hits rise substantially and accuracy drops under bf16 RTN, with many unfinished runs becoming exact repetition loops; reruns at 128K suggest part of the loss is due to longer-but-finite reasoning and part to non-termination. The paper is timely, technically focused, and generally well structured, and it contributes a useful caution for evaluation practice: state dtype, cap rate, and output length should be reported alongside accuracy. However, the evidence base is still constrained by limited domains, mostly single-seed main arms, partial throughput validation, and some unresolved model-specific exceptions, especially Nemotron-3. Several conclusions are strong but would benefit from tighter framing around external validity, reproducibility, and mechanism.

Potential Major Revisions

  1. Clarify the scope of the central claim and moderate generalization beyond the tested setup.

The abstract states that “With bf16 RTN state, five of the six models reason longer” and that “Same-width bf16 SR removes the length increase in four of the five affected models” while recommending that evaluations of hybrid models report recurrent-state dtype and cap statistics. This is an important practical observation, but the manuscript’s own limitations materially narrow its external validity: “Prompts are competition math for all six models and competitive programming for two; all runs use one engine version on one GPU type. Most arms have a single seed.” The statement “we recommend that evaluations of hybrid models report the state dtype, cap rate and output length alongside accuracy” is sensible, but the paper should more explicitly distinguish a robust engineering recommendation from a broader scientific claim about hybrid-model behavior in general.

A stronger version would explicitly define the study as evidence for a failure mode under the tested inference stack, workloads, and model families, rather than implying a general law of bf16 recurrent-state behavior. This matters especially because the manuscript itself reports heterogeneity: “Nemotron-3 is the exception” and “The Falcon-H1R null is unexplained.”

  1. Strengthen reproducibility and auditability of the pre-registration claims.

The manuscript repeatedly emphasizes that key tests were pre-registered: “Both predictions were pre-registered,” “The length, SR and cap tests were pre-registered,” and “Key tests were pre-registered: we wrote predictions and pass/fail criteria in a working notebook before running each arm.” Yet the manuscript also concedes that “The notebook is not externally timestamped.”

This weakens the evidentiary force of the pre-registration language. I think the paper should either:

  • soften the rhetoric around pre-registration and call it an internal prospective analysis plan, or
  • provide a stronger audit trail in the appendix or supplement, such as commit hashes, dated logs, server logs, notebook snapshots, or script manifests.

Without a more verifiable trail, claims of confirmatory status versus exploratory status remain partly trust-based. Since the paper relies on this distinction to support several mechanism claims, the manuscript should address this directly.

  1. Resolve or better delimit the mechanistic interpretation “bias, not resolution.”

One of the headline claims is: “the failure is mostly rounding bias, not missing resolution.” This is argued from the observation that “Same-width bf16 SR removes the length increase in four of the five affected models” and “removes the effect as fully” on Qwen3.5-9B with an integer-hash generator. While persuasive for those models, the paper also reports that “Nemotron-3 is the exception... a bulk shift of about ×1.05 remains, and it also appears when comparing against a second FP32 seed. We do not know its cause.”

That unresolved exception is not minor because it bears directly on the proposed mechanism. If one affected model retains a shift under unbiased same-width rounding, then the interpretation “mostly bias, not resolution” is plausible but incomplete. I recommend reframing the mechanistic conclusion more cautiously, e.g., that rounding bias explains most of the effect in most tested models, while at least one model shows an additional unexplained component. This would better align the claim with the evidence.

  1. Expand the methodology details for exact-loop detection and non-termination analysis.

A key empirical contribution is the distinction between “longer reasoning that finishes when given more budget” and “runs that do not terminate at any tested budget.” The paper defines an exact loop as a case where “its tail repeats with a fixed period for at least two periods and 192 tokens.” This is useful, but the manuscript should give more methodological detail on:

  • the search range for periods,
  • whether detection is exact string repetition or token repetition,
  • sensitivity of findings to the 192-token threshold,
  • whether near-loops or semantically repetitive but not exact loops were examined.

Because the non-termination claim is central, the loop detector should be described rigorously enough for independent reproduction and robustness assessment. At present, the operationalization is too brief for a high-confidence methods audit.

  1. Better justify the inference from math and code benchmarks to model behavior more broadly.

The code-generation section is helpful, but the manuscript acknowledges: “We did not rerun the capped code samples at a larger budget.” Thus, for code tasks, the decomposition between extra budget needs and true non-termination is not established with the same care as in math. Yet the abstract states that “On LiveCodeBench, both models tested also reason longer and lose 12.6 and 13.4 points of pass@1.”

I think the code results should be framed more explicitly as corroborative rather than mechanistically resolved. If feasible, add the same 128K rerun analysis for capped code items; if not, the limitation should be elevated into the main-text interpretation rather than left as a brief note.

  1. Provide fuller methodological transparency for throughput and systems claims.

The abstract and Section 6 make a practically important claim that SR can match RTN cost under an integer-hash generator: “With an integer-hash random generator, SR on Qwen3.5-9B matches FP32 on cap hits and accuracy ... at the memory and decoding speed of bf16 RTN.” Section 6 adds that it “ran at 10,393 tokens/s end to end, against 10,163 for a same-day bf16 RTN run,” and at matched batch size “decodes no slower than that run.” But the limitations section notes that the hash variant was not rerun for Qwen3-Next or the code benchmark, and throughput behavior with default Philox is much worse for GDN kernels.

This is a strong engineering claim with immediate deployment relevance, so it should include enough implementation detail for replication: kernel versions, exact batch-bin methodology, whether measurements include prefill, server settings, and variance across repeated runs. Otherwise, the practical takeaway may be overstated relative to the evidence shown for only one model/system path.

  1. Address possible paper-mill or fabrication concerns: low concern, but strengthen traceability.

I do not see strong paper-mill features. The manuscript is technically coherent, contains model-specific exceptions rather than uniformly positive results, and reports failed or mixed hypotheses, e.g., “Nemotron-3 is the exception,” “The Falcon-H1R null is unexplained,” and “The notebook is not externally timestamped.” These features argue against obvious fabrication. However, because most cited work is very recent preprint literature and repository pull requests, and because some central confirmatory claims rely on internal pre-registration, the paper would benefit from a more explicit reproducibility package. That would reduce any residual concern about the traceability of claims.

Potential Minor Revisions

  1. Tone down overconfident wording where the manuscript’s own evidence is mixed.

The sentence “so the failure is mostly rounding bias, not missing resolution” is sharper than the evidence warrants given the Nemotron-3 exception. Consider revising to “in most tested models, the failure appears primarily attributable to rounding bias.”

  1. Clarify ambiguous phrasing around “same-day” comparisons.

The statement “a same-day bf16 RTN run” is operationally vague. Please specify what controls were held fixed and whether any run-to-run variance was estimated.

  1. Improve terminology consistency for model lineage names.

The manuscript alternates among “Nemotron-Nano-v2,” “Nemotron-Nano-9B-v2,” and “Nemotron-Nano-v2 lineage.” Standardizing naming would improve readability.

  1. Clarify whether “reason longer” refers to chain-of-thought text length, total generated tokens, or all output tokens.

Much of the manuscript uses “length” and “verbosity” in ways that are understandable but not always formally defined in the quoted sections. Given the evaluation sensitivity to token caps, a precise definition should appear early and remain consistent.

  1. Tighten the explanation of selected subsets.

The manuscript correctly warns that “This subset is selected and near ceiling.” Since this is central to the interpretation, a short formal explanation of the selection effect in methods or an appendix would help readers who are less familiar with truncation-induced conditioning artifacts.

  1. Add fuller detail for the prior-art search limitation.

The manuscript says: “Our prior-art search covered arXiv and the SGLang and vLLM repositories only.” This is appropriately candid, but it should likely be moved from a brief limitations remark to a clearer statement in the methods or related-work section, because novelty claims depend on search scope.

  1. Typographic and formatting issues.

I did not identify many obvious typographic errors from the retrieved text snippets. The manuscript appears relatively clean at the sentence level in the sections visible through file search. However, there are formatting artifacts in extracted text such as broken hyphenation or character joins around “reproducibility,” “mixed-precision,” and section/page transitions, though some of this may be PDF extraction rather than manuscript formatting. For example, extracted snippets show splits such as “hybrid mod￾els” and “quantiza￾tion,” suggesting the authors should verify PDF typesetting, ligatures, and line-break behavior in the submitted version.

  1. AI content analysis for this post-2021 manuscript.

Estimated percentage of AI-generated content: low to moderate probability, approximately 10–25%.

Rationale: the prose is unusually uniform, compressed, and rhetorically controlled, with repeated contrastive structures such as “X shows..., Y hides this,” “bias, not resolution,” and frequent disciplined framing of confirmatory versus exploratory analyses. However, the manuscript also contains field-specific technical detail, negative results, caveats, and model-specific heterogeneity that are less typical of unedited generic AI drafting. The text does not show obvious large-scale hallucinated structure or generic filler in the retrieved excerpts. Relevant passages that might trigger AI-style suspicion are formulaic summary sentences such as “We recommend that evaluations of hybrid models report the state dtype, cap rate and output length alongside accuracy” and “We read this as a lead for future work, not a finding.”

Assessed epistemic impact: likely limited if AI tools were used for stylistic drafting only, because the manuscript’s contribution depends on empirical correctness, methodological transparency, and reproducibility rather than prose originality alone. The larger concern is not AI wording per se, but whether all reported experiments, pre-registered criteria, and implementation claims can be independently audited.

Recommendations

  1. Reframe the headline conclusion to distinguish:
  • robust empirical findings within the tested setup,
  • practical deployment recommendations,
  • tentative mechanistic interpretation.
  1. Add a reproducibility supplement containing:
  • inference engine versions and relevant flags,
  • kernel implementation details,
  • exact loop-detection pseudocode,
  • seed and batching information,
  • prompts or benchmark settings,
  • commit hashes or dated artifacts supporting the prospective analysis plan.
  1. Extend the higher-budget rerun analysis to code tasks if feasible, or explicitly present the code results as corroborative but not mechanism-resolving.

  2. Revise the “bias, not resolution” argument to account explicitly for Nemotron-3 and any other unresolved exceptions.

  3. Expand the discussion of external validity. The manuscript should say plainly that the observed effects are strong in the tested workloads and model families, but broader claims across hybrid architectures, hardware, engines, and prompt regimes remain to be validated. This is already partly acknowledged in “all runs use one engine version on one GPU type” and should be integrated into the main interpretation.

  4. Preserve the candid negative and mixed findings. They are a strength of the paper, not a weakness, because they increase credibility and make the work more useful to practitioners.

Overall, I judge the manuscript technically interesting, coherent, and potentially impactful for inference evaluation and deployment practice, but it would benefit from clearer delimitation of claims, stronger reproducibility scaffolding, and a more cautious mechanism narrative around unresolved exceptions.

Reviewer3 (AI review)

November 13, 2026

AI Review (reviewer3.com)

Overview

Identification of bf16 State Rounding Pathology

Contribution

The authors show that storing hybrid model recurrent states in bf16 with round-to-nearest rounding causes systematic reasoning output lengthening, increased token-cap hits, and exact repetition loops across multiple architectures.

Efficient Mitigation via Stochastic Rounding

Contribution

This study demonstrates that applying same-width stochastic rounding using an integer-hash random generator eliminates output lengthening and recovers FP32-level reasoning accuracy without increasing memory footprint or decoding latency.

Analysis of Closed-Loop Head Sensitivity

Contribution

The authors reveal that teacher-forced KL divergence misidentifies which recurrent heads drive closed-loop reasoning runaway, establishing that rounding bias rather than lack of numerical resolution primary causes performance degradation.

Comments (8)

1. Qwen3.5-9B all-heads F_log inconsistent across Table 3 and Section 8 — Editorial

Here too the effect looks spread, with the share removed roughly following the share of heads with SR (exploratory; all heads: 0.97). The concentration in one decay band is a property of Nemotron-Nano-v2, not

In Section 8 ('Transfer'), when discussing head-localization experiments on Qwen3.5-9B (GDN), the text reports that applying SR to 'all heads' yields F_log = 0.97: 'Here too the effect looks spread, with the share removed roughly following the share of heads with SR (exploratory; all heads: 0.97).' However, according to Table 3 and Section 6, the Qwen3.5-9B GDN SR arm (using Triton's default Philox generator, which is the generator used unless specified as the hash generator) has F_log = 0.87 [0.58, 1.26], whereas 0.97 [0.70, 1.37] is the F_log for the separate 'hash generator' rerun. Either Section 8 is quoting the hash generator F_log while the band/subset arms used the Philox generator (or vice versa without explanation), or it conflates the two all-head F_log values.

2. Sum of single-octile KL shares exceeds 100% and contradicts prose — Editorial

The slowest octile q 0 accounts for the most (49%), then q 1 (37%) and q 2

In Table 5 and Section 8, 'KL share' is defined as 'the share of the bf16 teacher-forced KL at positions beyond 4K tokens that is removed by keeping the octile in FP32', and footnote ‡ states that the multi-octile values (52% for q1+q2 and 86% for q0+q1) are the 'sum of the two single-octile shares, not measured jointly.' However, the single-octile KL shares reported for the slowest three octiles alone—q0 (49%), q1 (37%), and q2 (15%)—already sum to 101% (49% + 37% + 15% = 101%), even before accounting for octiles q3 through q7 (and in Section 8, keeping q3 in FP32 removes 14% of the 'Alternatively' marker excess). Furthermore, Section 8 prose states 'The slowest octile q0 accounts for the most (49%), then q1 (37%) and q2 (15%)', treating these single-octile removal fractions as additive shares of a total, which is mathematically inconsistent with their sum exceeding 100%.

3. Spatial correlation across elements within a decode step not evaluated — Major

Per-step unbiasedness does not imply independence across steps, so we checked that separately in a CPU emulation of the kernel’s formula ( 2 20 elements, 256 steps, criteria set before running): the correlation between an element’s draws at lags of 1 to 4 steps is at most 4 . 5 × 10 − 5 (limit 2 . 5 × 10 − 4 ), the variance of each element’s 256-step sum is 1.002 times its independent-draw

The CPU emulation validates the lowbias32(i * 0x9E3779B1 + seed) generator for single-element temporal independence across lags 1 to 4, but does not report cross-element spatial correlation across the state matrix within the same decode step (where all elements i share the same seed and differ only by the linear stride i * 0x9E3779B1 before a single 32-bit mixing function). Since Gated DeltaNet updates a matrix state per head where coherent noise across state dimensions could interact with key/value projections, verifying negligible cross-element correlation at fixed seed would strengthen the RNG validation.

4. Request concurrency confound not directly controlled in three models — Minor

Because an FP32 state doubles cache size, bf16 arms can admit twice as many concurrent requests; we control for this on Nemotron-3 (§4), and on Qwen3.5-9B the cap never binds (Appendix B). Each arm is one seed unless stated.

While request concurrency is explicitly matched on Nemotron-3 and shown not to bind on Qwen3.5-9B (where 590 items across 8 ranks is 74 requests per rank, below the 122 cap), in Nemotron-Nano-9B-v2 (cap 44 vs. 88), Nemotron-Nano-12B-v2 (cap 38 vs. 76), and Qwen3-Next (TP4xDP2, 295 requests per DP group vs. cap 223/436) the FP32 running-request cap binds during the 590-item batch while the bf16 cap is twice as large. Although the SR arms match the bf16 RTN concurrency cap and thus isolate the rounding effect relative to SR, the direct FP32-vs-bf16 RTN comparisons for those three models still differ in batch concurrency.

5. Signed-value bit manipulation for negative numbers and subnormals — Major

GDN decode kernel. When enabled, the kernel adds 16 random bits below the bf16 mantissa of each FP32 state value and then truncates, so the value is rounded up with probability equal to its fractional position between the two bf16 neighbours (at 16-bit resolution).

The integer bit-manipulation formula bits = (bits + (rand32 & 0xFFFF)) & 0xFFFF0000 is described as rounding a value up with probability equal to its fractional position between its two bf16 neighbours. Because IEEE-754 floating-point numbers use sign-magnitude representation, adding unsigned random bits to the magnitude and truncating always rounds away from zero with probability proportional to the fractional magnitude—which is mathematically unbiased (rounds toward -infinity with that probability for negative numbers, and toward +infinity for positive numbers), but technically rounds down in value rather than 'up' when h < 0, and can overflow into the sign bit if h is NaN or +Inf. Clarifying the sign-magnitude behaviour and confirming that recurrent states do not encounter subnormal underflow/NaN edge cases would make the kernel specification more precise.

6. Right-censoring at the token cap compresses length ratios — Major

Length ratio : geometric mean of paired per-item output-length ratios, with a Wilcoxon signed-rank test on log ratios. Capped : the run stopped at the token cap.

Because runs that hit the 32,768-token cap are truncated at 32,768 tokens andbf16 RTN hits the cap more frequently than FP32, the geometric-mean output-length ratio is right-censored and systematically underestimates the true unconstrained length increase (especially on models like Qwen3.5-9B where 38% of bf16 runs cap). Reporting a survival-analysis estimator (e.g., restricted mean or median time-to-completion from Figure 2) alongside the raw truncated geometric mean would prevent readers from comparing length ratios across models with vastly different censoring rates.

7. Asymmetric intervention between teacher-forced KL and closed-loop tests — Major

two single-octile shares, not measured jointly). KL shares keep an octile in FP32, whereas the closed-loop arms apply SR to it.

As noted in the caption of Table 5, the teacher-forced KL shares are measured by keeping a single octile in FP32 while the rest use bf16 RTN, whereas the closed-loop generation experiments apply bf16 SR (not FP32) to the selected octiles. Although all-head SR removes 94% of the KL divergence, it would be cleaner to verify that single-octile teacher-forced KL rankings are identical when the octile is placed in bf16 SR rather than FP32 so that precision resolution and unbiased rounding are not conflated in the q0 vs. q1+q2 comparison.

8. Rerunning from scratch conflates sampling variance with budget continuation — Major

settings (Table 2). Reruns sample from scratch rather than continue the 32K runs, identically in both arms, so they ask whether an item can finish, not how a given run would have continued. Both pre-registered tests

In the 128K budget experiment, items that hit the 32K cap were resampled from scratch at temperature 0.6 rather than continuing the exact 32K trajectories or using deterministic greedy decoding. Because a fresh random trajectory can finish in under 32K tokens simply due to lucky sampling (as shown by the FP32 median rerun length of 19,951 tokens on items that previously exceeded 32,768 tokens), this operationalization overestimates how many 32K-capped traces would genuinely finish if given a larger budget.