XSci

Feedback That Backfires in Games: Do LLM Players Repeat the Move They Just Watched Fail?

Esmail Gumaan

Submitted by Esmail Gumaan

Published October 23, 2026 · Version v1, October 23, 2026 · DOI 10.66977/xsci.2610.002k

Artificial Intelligence, Machine Learning

Abstract

When an LLM agent's move fails, the harness writes the move and the failure message into the transcript and asks again. In tool calling, this record makes small models more likely to repeat the failed action. We test whether the same holds in games (Sokoban, Connect Four, Othello), where a move can fail by being illegal or by being legal but losing, both verified exactly. On fixed probe items built by perturbing a correct move, the record backfires: pooled over 8 instruction-tuned models (135M-1.7B) it makes the failed move more likely (corrective gain G = -1.38 nats per token), through the move's surface form, for illegal moves but not for losing ones, and not for every model. An exploratory analysis shows what sets the sign: after the record, the failed move's log-probability barely depends on how likely it was before (slope 0.18). Moves the model would not have played become more likely, and greedy decoding starts to produce them; moves it was about to play become less likely, but greedy decoding still repeats 38% of them. A description of the failure that omits the move string keeps the model's preference instead. Along the OLMo 2 1B post-training ladder the inversion is present in the base model and largely removed at the DPO stage. RAGEN's default models invert in RAGEN's own Sokoban format before any RL. A "do not repeat" instruction does nothing, a clean retry is the worst harness, and a decoder ban stops exact repeats but not illegal play. All measurements ran in float32 on a free Colab GPU, which first reproduced the original paper's number.

View PDF

Review reports

Reviewer3 (AI review)

October 23, 2026

AI Review (reviewer3.com)

Comments (11)

1. Model count and SmolLM2-1.7B direction mismatch in Section 10 — Editorial

tool calling. It is not a fix for every model: it lowers the failed move for 5 models (SmolLM2-135M, SmolLM2-360M, Qwen3-0.6B, SmolLM2-1.7B and Qwen3-1.7B) and raises it for Qwen2.5-0.5B and Llama-3.2-1B, models whose copy term is negative (Table 1). Section 6 adds a second caveat:

In Section 10 (page 12), the text states that across the 8 RQ1 models, the abstract description "lowers the failed move for 5 models (SmolLM2-135M, SmolLM2-360M, Qwen3-0.6B, SmolLM2-1.7B and Qwen3-1.7B) and raises it for Qwen2.5-0.5B and Llama-3.2-1B, models whose copy term is negative (Table 1)." This conflicts with Figure 5 (right) and Table 1 in three ways: (1) only 7 of the 8 RQ1 models are listed, omitting Qwen2.5-1.5B; (2) in Figure 5 (right), Qwen2.5-1.5B is actually one of the 5 models where the abstract description increases corrective gain G (from roughly -1.0 to -0.3 nats, thus lowering the failed move's probability), whereas for SmolLM2-1.7B the abstract description slightly decreases G (making the failed move more likely, not less); and (3) in Table 1, SmolLM2-1.7B also has a negative point estimate for the copy term (-0.19 nats/tok), just like Qwen2.5-0.5B (-0.33) and Llama-3.2-1B (-0.52).

2. Uncontrolled prior move occurrences in shared probe contexts — Major

A check found that the failed move is not always new to the context: in 15/80 of the Connect Four and 24/100 of the Sokoban decision points the failed move equals one of the agent's own earlier moves (in Sokoban, the deadlock push often continues the previous direction). Tool calls in the original never had this property; we record it per item and use it as a covariate.

In Section 4.2, the authors note that in 15/80 of the Connect Four and 24/100 of the Sokoban decision points, the failed move being probed already appears in the context as one of the agent's own earlier moves, and state that they record this per item and use it as a covariate. However, in the primary RQ1 analysis (Section 5, Tables 1 and 2), no adjustment, stratification, or covariate coefficient for prior context occurrence is reported. Because the paper's central mechanism is surface-form copying from context, items where the move string is already present in pre change the baseline copying state (moving from 1 to 2 occurrences rather than 0 to 1) and are disproportionately concentrated in Sokoban deadlock pushes, which the authors specifically note often continue the previous direction. The authors should report the RQ1 results (and particularly the illegal vs. losing comparison) excluding or explicitly controlling for items where the move already appeared in the prefix.

3. Differential weighting of models in pooled RQ1 estimates — Major

items are added per game; Appendix E gives the counts by failure type, source and operator. A run scores a prefix of one fixed order of the positions that cycles over failure types: 40 positions per game for the RQ1 models up to 0.6B and for the remedy study, 30 for the four larger RQ1 models, 20 for each OLMo 2 checkpoint, and 60 Sokoban positions in the RAGEN format. These sample sizes were

In Section 5, the authors report pooled corrective gain and decomposition estimates across the 8 instruction-tuned models over 840 (model, item) pairs. However, as described in Section 4.2, the 4 smaller models (up to 0.6B, which include the strongest inverters like Qwen3-0.6B at -5.45 and SmolLM2-135M at -1.64) were evaluated on 40 positions per game (120 items per model), whereas the 4 larger models (1B to 1.7B, which have much smaller or positive gains) were evaluated on only 30 positions per game (90 items per model). Pooling across all 840 pairs weights the smaller models 33% more heavily than the larger models and evaluates the two groups on different subsets of positions. The authors should report an equally weighted mean across the 8 models on the shared 30-position prefix (720 pairs) to ensure the pooled headline numbers are not skewed by unequal sample sizes across model scales.

4. Mathematical coupling and collinearity in the mechanism regression — Major

. Positive coefficients mean less inversion. The “+ prior” columns add the failed move’s log-probability before the attempt, which also enters G itself (see text).

In Section 8 (Table 4), the authors regress G(a^x) = log pi(a^x | C) - log pi(a^x | C + (a^x, o^x)) on notation, z-scored move length in tokens, and token overlap with legal moves, and in the '+ prior' columns add the prior log pi(a^x | C) as a covariate. Two statistical issues affect this specification. First, because the dependent variable is G = prior - post, adding prior to the right-hand side is algebraically identical to regressing -post on prior (with coefficient 1 - beta_prior); any measurement noise or model misspecification in prior mechanically drives a large positive coefficient (+0.81 and +0.92) and induces coupling bias in the remaining covariates. Second, move notation (short vs. verbose vs. index) deterministically sets move token length (1 token for short/index vs. 6-9 tokens for verbose), creating severe collinearity between the notation: verbose indicator and move length in tokens (z). This collinearity explains why notation: verbose has a positive coefficient (+0.53) while move length has a strong negative coefficient (-0.84) in the ladder, whereas in the RQ1 chat models notation: verbose is strongly negative (-2.12) and move length is zero (-0.03). The authors should avoid including both notation indicators and move length in the same linear model (or estimate length effects within notation) and should clarify that the '+ prior' regression is algebraically equivalent to modeling post-record log-probability directly.

5. Erosion of DPO gain during RLVR and Instruct in chat format — Major

RLVR (-0.03 [-0.06, -0.00]) and the final Instruct model (-0.01 [-0.03, +0.01]) change little after that. In their own chat template the post-trained checkpoints show the same step from SFT to DPO (+0.46 [+0.33, +0.59]).

In Section 7, the authors claim that after DPO, 'RLVR (-0.03 [-0.06, -0.00]) and the final Instruct model (-0.01 [-0.03, +0.01]) change little after that. In their own chat template the post-trained checkpoints show the same step from SFT to DPO (+0.46 [+0.33, +0.59]).' However, looking at the chat template column of Table 3, DPO reaches a positive G/tok of +0.16, whereas RLVR1 drops to -0.06 and Instruct drops further to -0.16—a loss of -0.32 nats/tok from DPO to Instruct that reverses two-thirds of the +0.46 DPO gain and returns the point estimate to feedback inversion. Thus, the statement that RLVR and Instruct 'change little after that' only holds in the completion format, not in the chat template that the post-trained models are actually deployed with. The authors should revise Section 7 to explicitly note that in the chat template, RLVR and Instruct erode most of the positive gain achieved at DPO.

6. Pooled OLS slope conflates between-model, between-game, and within-model variance — Major

the prior with slope one. It does not (Figure 3, left). Over all 2079 RQ1 items, shared and model- proposed, the per-token log-probability of the failed move after the verbatim failure record has a slope of +0.18 [+0.14, +0.21] on its prior; per model the slope lies between 0.02 and 0.35. After the record,

In Section 6, the authors estimate the preference-flattening slope (+0.18 for verbatim fail, +0.24 for valence-free, +0.75 for abstract) by pooling all 2,079 RQ1 items across 8 models and 3 games, where action space sizes and baseline per-token log-probabilities vary substantially across games and models, and where each position is repeated across multiple models and item sources. Although the text mentions that per-model slopes range from 0.02 to 0.35, pooling across models and games without fixed or random effects for model and game can distort the overall slope via ecological/Simpson's effects, and furthermore, the '<1%' and '>30%' subgroup cutoffs appear to use the 4-candidate normalised probability while the regression uses raw per-token log-probability. The authors should clarify which probability metric defines the '<1%' and '>30%' bins, and report the within-model, within-game slope from a mixed-effects or fixed-effects specification.

7. Pooled free-play repeat rate hides extreme model-by-game divergence — Major

6 (model, game) combinations. Averaged over those combinations, the share of failed moves that repeat an earlier failed move exactly is 72% under the verbatim harness, 71% with the instruction, 75% with the abstract description and 94% with the clean retry, which, as in the original, is the worst harness for repetition: greedy decoding in the restored context writes the same move again. Here a

In Section 10, the authors summarize free play by averaging over the 6 (model, game) combinations to state that the exact repeat rate is 72% under the verbatim harness vs. 75% under the abstract description and 71% with the instruction. However, inspecting Table 12 in Appendix F reveals that this 72% vs. 75% average is an artifact of pooling two opposite regimes: for Qwen2.5-0.5B in Othello, the exact repeat rate is 1% under verbatim (and 2% under verbatim+instr) versus 86% under abstract and 98% under drop, whereas in 4 of the other 5 (model, game) cells, abstract actually reduces the exact repeat rate relative to verbatim (SmolLM2-360M drops from 97% to 73% in Connect Four, 97% to 80% in Othello, and 89% to 49% in Sokoban; Qwen2.5-0.5B drops from 96% to 89% in Connect Four, increasing only in Sokoban from 52% to 76% and Othello from 1% to 86%). Thus, for SmolLM2-360M, abstract consistently reduces exact repetition across all three games (from 94% mean to 67% mean), and the claim that abstract repeats more than verbatim in free play (75% vs 72%) is driven by Qwen2.5-0.5B's negative copy term in Othello (1% vs 86%). The authors should report the free-play summary separately by model (as they noted in the probe that Qwen2.5-0.5B has a negative copy term) rather than implying that abstract generally underperforms verbatim in free play.

8. Turn-structure and extra-phrase confound in abstract description — Critical

, byte for byte. Every post-attempt condition re-shows the same position s after its message, so conditions differ in exactly one thing, the record of the attempt (Table 8 in the appendix lists them all). The core set is the original’s:

The headline conclusion and Sections 6 and 10 argue that replacing a verbatim failure record with an abstract description preserves the model's prior preference (slope +0.75 vs +0.18) and reduces probe inversion because it omits the move string. However, according to Table 8 and Table 9, the fail and neut conditions append both an agent turn (containing the move) and a runtime observation turn, whereas abstract and abstract_min omit the agent turn altogether and insert a bracketed counter and extra clause ([{ordinal} attempt failed: {gloss}; that move is not repeatable]). Consequently, abstract differs from fail and neut not only in whether the move string is present, but also in conversational turn structure (missing an assistant turn in the chat template) and in the added directive ; that move is not repeatable. While fail_other in RQ4 (tested on 3 models) keeps the two-turn structure by substituting a different failing move, the core 8-model comparison and the prior-flattening slope analysis in Section 6 rely on abstract without controlling for the omitted agent turn. The authors should either include a two-turn control condition (such as fail_other or a masked placeholder turn) in the 8-model and prior-flattening analyses, or explicitly caveat that abstract changes both the turn structure and message wording alongside omitting the move string.

9. Unit mismatch between per-token and total-nat G values — Minor

Pooled, the gain is −1.61 [−2.12, −1.14] nats under the verbatim record and −0.39 [−0.54, −0.24] under the abstract description, which removes 76% of the inversion, close to the share the original reported for tool calling. It is not a fix for every model: it lowers the failed move for 5 models (SmolLM2-135M,

In Section 10 (page 12), when reporting the 8-model comparison between the verbatim record and the abstract description, the authors state that 'Pooled, the gain is -1.61 [-2.12, -1.14] nats under the verbatim record and -0.39 [-0.54, -0.24] under the abstract description, which removes 76% of the inversion.' In Section 5 (Table 1 and text), the pooled gain over the same 8 models and 840 shared items in short notation is reported as G = -1.38 [-1.85, -0.95] nats per move token (while -1.61 was the per-token value for illegal moves only in Section 5). This discrepancy occurs because Section 10 switches from nats per move token to unnormalised total nats across the move string (where Othello moves in short notation like d3 are 2 tokens, weighting Othello twice as heavily as 1-token Sokoban and Connect Four moves). The authors should either report the per-token values alongside the total-nat values in Section 10 or explicitly note why the pooled verbatim baseline changes from -1.38 nats/token in Section 5 to -1.61 nats in Section 10.

10. Selection-induced regression to the mean in model-proposed items — Minor

(a) Model-proposed items are the realistic case and are model-specific by construction: a proposal pass scores every move of the game, legal and illegal, at pre , and the item is the highest-ranked failing move among that model’s top three. They are analysed as a robustness check, never pooled

Model-proposed items are constructed by scoring all expressible moves at pre and selecting the highest-ranked failing move among the model's top three. Because these items are selected specifically for having an extreme high log-probability at pre, any subsequent context change or perturbation will naturally induce regression toward the mean (lowering log-probability and yielding a positive G), even in the absence of a specific copy-flattening mechanism. While the comparison between fail and abstract in Section 6 helps isolate the effect of the move string, the raw sign flip of G = +1.34 in Section 5 is partly an artifact of conditioning on the top-ranked actions at pre. The authors should acknowledge the role of selection-induced regression to the mean when interpreting the positive G on model-proposed items and report the corresponding G under the abstract condition in Section 5 for direct comparison.

11. Losing vs illegal comparison is confounded with prior plausibility — Minor

Figure 2 and Table 2 separate the two kinds of failure. Illegal moves invert clearly: G = − 1 . 61 [−2.16, −1.11] nats per token, negative on 66% of items, with a copy term of +1.52 [+1.07, +2.01] . Legal but losing moves invert much less, G = − 0 . 39 [−0.90, +0.06] (23 positions), an interval that includes zero. Their copy term is still positive (+0.59

The authors conclude that illegal moves invert clearly (G = -1.61) whereas legal but losing moves do not invert significantly overall (G = -0.39), refuting H2. However, Table 2 shows that before the attempt, the failed move is substantially more plausible for losing moves than for illegal moves: P(repeat) at pre is 0.45 for Sokoban losing (38% greedy repeat) and 0.26 for Connect Four losing (16% greedy repeat), compared to 0.20-0.25 (1%-18% greedy repeat) for illegal moves. Because Section 6 demonstrates that high-prior moves have positive G due to preference flattening, and Section 4.2 notes that Sokoban deadlock pushes often repeat the agent's previous move in the prefix, the difference between illegal and losing moves in Table 2 is heavily confounded by prior plausibility and prefix occurrence rather than failure semantics alone (indeed, in Table 4, controlling for the prior reduces the losing vs. illegal difference to non-significance: -0.37, SE 0.24 in the ladder and -0.17, SE 0.13 in chat models). The authors should qualify the claim in Section 5 and the Abstract to explain that losing moves fail to invert largely because they are much more plausible (and often already present in the context) prior to the attempt.

xPeerd (AI review)

October 23, 2026

AI Review (xPeerd — DAReview)

Summary

This manuscript studies whether failure feedback in game-playing LLM agents causes repetition of the failed move, extending earlier tool-calling results to Sokoban, Connect Four, and Othello. The core claim is that recording the failed move in the transcript often “backfires” by increasing the probability of repeating it on fixed probe items, primarily through the copied surface form rather than the semantic content of the failure message. The paper is strongest when it clearly distinguishes illegal from losing moves, decomposes the effect into copy and semantic terms, and tests several harness variants with concrete quantitative outcomes. The manuscript is also commendably transparent about reproducibility, reporting hardware, software, and a code/data release strategy in detail.

The main weaknesses are not conceptual incoherence but limits in scope, statistical coverage, and framing. The study itself concedes that the losing-move subset is small, that some intervals including zero may reflect underpowered samples, and that key planned extensions—especially the RL half of RQ3 and larger-scale checks—were not run. The paper’s strongest evidence therefore supports a narrower claim than some parts of the abstract and discussion imply: the effect is robust for illegal moves on fixed probe items, more conditional for realistic model-generated failures, and not yet comprehensively established across training regimes, scales, or broader gameplay settings.

Potential Major Revisions

  1. Narrow and recalibrate the central claim to match the actual evidential scope.

The abstract presently states, “the record backfires: pooled over 8 instruction-tuned models (135M–1.7B) it makes the failed move more likely” and presents this as a headline result. Yet later sections qualify this substantially: “The effect holds for illegal moves and not for losing ones, and it is not universal” and “4 of the 8 models invert clearly” while “Llama-3.2-1B goes the other way.”

The current rhetoric risks overstating generality. The empirical record in the manuscript supports at least three different phenomena:

  • strong inversion on fixed probe items, especially illegal moves;
  • weaker or mixed behavior for losing moves;
  • sign reversal on “realistic” model-proposed items, where “the sign flips: G = +1.34” and the record lowers repetition probability relative to pre-attempt likelihood.

Because these are not minor caveats but structurally important distinctions, I would revise the abstract, introduction, and conclusion to foreground them earlier and more explicitly. For example, the framing should say that the paper identifies a transcript-induced flattening effect whose observed consequence depends on prior move plausibility, instead of leading with a simpler “backfire” claim. The discussion already contains the more precise formulation: “The simplest description of our results is not ‘the record makes the failed move more likely’ but ‘the record makes the failed move’s probability stop depending on what the model thought of it before’.” That wording is arguably closer to the actual contribution and should be elevated from Discussion to the abstract-level framing.

  1. Address the mismatch between fixed probe items and ecologically realistic failures more directly.

Section 3.4 is methodologically careful in distinguishing perturbation items, random items, and model-proposed items. The manuscript admits that “Items are perturbations, not samples,” and later states that “The fixed items of the original design lean towards the first kind” and that “A failure in free play is of the second kind by construction.”

This is an important limitation because the paper’s headline pooled estimates depend heavily on the fixed-item design. Indeed, the discussion explicitly says, “Our results suggest that such items mostly measure induced repetition, of actions the model would not otherwise have taken, and say less about the repetition of a model’s own likely errors.”

I think the manuscript should strengthen this section in two ways:

  • provide a clearer rationale for why fixed perturbation items remain the primary design despite reduced ecological validity;
  • report the model-proposed and free-play findings more centrally, not as secondary qualifications.

At present, the manuscript may be read as demonstrating a general failure-feedback pathology in gameplay, while the strongest evidence may instead be about context contamination under artificially perturbed failure candidates. That is still valuable, but the contribution should be stated precisely.

  1. Strengthen statistical justification and uncertainty reporting.

The paper repeatedly notes limited sample sizes. In Section 12/limitations material, it states: “Small samples. The studies used the sample sizes planned for a laptop CPU…” and adds that intervals including zero “may therefore reflect too few items rather than the absence of an effect.”

This admission is honest, but it creates a burden on the manuscript to avoid over-interpreting marginal or null-looking results. This is particularly relevant for:

  • losing-move analyses, where “Losing moves are also the smaller class: 23 of the 120 positions in RQ1”;
  • deadlock pushes, which “rest on only 15 positions”;
  • training-stage inferences in RQ2, especially around stages where intervals include zero;
  • remedy comparisons that appear heterogeneous across models and games.

Major revision should therefore include a more formal statistical framing: what hypotheses are confirmatory versus exploratory, what the effective unit of inference is, and where the reader should interpret evidence as suggestive rather than decisive. A compact “evidence strength” statement per RQ would materially improve rigor.

  1. Clarify how much of the interpretation depends on the copy-mechanism explanation rather than direct evidence.

The paper frequently interprets the findings through copying. For example: “What drives the repetition is the specific string in the transcript,” and in the conclusion, “The effect is carried by the move string.”

This explanation is plausible and well-motivated, but the manuscript should distinguish more carefully between operational decomposition and mechanistic inference. The decomposition into ∆copy and ∆sem is useful, yet some language in Discussion moves toward causal-mechanistic claims tied to pretraining and induction-head-like behavior. The RQ2 result that “The base model already inverts” certainly supports pre-post-training continuity, but it does not by itself isolate a specific internal mechanism.

I recommend softening any wording that sounds stronger than the evidence supports, unless the authors add targeted mechanistic probes. At minimum, note explicitly that “copy term” here is a behavioral decomposition term, not direct evidence of a particular circuit-level mechanism.

  1. Resolve the tension between free-play practical conclusions and benchmark weakness.

The manuscript uses free play to motivate practical recommendations, but it also states that “Both are weak players: 81% of all moves are rejected and 68% of episodes end at the step limit.”

This creates a validity concern: if the agents are extremely weak in the chosen settings, then some operational conclusions about harness design may be partly specific to low-competence regimes. For example, “a decoder ban stops exact repeats but not illegal play” is interesting, yet on these weak models it may mainly show substitution among many bad options rather than a general limit of banning as a remedy.

The paper should either:

  • narrow the practical recommendations to “small weak models in these settings,” or
  • justify why the observed pattern should generalize beyond that regime.

Without that clarification, the recommendations risk sounding more universal than the experiments warrant.

  1. Expand the anti-fraud / coherence assessment around AI assistance and authorship boundaries.

The manuscript explicitly states: “The author used Anthropic’s Claude… to write the experimental and analysis code and to draft and revise the manuscript text.”

This disclosure is appropriate, but it raises a reproducibility and authorship-quality issue that deserves slightly more discussion, especially because the paper’s subject is itself LLM behavior. I do not see obvious signs of fabricated results; on the contrary, the manuscript is unusually transparent about code, macros, environment control, and regeneration of every number.

However, the author should clarify how the generated code and prose were validated, especially for analysis scripts that produce headline quantitative conclusions. This is not an accusation of misconduct; rather, in a paper about model behavior, the boundary between author oversight and AI-assisted production should be documented with special care.

Fraud / paper-mill warning: I do not find strong paper-mill indicators. The manuscript has a coherent paradigm, explicit methods, concrete effect sizes, transparent limitations, and a reproducibility appendix. The main concern is overclaiming relative to sample scope, not fabrication.

Potential Minor Revisions

  1. Typographic and formatting issues in title/header area.

The title appears split awkwardly across lines: “Feedback That Backfires in Games: Do LLM Players Repeat the Move They Just Watched Fail?” The PDF extraction suggests a line-break artifact between “Watched” and “Fail?” that may be acceptable in typesetting, but it visually reads abruptly in the extracted text and should be checked in the final layout.

  1. Inconsistent hyphenation and dash conventions.

The manuscript alternates among forms such as “instruction-tuned,” “post-training,” “tool calling,” “free play,” and “multi-turn RL.” Some are field-standard, but consistency should be enforced. For example, “free play” versus “free-play” appears in nearby sections and appendix labels. See “In free play” versus “free-play episodes.”

  1. Occasional overly compressed prose reduces readability.

Some sentences are elegant but too dense for a broad ML readership. For example: “The simplest description of our results is not ‘the record makes the failed move more likely’ but ‘the record makes the failed move’s probability stop depending on what the model thought of it before’.” This is insightful, but several paragraphs in Discussion maintain this compressed argumentative style, which could be broken into shorter sentences.

  1. Ambiguity in section cross-referencing.

The abstract says, “All measurements ran in float32 on a free Colab GPU, which first reproduced the original paper’s number.” This is useful, but the pointer to where this is established could be more explicit in the abstract or introduction. The detailed explanation appears later in the hardware/reproduction sections.

  1. Reference formatting irregularities.

The references include some entries with unusual capitalization or formatting, for example “2 olmo 2 furious” for the OLMo report, and “Qwen, :, An Yang, et al.” which appears malformed in the extracted references. These may partly reflect PDF text extraction, but the bibliography should be checked carefully in source form.

  1. Numeric-expression readability.

There are several places where multiple numeric contrasts are packed into a single sentence, e.g., “Pooled, the gain is −1.61… and −0.39…” or “0%… 22%… 0%.” These are accurate but cognitively heavy; using a short table or sub-bullets more often in results-heavy paragraphs would help.

  1. Clarify notation on first use.

Terms such as “G,” “∆copy,” and “∆sem” are central, but readers may benefit from repeated brief glosses when these return after several pages, especially in Discussion and Conclusion. The paper is mathematically modest enough that a small redundancy would improve accessibility.

  1. Appendix-table legibility.

The free-play appendix table is information-rich but dense. For example, rows such as “qwen2.5-0.5b Othello verbatim 0% 1% 100% 40.0 50%” are difficult to parse without stronger visual grouping. Reformatting the appendix tables would materially improve usability.

AI content analysis for this post-2021 work

Estimated percentage of AI-generated content: 35–55%.

Rationale:

  • The prose is highly regular, stylistically controlled, and unusually consistent in argumentative cadence.
  • The manuscript explicitly discloses AI assistance: “The author used Anthropic’s Claude… to write the experimental and analysis code and to draft and revise the manuscript text.”
  • This estimate should not be interpreted as misconduct because the assistance is openly declared.

Highlighted likely AI-assisted sections:

  • The highly compressed abstract and claims packaging on page 1, especially the long multi-result sentence beginning “On fixed probe items built by perturbing a correct move…”
  • The polished comparative synthesis in Discussion, particularly “The simplest description of our results is not…” and the surrounding interpretive paragraphs.
  • Reproducibility and availability statements, which have the characteristic template-like precision of AI-assisted scientific drafting, e.g., “Every number in this paper is produced by code… nothing is typed by hand…” and “The author reviewed and edited the content and takes full responsibility for it.”

Assessed epistemic impact:

  • Low to moderate.
  • Because the AI use is disclosed and the manuscript provides unusually strong reproducibility scaffolding, the main epistemic risk is not hidden authorship but possible over-optimization of narrative clarity relative to evidential nuance.
  • The most important safeguard remains independent rerunning of the code and closer checking of the statistical claims.

Recommendations

  1. Rewrite the abstract and conclusion so the main contribution is framed as transcript-induced preference flattening, with “backfire” treated as one regime-specific consequence rather than the universal summary.

  2. Move the ecological-validity limitation from later discussion into the introduction or end of Section 3, making clear that fixed perturbation probes and model-generated failures answer related but distinct questions.

  3. Add a concise evidence-strength paragraph for each RQ:

  • strong evidence for illegal-move inversion on fixed probes;
  • weaker evidence for losing moves;
  • suggestive evidence for training-stage attenuation;
  • incomplete evidence for RL effects because “We could not run the RL half of RQ3 or a 7B scale check.”
  1. Temper mechanistic language around copying unless additional probes are added. Keep the behavioral finding, but present the circuit-level interpretation more cautiously.

  2. Reframe practical recommendations so they are explicitly conditional on model size, competence, and harness setting. This is especially important because free-play performance is weak and may not support broad operational prescriptions.

  3. Add a short validation note describing how AI-assisted code and drafted text were checked by the author before inclusion, given the explicit AI-use disclosure and the paper’s methodological emphasis on exact quantitative effects.

  4. Perform a final copyedit of references, table formatting, hyphenation, and notation reminders to improve polish without changing substance.

XSci (AI review)

October 23, 2026

Summary

When an agent harness records a failed action and asks again, the record may make that action more likely rather than less. The authors carry their earlier tool-calling measurement of this "corrective gain" G into three games (Sokoban, Connect Four, Othello), where a move can fail by being illegal or by being legal but losing, both verified exactly. On fixed probe items built by perturbing a correct move the record backfires: pooled over eight instruction-tuned models of 135M-1.7B parameters, G = -1.38 [-1.85, -0.95] nats per move token, carried by the surface form (Delta_copy = +1.34) rather than the failure message (Delta_sem = +0.04). An exploratory analysis finds that the record flattens the model's prior preference (slope 0.18), so implausible moves become more likely and likely ones less so. The inversion is present in the OLMo 2 1B base model, largely removed at DPO, and clear in RAGEN's format before any RL.

Contribution and Significance

The core contribution is a counterfactual decomposition of what a recorded failure does to the probability of the failed move, in a setting where ground truth is exact and one model can be followed through post-training. The most valuable result is not the inversion but Section 6: the record does not shift the failed move uniformly, it decouples the move's probability from what the model thought of it before. That reconciles the negative gain on perturbed items (G = -2.62 for moves with prior below 1%) with the positive gain on the model's own likely mistakes (G = +1.34, 759 items), and it tells harness designers something no benchmark leaderboard shows.

Relative to the closest prior work the advance is real but narrow. Gumaan [2026] already defines G, Delta_copy and Delta_sem, fixes the perturbation item design and reports the same remedy ranking, and the code is vendored unchanged; new here are the games setting, the post-training ladder, the RAGEN-format probe and the flattening account. Against Schmied et al. [2025], whose frequency bias is behavioural, the counterfactual framing is a genuine improvement. The paper's own framing overstates the headline, however: the inversion as stated in the title and abstract holds for an item distribution the paper itself shows is unrepresentative.

Strengths

  • The measurement discipline is exceptional: conditions byte-identical except for the attempt record (Table 8), checkpoints pinned to revisions (Table 7), cached and uncached scoring agreeing to 1e-3 nats, and the pipeline required to reproduce the prior paper's G = -23.92 to 9.5e-5 nats before any game was scored.
  • Both failure classes are verified rather than judged: a one-ply check in Connect Four, an exhaustive breadth-first search for Sokoban deadlocks (Section 3.2).
  • The OLMo 2 ladder (Table 3) is well chosen and its decomposition informative: DPO removes the inversion by killing the copy term (+0.49 at base to -0.23) while the semantic term moves the other way (+0.03 to +0.35).
  • The RAGEN probe (Table 5) uses RAGEN's own prompt-building functions and its own invalid-action feedback, so it measures the exact starting distribution of multi-turn RL.
  • The reported figures are internally consistent: Tables 1, 2, 3, 6 and 12 reproduce the text, including the position-weighted pooling behind G = -1.38 and the 61% of 840 pairs.

Recommendation

Major Revision for TMLR. The measurements are sound and the data are largely in hand to support correct claims, but two of the abstract's four claims do not survive the paper's own analysis as stated, and the headline quantity is governed by an item distribution the paper itself identifies as atypical. What is needed is re-analysis and restatement rather than new experiments, with one exception: the illegal-versus-losing contrast needs more losing items. Nothing decisive fails, and the two substantive new designs stand.

Major Concerns

  1. The headline claim is stated for an item distribution the paper shows is unrepresentative (Abstract; Section 5 "Answer"; Sections 6 and 13). Section 6 establishes that 48% of the shared items have a prior below 1% against 2% of the model-proposed items, and that on the latter the sign reverses to G = +1.34 [+1.18, +1.49]. The abstract, the Section 5 answer and the conclusion nevertheless lead with the inversion, so the quantity the title asks about is measured almost entirely on moves no model would have played; the Discussion concedes that such items "mostly measure induced repetition". The claims should be restated around the flattening result, which is currently labelled exploratory and unplanned.

    The flattening account also needs firmer evidence before it can carry the paper. The shared-versus-proposed contrast is confounded by more than the prior, since the two sources also differ in failure type and game, and no reported analysis holds those fixed. The fail_other condition (Table 8) is the natural placebo for the slope analysis, lengthening the context with an unrelated attempt record without writing the scored move, yet Section 6 reports slopes only for fail, neut and abstract. That slope, with a prior-stratified analysis within one item source, would settle it.

  2. The illegal-versus-losing dissociation is not established (Section 5 "Illegal versus losing moves"; Tables 2 and 4). The abstract states the effect holds "for illegal moves but not for losing ones", but the losing result is a null on 23 positions with an interval including zero (-0.39 [-0.90, +0.06]), Othello contributes no losing items, and Section 12 concedes such intervals "may reflect too few items rather than the absence of an effect". More seriously, the two classes differ in exactly the variable Section 6 identifies as decisive: in Table 2 the Sokoban losing items start at P(repeat) = 0.45 and 38% greedy repetition against 0.25 and 18% for the illegal items, because a legal move is a plausible move. The only failure-type coefficient reported, with the prior controlled, is indistinguishable from zero in both fits (-0.37 (SE 0.24) in the ladder, -0.17 (SE 0.13) in the chat models, Table 4), and Section 8 never discusses it. On this evidence the dissociation is a prior effect, not a semantics-of-failure effect. Resolving it needs the failure-type contrast at matched priors and enough losing positions to distinguish a null from low power.

  3. The surface-form attribution holds only in the pooled average (Section 5; Tables 1 and 2). The claim that the increase is carried by the move's surface form rests on Delta_sem = +0.04 [-0.10, +0.17], but that near-zero averages model-level terms of opposite sign whose intervals exclude zero: +0.82 [+0.63, +1.02] for SmolLM2-1.7B, +0.38 [+0.13, +0.65] for Qwen2.5-0.5B, -0.98 [-1.76, -0.21] for Qwen3-1.7B. SmolLM2-1.7B is one of the four models listed as inverting clearly, yet its copy term is -0.19 [-0.55, +0.20]: its inversion is carried by the failure message, not the move string. Table 2 shows the same reversal across cells (-0.58 for Sokoban losing, +0.31 for Othello illegal, both excluding zero). The mechanistic claim, on which the Discussion and the harness recommendations rest, should be restricted to the models it holds for, or supported by an analysis of the sign of Delta_sem.

Minor Comments

  1. Section 4.2 announces a covariate for whether the failed move repeats one of the agent's own earlier moves (15/80 Connect Four, 24/100 Sokoban decision points), but no reported analysis uses it; it belongs in Table 4.
  2. Report the Section 6 slope for fail_other, and the remedy conditions for the remaining RQ1 models.
  3. Section 10 reports gains in nats (-1.61, -0.39) while Section 5 reports nats per move token (-1.38), and -1.61 is also Section 5's per-token illegal gain; state the unit in each sentence.
  4. Table 2's caption reads "KQI by game and failure class", presumably for RQ1.
  5. The interval notation in Section 6 ("+0.18 +/- 0.14, +0.21") is ambiguous; use the bracketed form used elsewhere.
  6. In Table 12 the verbatim and drop harnesses give identical values in all five columns for qwen2.5-0.5b on Connect Four and smollm2-360m on Connect Four and Sokoban, which needs explaining.
  7. Section 11's harness recommendation has a counterexample in Table 12: for smollm2-360m on Sokoban the abstract description gives 14.5 wasted moves and 100% looping episodes against 9.5 and 0% under the verbatim record.
  8. In Table 4 the dependent variable is G per move token while move length is both a covariate and the denominator; report the unnormalised regression too.
  9. Othello carries no losing labels and Connect Four's is one ply deep (Sections 4.1, 12); say whether deeper or Othello losing labels are feasible under the same exact-verification standard.
  10. The related work omits the negative-results literature on self-correction with reliable external feedback, which is this paper's setting: Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song and Denny Zhou, "Large Language Models Cannot Self-Correct Reasoning Yet", ICLR 2024; and Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han and Rui Zhang, "When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs", TACL 12:1417-1440, 2024.
  11. The motivating observation from XiangqiBench concerns frontier players while every number here comes from models of 135M-1.7B parameters that reject 81% of their own moves in free play; carry that scope into the title and abstract.