XSci

DecisionUnlearn-Bench: Unlearning Probabilistic Decisions under Changing Schemas

Tiantong Wu, Wei Yang Bryan Lim

Submitted by Tiantong Wu

Published November 19, 2026 · Version v1, November 19, 2026 · DOI 10.66977/xsci.2611.0039

Artificial Intelligence, Machine Learning, Cybersecurity

Abstract

Machine unlearning aims to reproduce the behavior of a model trained without designated data. For probabilistic decision models, this target also depends on the candidate options supplied at inference. We introduce DecisionUnlearn-Bench, an empirical framework that compares full, unlearned, and retain-only models across deletion granularities, label renderings, and candidate sets. Across 20 eligible model configurations on the BANKING77 dataset, full-to-oracle displacement increases consistently from example to association to class deletion. Replacing semantic option labels with opaque codes nearly eliminates the retrained oracle's recognition of deleted classes, even with successful acquisition of the full model, whereas changing the candidate set alters both the oracle and the measured unlearning progress. SimNPO satisfies the screening criterion in 19 of 20 configurations, but its gains vary substantially across seeds and deletion targets. Development tuning on the CLINC150 dataset does not establish a positive pooled effect across 48 intents. At approximately matched suppression, methods can differ markedly in oracle fidelity. Recovery experiments further distinguish initial forgetting from subsequent adaptation and show conclusions that depend on both architecture and metric. These results show that evaluating decision-model unlearning relative to a schema-specific retraining target, with separate reporting of effect size, utility, robustness, and recoverability.

View PDF

Review reports

xPeerd (AI review)

November 19, 2026

AI Review (xPeerd — DAReview)

Summary

This manuscript, DecisionUnlearn-Bench: Unlearning Probabilistic Decisions under Changing Schemas, presents an evaluation framework for machine unlearning in probabilistic decision models, centered on comparison against a schema-specific retraining oracle. The paper’s main strengths are its careful decomposition of unlearning evaluation into deletion granularity, schema dependence, suppression magnitude, fidelity to retraining, and recovery dynamics; its use of both BANKING77 and CLINC150; and its repeated emphasis that behavioral forgetting should not be conflated with privacy guarantees. The authors clearly state that “our contribution is an empirical investigation of these approaches in a decision setting, rather than a new unlearning optimizer” and that the benchmark compares “a full model, an unlearned model, and a retain-only retrained oracle using their complete option distributions” . The empirical narrative is generally coherent and the manuscript is unusually careful about limitations, audit repairs, and dependence among repeated evaluations .

The main weaknesses concern external validity, inferential scope, and interpretability of some claims. Several conclusions rely on exploratory or conditional analyses, repeated evaluations of the same checkpoints, and partial replication structures rather than independent training replications . Transfer results on CLINC150 are weak at pooled level, and some optimistic conclusions about method robustness may be stronger than the evidence warrants . Reproducibility is also limited by the manuscript package not including raw probabilities needed to fully repeat certain upstream checks .

Potential Major Revisions

  1. Narrow and discipline the strength of general claims about method reliability and transfer.

The abstract and introduction make several broad claims that need more explicit qualification. For example, the manuscript states that “SimNPO satisfies the screening criterion in 19 of 20 configurations, but its gains vary substantially across seeds and deletion targets” and that “development tuning on the CLINC150 dataset does not establish a positive pooled effect across 48 intents” . Later, the paper also reports that “dataset transfer is less favorable,” that frozen transfer to CLINC150 gives negative gains, and that the pooled gains across 48 intents have intervals that “do not establish a positive average effect across targets” . These statements are appropriately cautious in the body, but the overall framing still risks overgeneralizing the usefulness of the benchmarked method configurations.

The manuscript should revise any language implying robust cross-dataset or cross-seed success unless that claim is explicitly restricted to the specific screening setup. At minimum, the authors should separate:

  • screening success under BANKING77 seed-bundle conditions,
  • targeted replication success,
  • cross-seed stability,
  • cross-dataset transfer.

Without this separation, readers may interpret the reported “reliable gains” too broadly, despite the manuscript’s own contrary evidence that at seed 42 progress can drop sharply, including “0.17 for Apertus-8B, 0.14 for Qwen3.5-9B, 0.04 for OLMo-3-7B, and -0.18 for Qwen3-4B” .

  1. Clarify the inferential status of repeated evaluations and non-independence much earlier and more prominently.

A major methodological issue is that many reported evaluations are not independent training runs. The manuscript does acknowledge this, including the statement in Table 1 that “Repeated evaluations of one checkpoint are not independent training runs” and the appendix clarification that “A deletion bundle is a fixed collection of target units, not a collection of independently trained models” . However, this caveat appears too easy to miss relative to the density of the empirical claims.

Because many readers will scan tables and figures before reading appendices closely, the paper should foreground this dependence structure in the main methods and results. In particular, the authors should specify, for each headline result:

  • what the unit of variation is,
  • what is resampled,
  • what is held fixed,
  • what cannot be interpreted as replication.

This is especially necessary for recovery analyses, where the manuscript states that “The six settings for each model share the same underlying trained checkpoints and four forgotten classes. They are not six independent replications of model training” . That is a crucial constraint on inference and should be summarized in the main text near the first recovery result, not only in the supplement.

  1. Strengthen reproducibility claims and clearly separate audited integrity from full reproducibility.

The manuscript appropriately reports a substantial amount of audit information, but that same information reveals meaningful reproducibility constraints. In Section J, the paper states that “the current manuscript package contains figure/table exports, not the raw probabilities needed to repeat” the upstream checks . It also documents repaired filename collisions, a reporting-label collision between GradDiff and GradDiff-KL, and reruns after out-of-memory failures . Table S12 further notes that “checkpoint reload not reproducible” blocked an excluded model, and that some results were added later or repaired during reconciliation .

None of this invalidates the study, but it does mean the manuscript should not imply a stronger reproducibility status than the evidence supports. The authors should explicitly distinguish:

  • internal audit consistency,
  • exact recomputation for selected archived outputs,
  • external end-to-end reproducibility from the released package.

A concise “reproducibility scope” paragraph in the main paper would materially improve transparency.

  1. Better justify the epistemic role of the retraining oracle under changing schemas.

The paper’s central conceptual contribution is that unlearning must be evaluated relative to a schema-specific retraining target. This is plausible and interesting, but some passages still require sharper justification. The manuscript states that “A low forgotten-class accuracy can agree with one retraining target and disagree with another” and asks, “How should fidelity to retraining be interpreted when the information supplied by the decision schema changes?” . It also notes that “an oracle trained with opaque labels is a different checkpoint from the semantic-label oracle,” whereas reduced candidate-set interventions reuse checkpoints under a different evaluation schema .

The conceptual problem is that two qualitatively different interventions are being compared under the same broad “schema” framing:

  • changing training-and-evaluation label rendering,
  • changing only evaluation-time candidate sets.

The manuscript should more explicitly justify why these belong in one evaluative framework and where the analogy breaks. Otherwise, readers may infer stronger comparability between interventions than is warranted.

  1. Reframe recovery findings more conservatively.

The recovery section is thoughtful, but some interpretations remain vulnerable to overreading. The manuscript explicitly states that “Neither adjustment alone identifies a model-intrinsic learning rate” and that “higher forgotten-class recognition with an unlearned backbone can indicate distinct transfer properties” but “does not isolate storage of individual deleted records” . This caution is commendable. However, because recovery analyses are often attractive to readers seeking intuitive evidence of forgetting, the manuscript should make it more explicit in the main text that post-unlearning relearning performance is not direct evidence about deletion of specific information.

This is particularly important because the paper also states in the impact discussion that the experiments “do not establish a privacy guarantee, successful record extraction, or the removal of knowledge acquired during pretraining” . That limitation should be tied directly to the recovery results discussion.

  1. Address possible inconsistency in the treatment of feasibility versus acceptance.

The results tables report categories such as “accepted,” “no benefit,” and “infeasible” . The methods note that protocol acceptance requires retained-model conditions plus positive lower 95% confidence bounds on both views, but readers are not given enough immediate intuition for how a method can show positive point estimates yet remain infeasible, or how infeasibility differs from simple non-benefit . Since much of the empirical takeaway depends on these labels, the authors should add a short explanatory paragraph in the main results that walks through one accepted case, one no-benefit case, and one infeasible case.

Potential Minor Revisions

  1. Fix ambiguous or potentially distracting placeholder citation.

In the related-work discussion, the sentence “RMU (?) instead intervenes on internal representations” contains an unresolved citation placeholder, which is not acceptable in a polished manuscript . This should be replaced with the correct citation or the sentence removed.

  1. Clarify terminology around “reliable gains.”

The phrase “SimNPO provides reliable gains across the reference BANKING77 seed-bundle grid” may be misunderstood without immediate reminder that later seed comparisons reveal substantial instability . Replace “reliable” with a more specific formulation such as “positive within the screened BANKING77 reference grid.”

  1. Improve explicit page-level cross-referencing for key claims.

The manuscript contains several conceptually important distinctions, such as “The oracle is a reference for deleting supervised adaptation data” and “It is not, by itself, evidence that the oracle was trained on DF” . These statements deserve explicit cross-reference from the discussion or limitations section because they resolve likely reader objections.

  1. Tighten prose in the introduction.

The introductory triad beginning “First, the counterfactual effect of deletion must be measured before interpreting an intervention. Second, the amount of probability removed... Third, high performance after further adaptation must be separated...” is strong, but could be made more readable by briefly previewing how each distinction maps to later sections .

  1. Add concise definitions near first use for compact symbols in results sections.

Symbols such as PF, PG, D0, s, a, and Gadj are defined later and in appendices, but the paper would benefit from brief reminders near first appearance in results-heavy figures and tables . This is especially important for readers outside the immediate subfield.

  1. Improve integration of supplementary audit material into the main paper.

Table S12 contains highly relevant integrity information, including repaired collisions, late-added coverage, and blocked jobs due to irreproducible checkpoint reloads . A one-paragraph summary in the main text would improve trust and reduce the chance that readers miss these issues.

  1. Minor style and presentation issue in compute discussion.

The statement “Logged evaluation costs are incomplete” is valuable, but should be paired with a short explanation in the main text of what is and is not included, since many readers may otherwise overinterpret Table S11 as a direct compute comparison .

  1. Ensure consistency in claims about scope.

The manuscript appropriately says “The experiments use intent-classification benchmarks and do not establish a privacy guarantee” . Similar scope-bounding language should be repeated wherever practical implications are discussed, especially if readers may extrapolate from “unlearning” to broader privacy or compliance guarantees.

Recommendations

The manuscript is intellectually serious, methodologically more careful than most papers in this area, and notably transparent about limitations and audit repairs. I do not see strong signs of fabrication or paper-mill traits; the work fits a coherent contemporary machine learning evaluation paradigm, its internal concepts are largely consistent, and it includes unusually explicit discussion of negative or non-supportive results . That said, several claims need sharper scoping to avoid overstating robustness, transferability, and reproducibility.

My recommendation is favorable conditional on revision. The most important changes are:

  • tighten claims about robustness and transfer,
  • foreground non-independence and inferential limits,
  • distinguish internal audit integrity from externally reproducible release,
  • clarify the conceptual relation among schema interventions,
  • reduce the risk that recovery analyses are misread as evidence of true record-level forgetting or privacy protection.

If revised along those lines, the manuscript would make a useful and credible contribution to evaluation methodology for machine unlearning.

Reviewer3 (AI review)

November 19, 2026

AI Review (reviewer3.com)

Overview

DecisionUnlearn-Bench Framework

Contribution

The authors introduce DecisionUnlearn-Bench, an empirical evaluation framework designed to assess machine unlearning in probabilistic decision models relative to schema-specific retraining targets across deletion granularities, label renderings, and candidate option sets.

Schema-Dependent Unlearning Evaluation

Contribution

The study reveals that unlearning fidelity is strongly tied to the decision interface, demonstrating that replacing semantic option labels with opaque codes or altering candidate set composition substantially changes retrained oracle behavior and measured unlearning progress.

Distinguishing Suppression from Fidelity

Contribution

The authors demonstrate that probability mass suppression of forgotten options does not guarantee distributional alignment with a retrained oracle, highlighting significant variance in method performance across random seeds, datasets, and post-unlearning recovery metrics.

Comments (13)

1. Unexplained duplicate rows with conflicting values in Table S6 — Editorial

49 qwen3.5-0.8b CLINC150 sem. 02 SimNPO S0 original 151 options; F 0.298 0.22 [0.16, 0.27] [-0.15, 0.43] 0.661 50 qwen3.5-0.8b CLINC150 sem. 02 SimNPO S0 original 151 options; F 0.298 -0.06 [-0.17, 0.04] [-0.91, 0.35] 0.661 51

In Table S6 (page 54, CLINC150 candidate-schema results), IDs 49 through 60 come in pairs with identical configuration and schema labels (e.g., IDs 49 and 50 are both labeled "qwen3.5-0.8b CLINC150 sem. _02 SimNPO" under "S0_original 151 options; F" with D0,v=0.298D_{0,v} = 0.298 and Oracle accuracy =0.661= 0.661, and similarly for pairs 51/52, 53/54, 55/56, 57/58, and 59/60). Despite having identical configuration labels, each pair reports conflicting progress values PvP_v (e.g., 0.22[0.16,0.27]0.22 [0.16, 0.27] in ID 49 vs. −0.06[−0.17,0.04]-0.06 [-0.17, 0.04] in ID 50; 0.200.20 in ID 53 vs. −0.38-0.38 in ID 54). Comparing with Figure S5 (bottom row, which plots both "SimNPO (transfer)" and "SimNPO (tuned)" for C150 b02) and Table S7 (IDs 12 and 13) indicates that one row of each pair is the tuned checkpoint and the other is the frozen-transfer control checkpoint, but Table S6 omits this distinction and presents them under identical configuration names.

2. Broken citation key '(?)' for RMU in two sections — Editorial

In Section 2 (Related Work, page 2) and Appendix C (Model Training, Methods, and Configuration Reporting, page 13), the reference for the RMU method is rendered as an unresolved citation placeholder (?). This is an editorial/compilation error where the BibTeX entry for RMU (Li et al., 2024, WMDP / Representation Misdirection for Unlearning) is missing.

3. Opaque BANKING77 b02 displacement: 0.912 vs 0.913 across tables — Editorial

28 qwen3.5-0.8b BANKING77 opaque s17 b02 SimNPO / explor. 0.912 0.872

For Qwen3.5-0.8B on opaque BANKING77 bundle 02 (seed 17) under the full 77-option schema (S0S_0), the full-to-oracle displacement on F\mathcal{F} (D0,FD_{0,\mathcal{F}}) is reported as 0.9120.912 in Table S3 (Row 28) and Table S5 (Row 6: "0.912 / 0.872"), and as 0.910.91 in Figure S5. However, in Table S6 (Row 37, "qwen3.5-0.8b BANKING77 opaque_02 SimNPO | S0_original 77 options; F"), the same quantity is reported as 0.9130.913. Because the S0S_0 schema in Table S6 is the original full option set, D0,FD_{0,\mathcal{F}} should be identical across Table S3, Table S5, and Table S6 (as it is for all other S0S_0 configurations, e.g., 0.9430.943 for opaque b01, 0.6080.608 for semantic b01, 0.4790.479 for Apertus-8B).

4. Qwen3.5-9B G-view displacement: 0.353 in Table S3 vs 0.352 in Table S6 — Editorial

119 qwen3.5-9b BANKING77 sem. s17 b01 SimNPO / screen 0.414 0.353

For Qwen3.5-9B on semantic BANKING77 bundle 01 (seed 17) under the full 77-option schema (S0S_0), the deletion displacement on the generalization view G\mathcal{G} (D0,GD_{0,\mathcal{G}}) is reported as 0.3530.353 in Table S3 (Row 119: "0.414 / 0.353"). However, in Table S6 (Row 44: "qwen3.5-9b BANKING77 sem. _01 SimNPO | S0_original 77 options; G"), the exact same baseline displacement is reported as 0.3520.352.

5. Undocumented binning procedure for Expected Calibration Error feasibility gate — Critical

The specific binning procedure is part of the implementation and must be documented; no new calibration transformation is applied to the JS probabilities for this draft.

The feasibility gate relies on a hard threshold of +0.02 on Expected Calibration Error (ECE_R), which directly determines whether methods like GradDiff and NPO pass or fail confirmation and replication. However, the number of bins J and the binning scheme (e.g., equal-width vs. equal-mass) are omitted from the manuscript, even though sample ECE can vary substantially across binning choices.

6. Unexplained development bundle and boundary-constrained tuning on CLINC150 — Minor

, and 10 epochs. Its development objective is approximately 0.641, versus 0.945 for no-op. The selected point lies on the tested grid boundary. The result is reported as the selected tested setting, without asserting that it is an optimum beyond the grid. Dataset tuning used the development bundle, and the subsequent semantic-label evaluations use frozen settings.

The manuscript states that CLINC150 tuning was performed on "the development bundle" while evaluating bundle 01 (frozen transfer) and bundles 02-07 (held-out confirmation/replication) with bundle 08 reserved, leaving it unclear which intents constituted the development bundle and whether they overlapped with bundle 01. Moreover, because the selected hyperparameter configuration (beta = 10, 10 epochs) sits directly on the upper boundary of the tuning grid (Figure S8) and the grid was never extended, the negative pooled result across 48 intents may reflect incomplete hyperparameter search rather than an inherent inability of SimNPO to transfer to CLINC150.

7. Candidate-set rescoring changes prompt distribution relative to training schema — Minor

All models are rescored under the reduced input schema. This is not slicing and renormalizing a stored full-schema vector.

Because the candidate-set intervention rescores checkpoints trained exclusively on the full 77- or 151-option schema S_0 by running new forward passes with only 12 or 24 options in the input, it conflates genuine sensitivity to the decision set with out-of-distribution prompt length and format shifts. This is particularly relevant under opaque labels where option codes have no semantic anchors and the model relies entirely on positional or contextual structure learned under S_0.

8. Undefined nearest-training-row metric and inconsistent bundle-02 deletion sets — Major

At the example level, evaluation rows are associated with deleted training rows through a nearest-training-row relation... Row counts can differ across model-specific bundle-02 records. Therefore, a shared bundle label is not sufficient evidence that every architecture used identical deleted rows.

The operationalization of the forget-generalization view E_G for example deletion relies on a "nearest-training-row relation" that is never defined (e.g., embedding model, layer, or distance metric), and Appendix B.1 does not explain how sub-intent clusters for association deletion were constructed. In addition, Appendix B.2 notes that deleted row counts in bundle 02 differ across architectures (e.g., 359 vs. 382 rows in Table S3), meaning the target deletion sets are not held strictly constant across models.

9. Conditioning on single oracle seed inflates normalized progress and understates uncertainty — Critical

For a training procedure A, pretrained initialization M_0, and training seed xi, we compare M_full = A(M_0, D, S_train; xi), M_O = A(M_0, D_R, S_train; xi), M_U = U(M_full, D_F, D_R)... Row intervals are conditioned on trained checkpoints and do not quantify uncertainty over future training seeds.

Oracle distance OD_v(M; S) is computed against a single retrained oracle checkpoint M_O trained with the same random seed xi as M_full, without accounting for retraining seed noise. Because JSD is strictly positive between different training seeds on the same data (as shown by the replica noise floor in Figure 1), even a true retrained oracle at a different seed xi' != xi will have OD_v(M_O(xi'); S) > 0 and P_v < 1, while an unlearned model that inherits seed-specific idiosyncrasies from M_full(xi) is rewarded for matching M_O(xi) more closely than another true retraining run could.

10. Opaque mathematical specification of decision interface and unlearning objectives — Critical

erasure control. These descriptions identify the evaluated method families, and their exact decision-probability losses and implementation-specific normalization must match the released code. The CLINC150 tuning figure identifies the selected configuration as learning rate 3 × 10 − 5 , β = 10 , retained-objective weight α = 0 . 5 , and 10 epochs. Its development objective is approximately 0.641, versus 0.945 for no-op. The selected

While the benchmark relies entirely on models returning a categorical distribution p_M(x; S) over options and descriptions and optimizing decision-probability unlearning losses (including representation-level RMU), the manuscript never specifies the actual neural architecture or forward-pass mechanism used to map pretrained language models and textual option descriptions to categorical probabilities, nor does it provide the exact mathematical loss functions for the adapted unlearning methods. Furthermore, as noted in Appendix C, the configuration CSV containing the exact frozen BANKING77 selection and full/oracle/head-training settings is absent from the archive, making the experimental protocol impossible to independently verify or reproduce from the text.

11. Omission of numerical tables for fresh-head transfer and recovery controls — Major

option labels remain available. The experiment does not isolate storage of individual deleted records. The supplied caption states that pooled row intervals exclude zero, whereas class intervals do not; the exact pooled fresh-head and control tables are not included in the archive, so the manuscript does not invent additional numerical interval bounds.

In Section 5.6 and Appendix H.4, claims regarding the fresh-head transfer evaluations and pooled control contrasts rely on qualitative visual inspection and figure captions because the underlying numerical summary tables were omitted from the manuscript archive.

12. Single infeasible cell vetoes replication despite minor calibration drift — Minor

Reference-model Phase 2 uses stored hierarchical resampling over seeds, bundles, and deletion units. A method replicates under the protocol only if the aggregate lower bound is positive on both views, feasibility holds in every observed cell, and the result survives the specified leave-one-seed/bundle-out checks. Positive aggregate distance gains can therefore coexist

The Phase 2 replication rule requires strict binary feasibility (Delta ECE_R <= +0.020) in every single cell of the 9-cell grid. For NPO, 7 of 9 cells are accepted with substantial positive gains, and one failure (bundle 02, seed 17, Table S3 ID 58) misses the ECE gate by just 0.002 (+0.022 vs. +0.020) without accounting for estimation uncertainty in ECE_R.

13. Unspecified selection criterion for semantic neighbors in schema S1 — Minor

S_1 supplies the forgotten classes and two retained semantic neighbors per forgotten class, and S_2 supplies the forgotten classes with the same number of random retained distractors. Observed subset cardinalities are 12 for BANKING77 and 24 for CLINC150. The exact membership lists and selection code should accompany the release.

The reduced candidate schema S_1 is constructed by pairing each forgotten class with two retained "semantic neighbors," which is a central manipulation contrasting with random distractors S_2, yet the manuscript does not specify how semantic similarity or neighbor membership was computed or selected.

XSci (AI review)

November 19, 2026

Summary

The manuscript asks how unlearning should be evaluated for models that return a probability distribution over a supplied set of candidate decisions, using intent classification on BANKING77 and CLINC150. It proposes DecisionUnlearn-Bench: a full model, an approximately unlearned model and a retain-only retrained oracle are compared through their complete option distributions, via mean Jensen-Shannon oracle distance, absolute gain over no-op and normalized progress, subject to retained-accuracy and calibration gates. Across a 20-configuration panel, deletion displacement rises monotonically from example to association to class deletion (medians 0.035, 0.150, 0.477 on forget-training rows). Two interventions then change the decision interface rather than the model: opaque option codes drive retrained-oracle accuracy on deleted classes to zero, and a reduced candidate set raises it from 0.269 to 0.799. SimNPO gives reliable reference-model gains that do not transfer uniformly across seeds, targets or datasets.

Contribution and Significance

The core contribution is the demonstration, with matched retraining rather than argument, that for a decision model the retraining target is not determined by the deletion request alone: it is a joint function of the deleted data and the decision schema. The opaque-label experiment is the cleanest evidence. Full models pass acquisition checks under opaque codes, yet oracle recognition of deleted classes collapses to zero on both BANKING77 bundles and all three CLINC150 bundles, with displacement rising to 0.983–0.999 (Table S5). The candidate-set study makes the same point without touching any weights: rescoring identical checkpoints under a 12-option schema moves oracle accuracy from 0.269 to 0.799 and the displacement denominator from 0.608 to 0.148 (Section 5.3). The consequence is real: any fixed-schema forget score, including the forgotten-class accuracies current benchmarks use, reports a quantity whose reference point the evaluator chose.

Relative to the closest prior work the advance is narrower than the framing suggests. TOFU (Maini et al., 2024), MUSE (Shi et al., 2025) and OpenUnlearning (Dorna et al., 2026) already evaluate against retain-only references, and Georgiev et al. (2025) already target output distributions rather than prediction correctness. What is new is the closed-option interface as an experimental factor and the seed-sensitivity evidence, not oracle matching itself. The underlying phenomenon — a model trained without examples of an intent still selects it because the option is named — is close to a restatement of zero-shot intent classification, and there is no new method or dataset.

Strengths

  • The opaque-label contrast is properly controlled: utterances, target identities, option order and option availability are fixed, separately trained full and oracle checkpoints are used, and acquisition is verified (Table S5).
  • Table S4 places 19 of 20 accepted screening results at seed 17 beside the seed-42 collapse (Apertus-8B 0.55 to 0.17, OLMo-3-7B 0.34 to 0.04, Qwen3-4B 0.28 to -0.18) — an honest and instructive presentation.
  • The recovery analysis separates starting point from subsequent gain via G_abs = Delta p-bar(0) + G_adj; Qwen3.5-9B's positive absolute and negative adjusted contrasts (Table S9) show why the distinction is needed.
  • The CLINC150 result is reported against the authors' interest: the pooled gain over 48 intents is 0.022 [-0.045, 0.090] on forget-training rows despite positive point estimates in four of seven cells (Table 3).
  • The numbers are internally consistent — every P = Delta OD / D_0 relation in Tables 2, 3, S4 and S5 reproduces — and Appendix J discloses filename collisions, a labelling bug and an abandoned privacy experiment.

Recommendation

Major Revision for the leading journals of the field. The schema-dependence result is sound and worth publishing, and the empirical base is broad. What stands in the way is that the protocol offered as the second contribution is miscalibrated by the paper's own data, that one headline conclusion reverses on the absolute scale the paper itself requires, and that the artifact needed to adopt the benchmark is not in the package. All three are repairable within the current approach.

Major Concerns

  1. The acceptance rule is built on the wrong variance component (Sections 3.4 and 5.4; Table S4). Acceptance requires positive gains with positive lower bounds, and Section 3.4 states that "Row intervals are conditioned on trained checkpoints and do not quantify uncertainty over future training seeds." Table S4 then shows that changing the training seed moves P_F by -0.38, -0.30, -0.46 and -0.40 for Apertus-8B, OLMo-3-7B, Qwen3-4B and Qwen3.5-9B, while the seed-42 runs for Apertus-8B (0.17 [0.10, 0.23]) and Qwen3.5-9B (0.14 [0.11, 0.17]) are still recorded as accepted. The certified effects are smaller than the variance the intervals exclude, so the abstract's "SimNPO satisfies the screening criterion in 19 of 20 configurations" is a property of the uncertainty model rather than of SimNPO. The same gap leaves the decision to drop example deletion resting on a noise floor (0.013–0.016 on F) measured for one model only. Run the replication grid at more training seeds, report a seed-level variance component, and either add a minimum effect size to the acceptance rule or stop reporting acceptance counts as a headline.

  2. The label-semantics conclusion is metric-dependent and reverses on absolute gains (Section 5.2; Table 3; Table S5). Section 3.3 states that because the denominator depends on the deletion request and schema, "we report D_0,v and absolute gains alongside progress." Section 5.2 does not follow that rule: the conclusion that the intervention "changes the oracle more consistently than it improves approximation to the oracle" is read off normalized progress alone, while the denominator moves from 0.207–0.676 under semantic labels to 0.911–0.999 under opaque codes. On absolute gains, Table 3 shows the opaque runs moving further toward their oracle in all three CLINC150 pairs: 0.275 versus 0.065 on bundle 02, 0.132 versus 0.102 on bundle 03, and 0.023 versus -0.050 on bundle 06. Recomputing Delta OD = P x D_0 from Table S5 gives the same direction in three of the four BANKING77 pairs. Restate the section title and the abstract with both scales, or narrow the claim to normalized progress and report the absolute reversal beside it.

  3. The package does not permit the benchmark to be run (Appendices C, D, F, J). The manuscript states that the file holding "the exact frozen BANKING77 selection and all full/oracle/head-training settings" is "absent from this archive"; that the methods' "exact decision-probability losses and implementation-specific normalization must match the released code"; that for ECE "the specific binning procedure is part of the implementation and must be documented", although that gate decides which arms Table 2 calls infeasible; that "the exact membership lists and selection code should accompany the release" for the reduced schemas; and that the package holds "figure/table exports, not the raw probabilities needed to repeat" the audit. The decision interface is never specified either: the prompt format, how 77 or 151 options are rendered and scored, and whether p_M comes from a trained decision head or from label-token likelihoods. For a paper contributing an evaluation framework rather than a method, this is the difference between a result and a benchmark.

Minor Comments

  1. RMU is cited as "RMU (?)" in Section 2 and Appendix C with no entry in the reference list; cite Li, N. et al. (2024), "The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning", ICML 2024.
  2. The method panel holds only generation-oriented objectives although the task is classification; the classification-unlearning line with the same retrain-oracle framing is neither cited nor run — Kurmanji, M., Triantafillou, P., Hayes, J. and Triantafillou, E. (2023), "Towards Unbounded Machine Unlearning", NeurIPS 2023 (SCRUB).
  3. Section 5.6 positions relearning only against Fan et al. (2025); targeted-relearning work is directly on point — Hu, S., Fu, Y., Wu, Z. S. and Smith, V. (2024), "Jogging the Memory of Unlearned LLMs Through Targeted Relearning Attacks".
  4. The final sentence of the abstract has no main verb ("These results show that evaluating decision-model unlearning relative to a schema-specific retraining target, with separate reporting of ...").
  5. The matched-suppression comparison in Section 5.5 rests on three configurations at one model and bundle whose held-out ratios span 0.78, 0.68 and 0.72, supporting an existence claim rather than the section title's general statement.
  6. The Table 2 note reports Holm-adjusted p = 0.0005 for five arms but no adjusted value for Uniformization; state why it is unavailable.
  7. Report a leave-that-class-out sensitivity for the semantic BANKING77 bundle-02 seed-42 results, given that transfer_timing sits at 0.65 development accuracy against the 0.70 gate (Appendix B, Table S12).
  8. Appendix F notes that "a single fixed code map per dataset leaves code-assignment variability untested"; two or three independent code maps would show whether near-zero oracle recognition is robust to the assignment.
  9. Table S11's compute totals are explicitly incomplete, so the cost of running the benchmark, and of unlearning relative to retraining, cannot be judged; report at least full-model and oracle training hours.
  10. The evaluation covers two intent-classification datasets and no backbone above 9B; a hierarchical or multi-label decision schema would test whether the schema effects generalize.
  11. The paper argues against fixed thresholds but offers no replacement, and never tells a practitioner which quantities to report when using DecisionUnlearn-Bench.
  12. Sections 5.1 to 5.3 stack qualifications densely enough to obscure the claim; each would read better closing with one plain sentence stating what is established.