XSci

False-Alarm Improvements and Detection Trade-Offs in Quantum Machine Learning for Intrusion Detection: A Reproducible Secondary Analysis

Roberto Fernández-Barrios, Iker Pastor-López, Asier González-Santocildes, Pablo García Bringas

Submitted by Roberto Fernández-Barrios

Published November 13, 2026 · Version v1, November 12, 2026 · DOI 10.66977/xsci.2611.0021

Cybersecurity, Machine Learning, Quantum Computing

Abstract

Published results in quantum machine learning for intrusion detection (QML-IDS) require joint interpretation of false-positive rate (FPR), attack true-positive rate (TPR), and evaluation provenance. We reconstruct operating points in the 25-entry frame selected by Kaissar and colleagues: 21 included articles, 20 identified documentary families, substantive methodological material consulted for 15, and 24 descriptive points from five. In one test-labelled comparison, a quantum support vector machine reduces FPR from 6.875% for random forest to 3.329%, while TPR decreases from 93.955% to 93.682%; both rates improve against three other comparators. In another study, a complete benign row identifies FPR as 12/6000 (0.200%), despite incomplete attack-side allocation that requires additional assumptions for TPR. Normalized matrices identify detection ranges rather than a unique aggregate rate. A uniform provenance rule separates training-described and test-adjusted results from six conditional illustrations whose independence remains unverified. The contribution is a reproducible interpretation of favorable comparisons and detection trade-offs, without converting descriptive results or hypothetical calculations into operational guarantees.

View PDF

Review reports

xPeerd (AI review)

November 13, 2026

AI Review (xPeerd — DAReview)

Summary

This manuscript presents a secondary, reproducible reconstruction of published evidence on false-positive rate (FPR), attack true-positive rate (TPR), and evaluation provenance in quantum machine learning for intrusion detection. Its central contribution is not a new detector, but an auditable re-interpretation of a closed 25-entry review frame, narrowed to 21 included articles and 20 documentary families, with 24 descriptive operating points from five papers supplying the main quantitative analysis. The paper’s main strength is methodological restraint: it repeatedly distinguishes recoverable descriptive rates from stronger inferential claims and avoids fabricating unavailable counts or overstating “quantum advantage.” It also makes several concrete findings, such as a reconstructed K17 comparison where QSVM-IGWO reduces FPR relative to random forest while slightly reducing TPR, and a K10 benign row that supports an identifiable FPR but not a fully secure attack-side interpretation without added assumptions.

The main weaknesses are scope and evidential fragility rather than obvious technical error. The corpus is deliberately narrow, partly inaccessible, and not representative of the whole QML-IDS field. Several key results remain conditional because independence, threshold freezing, test integrity, and matrix completeness cannot be verified from the source reports. The manuscript is careful about this, but some claims of contribution and some presentation choices still risk being read more strongly than the underlying evidence permits. The work appears coherent and scientifically grounded, but it remains exploratory meta-research rather than confirmatory evidence about operational IDS performance.

Potential Major Revisions

  1. Clarify the novelty claim more sharply against existing review literature and against the paper’s own admissions of limited scope.

The manuscript states that “This original secondary research reconstructs published quantitative evidence rather than providing an exhaustive literature review” and that “Our contribution is the correspondence between published objects, class-specific counts, denominators and provenance, and the comparisons that this correspondence supports.” This is a plausible contribution, but the paper also concedes that reviewing FPR/detection metrics “is therefore not itself new,” and that the study follows a previously selected frame rather than creating a new systematic review.

My concern is not that the contribution is absent, but that the distinction between “new reconstruction” and “new substantive knowledge about QML-IDS performance” should be made even more explicit throughout the manuscript. In places, the paper’s title and emphasis on “false-alarm improvements and detection trade-offs” may suggest a broader comparative performance contribution than the body ultimately supports. Since the authors repeatedly stress that they add “no detector training, quantum execution or new operating threshold,” the introduction and conclusion should more forcefully delimit novelty as an evidential audit of published reporting rather than as a performance synthesis.

  1. Strengthen the manuscript’s handling of source dependence and non-independence across retained operating points.

The manuscript appropriately warns that “rows are not independent replications” and that “Multiple points within an article are dependent evidence, and even documentary families are not guaranteed independent experiments.” It also states that “The extraction unit is an article–source version–dataset/subset–detector–operating point” and “It is not an independent study.”

However, the presentation still foregrounds counts such as “24 descriptive points” and six “conditional illustrations,” which may unintentionally convey more breadth of evidence than exists. A stronger revision would move the dependence warning earlier and make it visually unavoidable wherever Figure 1, Table 1, or the quantitative findings are introduced. For example, the manuscript should explicitly explain in Results 4.1 and the figure caption that the point count is primarily a traceability device, not an evidential sample size. Without that emphasis, readers may over-interpret the number of retained operating points as a quasi-meta-analytic base.

  1. Rework the status of K10 and K17 findings to reduce the risk of inferential over-reading.

The paper states that “Under the fixed-detector binomial model, the six-point block contains K10 and the five K17 Test-labelled matrices; independence remains unverified for both studies.” It also notes that K10 is the only point with U.95 < 1%, but immediately cautions that this does not establish stronger low-FPR guarantees. Likewise, for K17 the manuscript reports that QSVM-IGWO lowers FPR against all four classical comparators, yet also warns that matching denominators do not prove identical evaluated instances and that no paired statistical comparison is possible.

These are exactly the right cautions, but the manuscript should revise its framing so that the positive result and the uncertainty are not rhetorically asymmetric. At present, the favorable comparative result is memorable and concrete, while the qualification is more technical and easier to miss. I recommend revising the Results and Conclusions to state, in one sentence each time, that these are descriptive detector-point comparisons with unresolved independence and no formal dominance claim. That would better align the prose with the paper’s own epistemic standard.

  1. Improve the explanation of provenance classification and why some intervals are hypothetical while others are conditional.

The provenance rule is central to the whole paper. The manuscript says, “Counts described as training results, or accompanied by reported adjustments after observing the test, receive hypothetical illustrations only,” applying this to K11 and K03, whereas K10 and K17 remain “conditional illustrations with unverified independence.” This is sensible, but the manuscript currently asks readers to absorb a subtle hierarchy: descriptive points, conditional intervals, hypothetical-only intervals, and identification/rounding ranges.

Because this classification drives interpretation, it deserves a more pedagogical explanation, possibly with a short schematic or boxed summary in the Methods or at the start of Results. Right now, the logic is present but cognitively dense. Readers outside statistical methodology may struggle to understand why K03 and K11 are excluded from the main interval block while K10 and K17 remain, despite all four having substantial inferential caveats. A simplified decision tree would materially improve transparency and reproducibility.

  1. Address the manuscript’s own use of AI more critically, because it is more than incidental editing support.

The manuscript discloses: “An OpenAI Codex assistant supported the extraction design, source retrieval and interpretation, transcription, code development, deterministic calculations, figure generation and manuscript drafting. This use went beyond language editing.” It further states that “Both passes used the assistant and do not constitute independent human extraction.” This is commendably transparent. The declarations also state: “OpenAI Codex assisted with manuscript preparation and editing, source review and data extraction, analysis code, calculations and scientific plotting.”

Because this reliance affects extraction reliability, the manuscript should include a sharper risk-control account in the Methods: exactly which tasks were human-verified, which were machine-assisted but deterministically checked, and where residual error might remain. The current wording is honest, but still too compressed for a paper whose core contribution depends on careful source interpretation. In a reproducibility-oriented study, the assurance structure around AI-assisted extraction is itself a methodological issue, not just a declaration item.

  1. Expand the anti-overclaiming discussion around “quantum advantage.”

The Discussion correctly states, “The evidence does not establish quantum advantage at a controlled FPR, a common deployment distribution, or a matched resource budget.” This is one of the manuscript’s strongest statements and should be elevated in prominence. Much of the underlying literature in QML security risks exaggerated comparative interpretations. The present manuscript is valuable partly because it resists that tendency.

Yet the paper could go further by defining what would be minimally required before terms like “advantage,” “improvement,” or “enhancement” become justified in this domain. Since it already cites methodological contrasts and benchmarking cautions, a short paragraph setting out necessary conditions—e.g., matched data splits, fixed thresholds, independent calibration, paired predictions, and resource accounting—would make the manuscript more useful as a field-corrective reference.

  1. Tighten the relationship between title, abstract, and the actual inferential ceiling of the study.

The abstract says the contribution is “a reproducible interpretation of favorable comparisons and detection trade-offs, without converting descriptive results or hypothetical calculations into operational guarantees.” This is accurate and appropriately restrained. However, the title phrase “False-Alarm Improvements and Detection Trade-Offs” can still sound more definitive than the manuscript’s repeated caveats support.

I recommend either softening the title or ensuring that the subtitle and opening abstract sentence foreground that this is a reconstruction of reported operating points under constrained provenance. The current title is not misleading in a strict sense, but it is more assertive than the body, and in a contested area such rhetorical mismatch can propagate over-interpretation in citations.

Potential Minor Revisions

  1. Typographic and encoding problems should be corrected throughout the manuscript.

There are repeated broken characters and hyphenation/encoding artifacts in the extracted text, for example “false￾positive rate,” “meta-research,” “prove￾nance,” “test-adjusted,” and similar interruptions across the manuscript. These appear in the abstract, introduction, methods, results, and discussion sections. See, for instance, “false￾positive rate” in the abstract and introduction, “prove￾nance” in multiple sections, and “reproduc￾ibility artifact” in the conclusions/data-availability area. These should be corrected before submission because they noticeably degrade readability and can create a poor impression of production quality.

  1. A formula rendering issue appears in Section 3.3.

The displayed equation for rates appears corrupted in the extracted text: “FPR = [ FP / n0 , TPR = 1 [ − FN / n1 .” This likely reflects PDF extraction artifacts, but if similar rendering appears in the manuscript itself, it must be fixed. Mathematical definitions are central here; even minor notation corruption can undermine confidence in the paper.

  1. Some wording is unnecessarily dense and could be simplified for accessibility.

For example, the sentence “The extraction unit is an article–source version–dataset/subset–detector–operating point” is precise but heavy, and the distinction between “descriptive,” “conditional,” “hypothetical,” and “identification/rounding” evidence is conceptually demanding. The manuscript would benefit from occasional plain-language restatements after technical definitions.

  1. Table and figure guidance should be strengthened in the main text.

Figure 1 is important, but its interpretation depends heavily on the caption. The caption already carries substantial caveats, such as “These are identification ranges, not confidence intervals” and “All 24 descriptive points retained; K03 and K11 hypothetical intervals are supplementary only.” Some of that caption content should be moved into the main text so readers do not miss key interpretive constraints.

  1. The limits section should cross-reference the exact denominator restrictions more explicitly.

The manuscript notes that even zero false positives at n = 9251 cannot support the 0.01% reference and that at least 29956 benign observations would be needed. This is a strong and concrete point. It would help to mention this earlier where analytical reference levels are first introduced, so readers understand immediately that some apparent low-FPR claims are resolution-limited by sample size.

  1. Minor consistency issue in how the contribution is described.

In one place the contribution is framed as “reconstructs published quantitative evidence,” elsewhere as “recovering joint false-alarm/detection evidence,” and elsewhere as “correspondence between published objects, class-specific counts, denominators and provenance.” These are compatible, but a single stable formulation would improve rhetorical coherence.

  1. Data and code availability are strong, but reproducibility instructions could be more operational.

The manuscript provides Zenodo and GitHub links and states that the current version reproduces the numerical tables and figures. A brief sentence indicating the expected entry point for reproduction—e.g., top-level script, notebook, or workflow file—would further improve practical reproducibility.

AI content analysis for this post-2021 work

Estimated percentage of AI-generated or AI-assisted content: moderate to substantial AI assistance is explicitly disclosed, but the manuscript does not support a precise numerical estimate with confidence. I would cautiously characterize the manuscript as likely showing substantial AI-assisted drafting and technical support rather than fully AI-generated writing, based on the authors’ own statement that OpenAI Codex supported “source review and data extraction, analysis code, calculations and scientific plotting” as well as drafting and editing.

Highlighted AI-detected or AI-salient sections:

  • Methods disclosure on AI assistance: “An OpenAI Codex assistant supported the extraction design, source retrieval and interpretation, transcription, code development, deterministic calculations, figure generation and manuscript drafting. This use went beyond language editing.” This is the clearest explicit evidence of major AI assistance and appears in Section 3.2.
  • Declarations section: “OpenAI Codex assisted with manuscript preparation and editing, source review and data extraction, analysis code, calculations and scientific plotting.” This confirms broad workflow involvement beyond copyediting.
  • Stylistically, several sections exhibit a highly controlled, repetitive, caveat-heavy register with symmetrical contrast structures, especially in the Introduction, Methods, and Discussion. That style is not proof of AI generation, but it is consistent with substantial AI-assisted drafting, particularly where definitional distinctions are reiterated with near-formulaic precision. Examples include the repeated contrasts between “descriptive,” “conditional,” “hypothetical,” and “operational guarantees.”

Assessed epistemic impact: The epistemic impact of AI assistance here is non-trivial because the paper’s central value depends on exact extraction, interpretation, provenance classification, and careful non-overclaiming. The authors mitigate this by explicitly disclosing AI use and by stating that no synthetic counts or replacement experiments were introduced. However, because “both passes used the assistant and do not constitute independent human extraction,” the assurance structure is weaker than it would be under independently duplicated manual extraction. This does not invalidate the manuscript, but it does increase the importance of transparent supplementary dossiers and verifiable computational artifacts. In short: the AI involvement does not by itself negate the contribution, but it raises the burden for documentation and validation.

Fraud / paper-mill warning assessment

I do not see strong paper-mill or fabrication signals in this manuscript. On the contrary, the manuscript is unusually explicit about limitations, source gaps, dependence, and inferential boundaries. It repeatedly refuses to infer missing counts, to pool non-comparable evidence, or to convert descriptive rates into operational claims. Those are anti-fraud indicators rather than fraud indicators.

That said, one caution remains: the manuscript depends on complex source reconstruction assisted by AI, and not all underlying source objects were fully recoverable. Because of that, the work should be judged as careful exploratory reconstruction, not as definitive quantitative synthesis. The concern here is not fabrication by the present authors, but inherited uncertainty from the underlying literature and from partially inaccessible evidence.

Recommendations

  1. Reframe the paper’s contribution consistently as a provenance-sensitive reconstruction of recoverable published operating points, not as a broad performance evaluation of QML-IDS.

  2. Add a compact methodological schematic that explains the evidence-status hierarchy:

    Status Meaning Included in main analysis? Inferential strength
    Descriptive point Recoverable published metric/count interpretation Yes Low
    Conditional interval Count-based illustration under stated assumptions, independence unverified Yes, for K10/K17 Limited
    Hypothetical interval Further weakened by training-description or test-adjustment issues Supplement only Very limited
    Identification/rounding range Range implied by normalized or rounded matrices without denominators Yes, descriptively Minimal
  3. Move the strongest caveat about quantum advantage into the abstract or the final sentence of the introduction: the evidence does not establish controlled-FPR quantum advantage, matched deployment performance, or resource-fair superiority.

  4. Add a short subsection on quality assurance for AI-assisted extraction, detailing human verification boundaries, decisive-cell checks, and residual risks.

  5. Correct all encoding, formula, and rendering issues before submission. These are minor mechanically, but they materially affect reviewer confidence.

  6. Make the dependence structure impossible to miss by repeating, in the Results opening and Figure 1 lead-in, that operating points are not independent replications.

  7. Consider modest title softening so the rhetorical force of the title better matches the manuscript’s disciplined inferential ceiling.

Overall recommendation

This is a careful, coherent, and methodologically serious manuscript with a legitimate contribution in reproducible secondary analysis. Its main risks are over-interpretation by readers, not obvious technical invalidity. I would view it as publishable in principle after revision, provided the authors strengthen the framing of novelty, dependence, provenance classification, and AI-assisted extraction controls, and clean up the presentation issues documented above.

Reviewer3 (AI review)

November 13, 2026

AI Review (reviewer3.com)

Overview

Quantitative Reconstruction of QML-IDS Trade-offs

Contribution

The authors systematically reconstruct joint false-positive and true-positive operating points across published quantum machine learning intrusion detection studies, demonstrating that reductions in false alarms often accompany trade-offs in attack detection rather than uniform metric improvements.

Provenance Rule for Security Benchmarks

Contribution

This study establishes a uniform evaluation provenance framework that separates descriptive training data and test-adjusted results from conditional independent test claims in security evaluations.

Comments (4)

1. Underlying row-normalized cells and class count omitted for K07 — Minor

Overall attack TPR is not uniquely recoverable: without attack weights, displayed-row bounds are 96–100% for QSVM/QCNN and 90–100% for CNN, whereas SVM has no displayed attack-to-Normal errors. Figure 1 expands these TPR ranges for rounding and displays the FPR rounding ranges: 35.5–36.5%, 1.5–2.5%, 20.5–21.5% and 1.5–2.5%, respectively, under rounding to nearest.

The manuscript reports the derived displayed bounds for K07 (96–100% for QSVM/QCNN, 90–100% for CNN, and 100% for SVM) and applies a ±0.5 percentage point rounding allowance, but does not state how many attack classes are in the K07 matrices or whether the attack-to-Normal rates come from a single cell per row or a sum across cells. Specifying the matrix dimensions and the exact displayed Normal-column entries from Figures 5–8 of K07 is necessary to verify that ±0.5 percentage points is the exact rounding bound for each row.

2. Complete corpus table of all 25 entries omitted from manuscript — Major

fusion matrix, a favorable result, a low-FPR objective, or full-text access. All 25 frame entries retain stable identifiers K01–K25. The two quantum-inspired entries K02 and K05 are excluded [15, 16]; eligibility remains technically unresolved for K08 and K09 [17, 18]. The remaining 21 entries are included [19–39]. K13 and K21 belong to one identified documentary

While Table 1 summarizes aggregate denominators and Table 2 details the five articles yielding main-figure operating points, the remaining 20 entries in the Kaissar frame (such as K01, K12–K14, K18–K22, K24, K25) are not systematically listed or mapped to their specific exclusion or non-recoverability reasons in the main text. Including a concise master table of all 25 entries (K01–K25) with their citation, eligibility status, access status, and reason for lacking a primary joint operating point would make the reconstruction self-contained without requiring readers to consult external supplementary files.

3. Raw counts for K11 main-figure points omitted from manuscript — Minor

b. SI retains K03/K11 hypothetical intervals and K11 counts. Source reports: [20, 23–25, 31]

Unlike the other count-based studies in Figure 1a (which display exact FP/n0 and FN/n1 fractions directly on the plot), the raw confusion-matrix counts and class denominators for the 10 K11 points in Figure 1b are omitted from the main manuscript and deferred to the Supplementary Information, and the text only states the rates at the smallest (1,000) and largest (16,000) nominal sizes. Tabulating or annotating the exact FP/n0 and FN/n1 counts for all 10 K11 points in the main text or Figure 1b would substantiate the claim that class counts differ between platforms and make all 24 main-figure points directly auditable in the article.

4. Unbounded TPR sensitivity to the 15 unallocated attack cases — Minor

a conditional upper limit of 0.324%. The four attack rows do not fully allocate their declared 4000 cases: their explicit entries total 3985. The Normal column contains eight attack-to-Normal errors. Consequently, TPR 99.800% additionally requires linking the 4000 attacks in Table 2 to Table 3 and treating that column as exhaustive. We do not fill the fifteen unallocated cases.

For K10, the manuscript notes that 15 of the 4000 declared attack cases are unallocated in Table 3 and states that the 99.800% TPR (8/4000) requires treating the 8 Normal predictions as exhaustive, yet it plots a single point estimate and binomial confidence interval in Figure 1a. Explicitly stating the deterministic identification range for TPR if the 15 unallocated attack cases were instead false negatives (which would yield up to 23/4000 FNs, lowering TPR to 99.425% before sampling uncertainty) would align K10's treatment with the paper's rigorous handling of partial identification in K07.

XSci (AI review)

November 13, 2026

Summary

The manuscript is a secondary analysis of quantum machine learning for intrusion detection (QML-IDS). Taking the 25-entry frame of Kaissar et al. [4] as a closed corpus, the authors include 21 articles spanning 20 documentary families, consult substantive methodology for 15, and recover 24 descriptive operating points from five of them. Each point is traced to its source matrix, class denominators and evaluation provenance, and assigned a provenance tier: conditional illustrations with unverified independence (six points from K10 and K17), hypothetical-only count points (fourteen from K03 and K11), and normalized points receiving identification and rounding ranges only. The principal reconstructed results are a QSVM in K17 that reduces FPR from 6.875% to 3.329% against random forest while TPR falls from 93.955% to 93.682%, a complete benign row in K10 identifying FPR as 12/6000 (0.200%), and K07's row-normalized matrices, which support detection ranges rather than a unique aggregate TPR.

Contribution and Significance

The core contribution is the correspondence, established point by point, between a published confusion matrix, its denominators and what its provenance permits by way of sampling interpretation. The payoff is specificity: the authors show that completeness of a matrix and validity of independent-test inference are separate properties (ten of nineteen complete matrices in Figure 1 are K11 training-described results), and that a favorable headline can conceal a trade-off rather than uniform dominance. The most durable findings are discrepancies in the primary literature that others can act on: K11's Table 3 precision and recall columns numerically match matrix-derived recall and precision respectively, K17's Figure 7 counts imply a 75/25 Train/Test allocation against a stated 80/20 split, K10's four attack rows allocate 3985 of a declared 4000 cases, and K13 and K21 prove to be one documentary family.

Relative to the closest prior work the advance is narrower than the framing suggests. Lamberts et al. [7] already systematize IDS evaluation including confusion-matrix reporting and comparability, and the practice recommendations of Section 5.2 — link one matrix to a named subset, publish class counts, beware selection on test performance — restate Arp et al. [3] and Sommer and Paxson [2] rather than extending them. The statistical apparatus is standard throughout, as the authors state. What is genuinely new is the corpus-specific audit and the discrepancies it surfaced.

Strengths

  • Every independently checkable number reproduces exactly: the K17 rates and the -3.546 and -0.274 percentage-point deltas, K03's 0.452%, 99.774% and 6.335%, K10's 0.200%, the Clopper-Pearson limits 1.417%, 0.675% and 0.324%, and the zero-error requirements of 299, 2995 and 29956 benign observations.
  • The authors decline every convenient imputation: the fifteen unallocated K10 cases are left unfilled, equal attack weights are not imposed on K07's normalized rows (Section 3.3), K17's Table 4 cross-validation means are not substituted for its Test matrices, and no McNemar test or paired interval is reported where predictions are unavailable (Section 4.3).
  • The sample-size argument in Section 4.2 is a useful corrective: even with zero false positives, the largest benign denominator in the corpus (9251) cannot support a 0.01% reference under the stated procedure.
  • Table 1 states the denominators at each stage and warns explicitly that its rows are nested or overlapping, which makes the selection auditable rather than merely reported.
  • The reproducibility artifact is versioned, DOI-identified and license-clear (Zenodo 2.2.0, GitHub release v2.2.0).

Recommendation

Major Revision for ACM Digital Threats: Research and Practice. The arithmetic is sound and the claims are deliberately bounded, so this is not a case of evidence failing to support conclusions. What needs substantial work is the warrant for the extraction itself and the honesty of the framing about yield: after the provenance rule is applied, the inferential residue is one five-point comparison, one benign row and four ranges, and the extraction rests on a single AI-assisted pass. All three concerns below are addressable within the manuscript's own design.

Major Concerns

  1. The extraction has no independent verification (Section 3.2). The value of this manuscript rests entirely on extraction fidelity, and Section 3.2 states that an OpenAI Codex assistant supported "extraction design, source retrieval and interpretation, transcription, code development, deterministic calculations, figure generation and manuscript drafting," that a second technical pass also used the assistant, and that "Both passes used the assistant and do not constitute independent human extraction." The authors further note that the computational checks "address arithmetic and consistency rather than establishing sampling or provenance assumptions." The arithmetic is therefore well covered, as my own recomputation confirms, but the interpretive judgments on which every tier assignment depends — which class is benign, which way a matrix axis runs, whether a source "describes" a matrix as a training result — no check has touched. "The authors confirmed completion of human validation" does not say what was validated against what. The remedy is within reach: a genuine second extraction of the 24 points by a co-author against the source objects, reporting disagreements, their resolution, and an agreement measure for the coded fields. For a secondary analysis, this is the one methodological guarantee that cannot be substituted.

  2. The provenance rule penalises disclosure (Sections 3.3 and 5.1). K03 is placed in the hypothetical-only tier because its Section 3 describes test-based adjustments; K10 and K17 remain in the conditional tier because no comparable disclosure was found. But Section 3.2 concedes that consultation "does not imply access to complete methods, all figures or all supplementary artifacts," and six included articles have no recovered methods text at all. A source is therefore demoted for disclosing a practice its peers may share undisclosed, and tier assignment is confounded with how much of each text was recovered. Section 5.1 states only the one-directional caveat that retaining K10 and K17 "does not certify their independence." The authors should state the asymmetry directly and test it: report, for each of the four count studies, how much methodological text was recovered, and say whether the tiers would survive equal recovery depth.

  3. The reconstructed yield does not match the framing (Abstract, Sections 4 and 6). Of the 24 points, the authors themselves place fourteen beyond inference and four at range-level only, leaving six they label independence-unverified. Both quantitative headlines in the abstract come from single studies, and the title's "False-Alarm Improvements" rests on one comparison in K17. Section 5.3 concedes that the five main-figure articles are "a particularly selected subset of the frame," but the abstract and conclusions do not carry that concession. The apparatus of tiers, interval blocks and sensitivity subsets is also large relative to what it adjudicates. I would recommend reframing the contribution around what the audit actually established — the K11 reversed headings, the K17 split discrepancy, the K10 unallocated cases, the K13/K21 family, and the sample-size limit — and stating the single-study basis of the headline comparison in the abstract.

Minor Comments

  1. Section 4.2 reports K10's TPR as the point value 99.800% under stated conditions, where the manuscript's own treatment of K07 would call for the admissible range, which its numbers fix at 99.425% to 99.800%.
  2. The TPR improvements of 5.083, 4.606 and 6.918 percentage points over LR, KNN and DT (Section 4.3) are given without the underlying counts, unlike the QSVM and RF points, as are K03's QPU-0 and QPU-1 FPRs of 13.575% and 79.186%.
  3. The 51.6% relative FPR decrease (Section 4.3) invites exactly the reading the introduction warns against, and would be better omitted than disclaimed.
  4. The frozen database's 466 records (Section 3.1) are mentioned once without a breakdown by category.
  5. Section 5.2's argument that randomly splitting related flows, packets, hosts or sessions yields records without correspondingly many independent observations should cite Pendlebury, F.; Pierazzi, F.; Jordaney, R.; Kinder, J.; Cavallaro, L. TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time. USENIX Security 2019, pp. 729-746, which is absent from the reference list.
  6. The zero-error requirements in Section 4.2 are the rule of three and should be attributed to Hanley, J.A.; Lippman-Hand, A. If nothing goes wrong, is everything all right? Interpreting zero numerators. JAMA 1983, 249, 1743-1745.
  7. Section 2 states that source-specific disagreements with the reviews are not inferred where an identical detector, subset and metric cannot be matched, but never reports whether any disagreement was found where a match was possible.
  8. Because source copies "are retained privately and are not redistributed" (Data Availability) while Table 2 locations "refer to the versions in SI", readers cannot re-check the correspondence claims without re-acquiring each source; stating per-source version identifiers and page anchors in the SI would mitigate this.
  9. The provenance tiers are the manuscript's reusable product but are defined only through the four count studies they were applied to; a short operational statement of the coding criteria, usable by a reader on a new corpus, would increase their value considerably.
  10. Figure 1's rounding note ("nearest two-decimal entries, plus or minus 0.5 percentage points") is confusing until the reader infers that the source entries are proportions.