Uncertainty Evaluation to Support Safety of the Intended Functionality Analysis for Determining the Applicability of Performance Targets Across Triggering Conditions in ML-Based LiDAR Object Detection
Milin Patel
Abstract
For safety of the intended functionality (SOTIF) analysis according to ISO 21448, a machine learning object detector for light detection and ranging (LiDAR) point clouds can meet a performance target, a limit on the mean per-frame share of wrong detections among accepted detections, by rejecting detections above an uncertainty threshold. A threshold selected on data of one condition carries a statistical guarantee only for data distributed like its selection data, whereas SOTIF analysis covers every triggering condition. This paper presents an uncertainty evaluation in which the threshold is transferred into a triggering condition, a test decides at a stated error probability whether the target is applicable there, not applicable or not determined, and a lower bound counts the independent units an applicable outcome requires. For PointPillars ensembles on KITTI and nuScenes, counted in recorded sequences, as the dependence between frames requires, neither dataset held enough independent units for any target limit, and no transfer had a not applicable outcome. Counted in frames, which are not independent, the same data gave not applicable outcomes at a target limit after the change of dataset and detector and, on some splits, after a change of distance, and at a limit above the target limits after transfers into far distance and into rain. Thresholds that met the target limit after a transfer rejected most correct detections, and in rain the area under the receiver operating characteristic curve did not decrease. A performance target met under one condition therefore counts as evidence under a triggering condition only after a test there, reported with its share of accepted correct detections and its number of independent recorded sequences.
Review reports
Version v1
xPeerd (AI review)
AI Review (xPeerd — DAReview)
Summary
This manuscript studies whether an uncertainty threshold for ML-based LiDAR object detection, once selected to satisfy a performance target under one condition, remains valid when transferred to another condition relevant to SOTIF analysis. The paper’s central contribution is an “applicability test” that classifies such transfers as applicable, not applicable, or not determined at a stated error probability, and a lower bound on the number of independent units needed to show that a target limit is met. The study evaluates PointPillars ensembles on KITTI and nuScenes across distance, weather, and dataset/detector changes. The main strength is that the paper addresses a practically important assurance question that is usually glossed over: a guarantee obtained under one data distribution should not be presumed under another. The manuscript is also commendably explicit about independence assumptions, sequence-vs-frame counting, and threats to validity. However, several limitations materially constrain the evidential force of the claims: the frame-level findings rely on a unit of analysis that the manuscript itself says violates independence; the weather analysis uses only 24 rain scenes; the detector construction introduces avoidable biases; and the practical safety meaning of the proposed target remains incomplete without a jointly enforced missed-object target. These issues do not invalidate the manuscript, but they do limit generalizability and the current strength of the SOTIF-facing conclusions.
Potential Major Revisions
- The manuscript’s headline empirical findings depend heavily on a frame-level analysis that the authors themselves argue overstates evidence.
A central tension runs through the paper. Early on, the manuscript states that “Recorded data can show that a limit is met only if they hold enough independent units” and that “Frames of one recorded sequence … are not independent of each other.” This is reiterated in the validity discussion: “Because frames of one recorded sequence are not independent, the frame-level evaluation overstates the evidence.” Yet many of the most concrete findings highlighted in the abstract and conclusion are derived from frame-level outcomes, including that transferred thresholds became not applicable under far distance, rain, or dataset changes, whereas sequence-level analysis found that “neither validation set is large enough to show a target limit to be met, and no transfer gave a not applicable outcome.”
This creates a substantial interpretive problem. If the independence assumption is violated at frame level, then the most actionable negative results are not on equal epistemic footing with the methodological standard the paper itself defends. The manuscript should revise its framing so that frame-level findings are explicitly secondary, exploratory, or sensitivity analyses, rather than appearing as principal empirical evidence. As written, the paper risks overstating practical conclusions from an analysis unit it regards as statistically invalid for strict guarantees.
- The weather-triggering-condition analysis appears underpowered and potentially confounded.
The manuscript defines rain scenes in nuScenes by text descriptions containing “rain” and excludes night scenes, but acknowledges that “The validation set holds 24 rain scenes in daylight,” that the label “gives no rain intensity,” and that “clear and rain scenes also differ in location and traffic.” It further admits in the statistical-validity discussion that the AUROC values “in rain … rest on frames of 24 recorded scenes.”
These admissions matter because one of the paper’s more interesting conclusions is that in rain, not-applicable outcomes can occur even when AUROC does not decrease, suggesting changes in spread over frames rather than separability deterioration. The paper states: “In rain, the estimated separation did not weaken. The AUROC of 𝑢e did not decrease … The FAP rose more than the pooled share, so the spread of wrong detections over frames added to the effect of more wrong detections.” That claim is plausible, but given the small number of rain scenes and acknowledged scene-level confounding, it currently reads stronger than the evidence warrants. The manuscript should either downgrade causal language further or add robustness analysis showing that this pattern is not an artifact of scene composition.
- The detector-merging procedure introduces methodological asymmetries that may distort both correctness and uncertainty.
Section 4.1 contains several choices that could materially affect the measured FAP and retention. Most notably, “The boxes of the first member are taken in descending order of confidence,” and “Every detection is anchored on a box of the first member, so a car that this member misses forms no detection even if the other three members detect it.” The manuscript also notes that “The heading of a merged box is the arithmetic mean of the member headings,” which “can misplace the box” when angles straddle ±180° or differ by about 180°, causing a box to “count as wrong.”
These are not minor implementation details; they can alter both the error process and the uncertainty indicators. Since the paper’s thesis concerns validity of threshold transfer across conditions, the study should do more to establish that findings are not artifacts of this particular aggregation rule. At minimum, the manuscript should provide a sensitivity discussion of why first-member anchoring and arithmetic heading averaging are acceptable for this evaluation, and ideally quantify the likely distortion. Without that, the reader cannot easily separate properties of the proposed applicability framework from properties of this specific detector-construction heuristic.
- The practical safety argument remains incomplete because the evaluated performance target addresses only accepted wrong detections.
The paper directly acknowledges a major asymmetry: “The performance target limits only the wrong detections that are accepted. A missed object can be a more hazardous error than an accepted wrong detection, so a lower threshold exchanges one error for another.” This is especially important because the conclusion emphasizes that some applicable transferred thresholds achieved the target “by rejecting most correct detections.”
This limitation weakens the SOTIF relevance of the results in their current standalone form. The manuscript already proposes that “a safety argument … needs a second target on missed objects.” I think this point should be elevated from a discussion remark to a major qualification near the abstract, introduction, and conclusion. Otherwise, readers may interpret the proposed applicability test as more safety-complete than it currently is. If feasible, even a small companion analysis on missed-object retention tradeoffs under transferred thresholds would significantly strengthen the manuscript.
- The manuscript’s contribution is conceptually sound, but the novelty claim should be sharpened relative to existing distribution-shift and risk-control literature.
The paper states that “The contribution lies in their use for the SOTIF question of a transfer across conditions” and not in proposing a new uncertainty indicator, detector, or threshold selection method. That is a fair and appropriately modest claim. However, because the method combines “known statistical elements” including LTT, a confidence interval on held-out data, and the zero-failure bound, the manuscript should more crisply articulate exactly what is new at the methodological level versus what is new only in application framing.
At present, the novelty appears to be primarily the operationalization of transferability testing for SOTIF conditions, not a fundamentally new statistical procedure. That is still publishable, but the paper would benefit from a more explicit statement of the conceptual novelty, decision logic, and implementation pipeline that distinguish it from generic distribution-shift validation. This would help prevent readers from seeing it as an engineering recombination without sufficient methodological originality.
- The paper needs a stronger anti-fraud / coherence clarification regarding scope and real-world applicability.
I do not see obvious paper-mill traits or fabricated conceptual structure. The manuscript is internally coherent, transparent about limitations, and technically specific. That said, there is one coherence risk the paper should address more explicitly: it repeatedly reminds readers that “Evidence for a product requires the test on data from its ODD, which KITTI and nuScenes do not represent.” This is appropriate, but it also means the real-world assurance implications are necessarily indirect. The abstract and conclusion should therefore avoid any wording that could be read as validating deployment-ready SOTIF evidence on the basis of public benchmarks alone. Tightening that language would improve integrity and reduce overclaim risk.
Potential Minor Revisions
- Typographic, spacing, and line-break issues
The PDF extraction shows several broken words and spacing inconsistencies that should be corrected in the source manuscript if they also appear in the typeset version:
- “therefore” in the conclusion/discussion transition on page 21 should be corrected to “therefore.”
- “screening” in “Use of generative AI. Generative AI tools were used to assist with the literature screening” on page 22 should be “screening.”
- “applicable or not applicable. The applicability is not determined if the data show neither” is clear, but the formatting around this paragraph appears uneven in extraction and should be checked in the source layout.
- “target limits(the required limits of the RQs)” on page 14 is missing a space before the parenthesis and should read “target limits (the required limits of the RQs).”
- “called target limits(the required limits of the RQs). The exploratory limits α = 0.05 and 0.1, marked † , were fixed…” also shows punctuation/spacing inconsistency around the dagger marker and parenthetical phrase.
- In the references, multiple entries show line-break hyphenation artifacts such as “functionality,” “Contributing,” “perception,” and “riskcontrolling,” which should be normalized in final production.
- Stylistic and wording improvements
- In the abstract, the sentence beginning “For PointPillars ensembles on KITTI and nuScenes, counted in recorded sequences…” is dense and difficult to parse because it stacks several subordinate clauses and contrasts. Consider splitting it into two or three shorter sentences.
- The sentence “A performance target met under one condition therefore counts as evidence under a triggering condition only after a test there…” is strong and effective, but it appears both in the abstract and conclusion in nearly identical form. Consider slight rephrasing in one location to avoid redundancy.
- “This paper does not show that they initiate a hazardous behaviour” is honest, but in Section 4.2 the sentence may read as slightly abrupt; it could be integrated more smoothly with the rationale for treating far distance and rain as potential TCs.
- Formatting and presentation issues
- Figures and tables appear central to the argument, especially Fig. 1, Fig. 3, Fig. 4, Fig. 5, and Fig. 6, but the prose occasionally asks the reader to integrate many moving parts across sections. Additional recap captions or a compact notation table could improve readability.
- Section 6.2 “Threats to Validity” is strong, but some of its limitations are severe enough that cross-references earlier in the manuscript would help. For example, the issue that the detector members on nuScenes “were trained for about half of the configured epochs” deserves earlier visibility, not only in the external-validity subsection.
- The manuscript uses both target limits and exploratory limits. A visually distinct table summarizing which limits define actual performance targets versus exploratory stress-test thresholds would help prevent misreading.
- AI content analysis for this post-2021 manuscript
Estimated percentage of AI-generated or AI-assisted content: low-to-moderate AI assistance is plausible, but strong direct AI authorship signals are not dominant. A cautious estimate would be 15–30% AI-assisted phrasing, consistent with the authors’ explicit declaration that “Generative AI tools were used to assist with the literature screening, the analysis and figure scripts, and the writing of the text.”
Highlighted AI-detected sections:
- The abstract and conclusion use highly polished, balanced, and repetitive framing that is stylistically consistent with assisted drafting, especially the recurring claim that evidence under a triggering condition requires testing there.
- The “Threats to Validity” section is unusually systematic and template-like in its subdivision into construct, internal, statistical, and external validity, which could reflect structured AI-assisted editing, although it is also normal scholarly practice.
- The declaration of AI use itself supports the inference of partial AI assistance.
Assessed epistemic impact:
- The declared use of generative AI does not, by itself, undermine the manuscript’s contribution.
- The main epistemic concern is not AI authorship but whether AI may have amplified rhetorical confidence in claims whose evidential base is narrower than the prose suggests, especially around frame-level findings.
- Since the paper openly declares AI use and the technical content is detailed, field-specific, and limitation-aware, I do not see this as a hidden-authorship concern. The more relevant issue is precision of claims, not provenance of wording.
Recommendations
-
Reframe the evidence hierarchy more strictly. Make sequence-level analysis the primary inferential basis and present frame-level results explicitly as exploratory or illustrative sensitivity analyses.
-
Strengthen the weather-condition analysis. Either add robustness checks for the 24-scene rain subset or more clearly narrow the claim to a preliminary observation under limited and confounded data conditions.
-
Add a sensitivity discussion, or preferably an ablation, on the ensemble-merging rule. The consequences of first-member anchoring, vote-based detection formation, and arithmetic heading averaging should be empirically bounded.
-
Elevate the missed-object issue. Because the manuscript shows that some thresholds meet FAP limits by rejecting many correct detections, a paired target on retained true objects should be presented as necessary for any realistic safety argument, not merely desirable future work.
-
Clarify novelty with sharper language. State explicitly that the methodological contribution is the SOTIF-specific decision framework for threshold transfer across conditions, built from known statistical components, rather than a new statistical guarantee in the abstract.
-
Tighten the practical-application claims. Reiterate near the abstract and conclusion that benchmark-based evidence does not substitute for ODD-specific product evidence.
-
Revise for readability. Shorten several long sentences in the abstract, introduction, and conclusion, and correct spacing/hyphenation artifacts throughout the manuscript.
Overall, this is a serious and coherent manuscript with a relevant assurance-oriented contribution. Its strongest asset is not the particular detector result but the insistence that target satisfaction must be re-tested under changed conditions rather than assumed to transfer. Its main weakness is that the empirical evidence most clearly demonstrating that point is presently strongest where the independence assumption is weakest. That imbalance should be corrected before publication.
XSci (AI review)
Summary
ISO 21448 requires a performance target for a perception element to hold under the triggering conditions of the operational design domain, whereas an uncertainty threshold selected with Learn then Test carries a guarantee only for the distribution of its selection data. The authors turn this gap into an applicability test: a threshold selected in one condition is transferred unchanged to another, a Hoeffding–Bentkus interval [L, U] for the false acceptance proportion is computed there on held-out data, and the outcome is applicable (U ≤ α), not applicable (L > α) or not determined, with a zero-failure bound (Eq. 5) for the number of independent units required. The test is applied to ensembles of four PointPillars networks on KITTI and nuScenes across changes of object distance, of weather, and — as a control — of dataset and detector. Counted in frames, 168 of 240 outcomes are informative; counted in recorded sequences, only 5 of 240 are, and no transfer gives not applicable.
Contribution and Significance
The contribution is an evaluation protocol and a data-sizing rule rather than a new statistical method, as Sect. 1.2 states plainly. Its value lies in converting an assurance question into a decision with three named outcomes, each attached in Table 5 to a measure for the detector supplier or the vehicle manufacturer, and in Eq. (5), which gives a number of independent units per condition computable before recording begins. The second contribution, and the firmer one, is the negative result of Sect. 5.3: with the recorded sequence as the unit, the KITTI and nuScenes validation sets are one to two orders of magnitude too small to show any pre-registered target limit to be met.
Relative to the closest prior work the novelty is real but narrow. Kumar et al. [32], Lee et al. [33] and Li et al. [34] select or adapt a threshold per condition but never test a limit in a second condition; [5] and [6] repair a guarantee under shift but are cited rather than used. What is new is the three-way decision, its mapping onto ISO 21448 measures, and the measurement. The underlying insight — a guarantee for one distribution says nothing about another — restates the assumption of LTT; the paper is strongest where it quantifies that consequence, not where it announces it.
Strengths
- The study is pre-registered (Sect. 4.4): δ, the limits, the conditions, the per-frame form of the FAP, and predictions with what would count as their failure were fixed and date-stamped in version control before any outcome was computed.
- Equation (3) identifies that the target is trivially met when almost no frame holds an accepted detection, and the authors therefore report retention with every applicable outcome (Fig. 4, right).
- Equation (5) is correct, simple and actionable: 299 units for selecting and 368 for testing at α = 10⁻², rising to 2,995 at 10⁻³, enters a recording plan directly.
- The three evaluations of Sect. 4.3 separate the effect of the division from that of the unit, and Sect. 6.2 states the threats to validity unusually completely.
Recommendation
Major Revision for SAFECOMP. The instrument and the data-volume result are a contribution the venue's readership can use, and the negative finding is reported rather than buried. What stands in the way is that the answers to RQ1 and RQ2 rest on an evaluation the authors declare not to carry the stated error probability, and that the one target-limit result robust across splits is the control transfer. Both are addressable within the paper's own approach.
Major Concerns
-
The empirical answers rest on an evaluation the paper declares invalid (Sects. 5.1, 6.1, 6.2; caption of Fig. 4). Section 6.2 states that "the frame-level evaluation overstates the evidence" and Sect. 6.1 that the unit "is the recorded sequence unless the independence of frames is shown". In the sequence-level evaluation only 5 of the 240 outcomes are informative, all on nuScenes and all applicable at the exploratory limit 0.1†, which by Sect. 4.4 defines no performance target. Every substantive answer to RQ1 and RQ2 — far distance, rain, the dataset transfer, the accompanying changes of Table 4 — therefore comes from the frame-level evaluation, which carries no guarantee, yet the abstract and conclusion present these findings without that qualification.
The authors already cite the tool that would close this: Barber et al. [6] bound the loss of guarantee for non-exchangeable data. A sequence-level block bootstrap, an estimate of the effective number of independent units, or an inflated n in the HB p-value would give the frame-level results a valid, if weaker, error probability. Failing that, they should be labelled exploratory wherever they appear, the abstract included.
-
The one robust target-limit result is the control, and the abstract does not separate target from exploratory limits (Sects. 1.1, 4.1, 4.4, 5.1; Table 3). At 10⁻², the outcome was not applicable on all five splits only for the transfer of the u_s threshold from nuScenes to KITTI. Section 1.1 states that a change of dataset "does not isolate a TC" and serves as a control; it also changes the detector, and Sect. 4.1 reports that the nuScenes members were trained "for about 10 of its 20 epochs", with Table 3 showing that detections after the vote match only 0.398 of 65,465 annotated cars. The firmest transfer is thus the one bearing least on triggering conditions, and it is confounded with an undertrained detector. For the potential triggering conditions, no weather transfer gave not applicable at 10⁻², and distance gave it on one and on three of five splits — the order of the "at most 6" erroneous outcomes among 240 that Sect. 6.2 computes. No threshold could be selected at 10⁻³ or 10⁻⁴.
The abstract nevertheless reports "not applicable outcomes at limits fixed in advance after transfers into far distance, into rain and between two separately trained detectors", merging the target limits with exploratory limits that define no performance target. Please separate the two throughout, give the split counts in the text, and either repeat the dataset transfer with a fully trained nuScenes ensemble or state that its outcome cannot be attributed to the change of condition alone.
-
No applicable outcome in the study is usable evidence (Sects. 3.1, 4.1, 5.1, 6.1). By Eq. (3), A_c(τ) ≤ P(N_f(τ) > 0), so a threshold that rejects nearly everything meets any limit; Sect. 5.1 confirms that "A retention near zero, as for u_s from KITTI to nuScenes, means that almost no frame holds an accepted detection, so the limit is met by Eq. (3)". Transferred thresholds with an applicable outcome at 10⁻² had a retention of at most 29 % on every split, and every u_e threshold at that limit lay below 0.25, accepting only detections found by all four members and so merely re-applying the vote of Sect. 4.1. The second target on missed objects that would make an applicable outcome meaningful is set out at the end of Sect. 6.1 but never instantiated, although the same interval and Eq. (5) apply to it unchanged. Instantiating it on the same data and reporting joint outcomes would show the test used positively, which the evaluation never does.
Minor Comments
- The symbol–definition pairing in Table 2 appears shifted by one row (n is given as "number of accepted detections in f_c" and L, U as "number of independent units").
- Equation (5) is argued informally yet asserted to hold "for every test that is valid for any distribution of the per-unit values in [0, 1]"; a formal statement and proof would make it the theorem it deserves to be.
- Cite the rule of three: Hanley, J.A., Lippman-Hand, A.: If nothing goes wrong, is everything all right? JAMA 249(13), 1743–1745 (1983).
- The claim that AUROC does not fall in rain rests on 0.844 → 0.856 (Table 4), point estimates on split 0 that Sect. 6.2 itself notes carry no interval.
- The cross-dataset AUROC of Table 4 (0.749 → 0.905) compares two separately trained detectors on different data, so state what it can and cannot show.
- The near/far boundaries differ between datasets (34.8 m, 21.4 m; Table 3) and are medians of the detector output, so "far distance" is not one condition across them.
- Missing, and relevant to RQ3: Kalra, N., Paddock, S.M.: Driving to safety: how many miles of driving would it take to demonstrate autonomous vehicle reliability? Transportation Research Part A 94, 182–193 (2016).
- Missing from Sect. 2.3: Feng, D., Harakeh, A., Waslander, S.L., Dietmayer, K.: A review and comparative study on probabilistic object detection in autonomous driving. IEEE Trans. Intell. Transp. Syst. 23(8), 9961–9980 (2022).
- Results that central claims depend on are supplementary-only (Sects. 4.4, 5.2 and 6.1) and should be summarised in the paper.
- The novelty claim of Sect. 2.4 rests on "a documented search" that is not described; give databases, queries and dates.
- No computational cost is reported for the M = 4 ensembles or the 100-candidate grid.
- A second architecture or ensemble size would show the outcomes are not artefacts of PointPillars with four members.
Reviewer3 (AI review)
AI Review (reviewer3.com)
Comments (3)
1. Selection Conditioning Omitted in Test Interval — Major
(Fig. 2c). For a given true FAP, only one of the two erroneous outcomes is possible, so the test gives an erroneous applicable or not applicable outcome with probability at most 𝛿 / 2 . This holds if the test units are independent draws from 𝑐 ′ and independent of the selection units. This
The definition of the confidence interval [L, U] and its stated error probability delta/2 treat the selected threshold tau_c as fixed, omitting the conditioning on the selection event in condition c. While independence between the selection and test sets ensures the conditional coverage holds for each selected tau_c, explicitly writing the probability as conditional on the selection data (and on a threshold being selected at all) would make the formal statement in Section 3.3 complete.
2. Exact Threshold Required for Zero-Failure Bound — Major
and to the outcome applicable. If every per-unit value is 0 or 1 and the true FAP lies
Equation (5) is presented as an exact inequality for any valid test where fewer false acceptances do not weaken evidence, based on the argument that a true FAP just above alpha yields zero false acceptances with probability close to (1-alpha)^n. However, if the true FAP is strictly greater than alpha (i.e., alpha + epsilon), the probability of zero failures in a Bernoulli model is (1 - alpha - epsilon)^n, which only approaches (1-alpha)^n in the limit as epsilon goes to 0; furthermore, for the specific Hoeffding-Bentkus p-value used in the paper, the zero-failure p-value is min(exp(-n * h(0, alpha)), e * (1-alpha)^n) = (1-alpha)^n, so the bound holds exactly for the HB test by direct substitution of A_hat = 0 rather than requiring an approximate limiting argument.
3. No Uncertainty Quantification for Mechanism Metrics — Major
indicator each. Shares of wrong detections: all validation frames; other values: test part of split 0, FAP and pooled share at the transferred threshold selected at 0.1
The analysis of the four factors accompanying 'not applicable' outcomes in Section 5.2 and Table 4 relies on single-split point estimates (split 0) for AUROC, median indicator values, and pooled vs. frame-averaged shares without confidence intervals or cross-split variation. Although the text notes in Section 6.2 that AUROC values are point estimates on split 0, reporting the range or variability across the five splits (or bootstrap intervals on split 0) for the median shifts and pooled-vs-FAP differences would clarify that the attributed mechanisms—especially in the small rain subset—are not artifacts of a single random partition.
Version v2
Reviewer3 (AI review)
AI Review (reviewer3.com)
Comments (5)
1. Dependence between selection and test parts in distance transfers — Major
boundary is the median distance of the car detections in the grid part (Sect. 4.3). A frame contributes its near detections to the near condition and its far detections to the far condition, so near and far share frames.
In the frame-level evaluation, frames are randomly split into grid (10%), selection (45%), and test (45%) parts, and each frame contributes its near detections to the near condition and its far detections to the far condition. Because the same split assignment is used for both conditions, the selection part of condition c (e.g., near) and the test part of condition c' (e.g., far) are disjoint sets of frames, whereas in the self-check or if frames were split per condition they could overlap; however, the manuscript notes in Section 6.2 only that 'Near and far share most KITTI test frames, so the test data of the two conditions are not independent.' It would be helpful to explicitly confirm in Section 4.3 whether the 10%/45%/45% frame split is performed once on all frames (so selection frames of near and test frames of far are disjoint) or separately per condition (which would cause near selection frames and far test frames to share the same physical frames).
2. Uncertainties omitted for diagnostic metrics in Table 4 — Major
(Fig. 5). The medians of u e for correct and for wrong detections moved by at most 0.03, so the values did not shift. The FAP rose more than the pooled share, so the spread of wrong detections over frames added to the e ff ect of more wrong detections. Rain and
In Section 5.2 and Table 4, conclusions about whether indicator medians shifted, whether wrong-detection shares changed, and whether FAP rose more than the pooled share are based on point estimates from split 0, with a bootstrap confidence interval provided only for the clear-to-rain AUROC difference. Providing uncertainty intervals (or ranges across splits) for the indicator medians, FAP, and pooled share in Table 4 would substantiate the claims that the median shift of <= 0.03 in rain is negligible while the difference between FAP (0.09 -> 0.17) and pooled share (0.12 -> 0.17) represents a meaningful frame-spread effect.
3. Binomial tail inequality direction for lower bound L — Major
. The upper confidence bound U of A c ′ ( τ c ) is obtained by inverting the HB p -value at error probability δ/ 2, and the lower confidence bound L is one minus the same bound computed from the values 1 − a f , so [ L , U ] is an interval at confidence level 1
In Section 3.3, the lower confidence bound L is defined as one minus the upper HB bound computed from the transformed values 1 - a_f. However, the Bentkus binomial term in the HB p-value of Section 3.2 rounds the empirical sum up using the ceiling function (⌈n A_hat_c(tau_j)⌉), which is conservative for an upper bound (left tail) but anti-conservative when applied to 1 - a_f to bound the mean from below unless the ceiling is replaced by a floor (or the right tail is bounded directly). Please clarify the exact formula used for L to confirm that the finite-sample guarantee of δ/2 holds for the 'not applicable' decision.
4. Zero-failure bound does not hold for all distribution-free tests — Major
rule of three [40]). The same bound holds for every test that is valid for any distribu- tion of the per-unit values in [0 , 1] and in which fewer false acceptances never weaken the evidence, because a true FAP just above α gives n units without a false acceptance with a probability close to (1
Section 3.5 states that the zero-failure bound n >= ln(delta') / ln(1 - alpha) holds for every distribution-free test in [0, 1] where fewer false acceptances never weaken the evidence. While this holds for the HB test and for Bernoulli variables, a random variable in [0, 1] with expectation alpha can have P(X = 0) arbitrarily close to 1 - alpha (not (1 - alpha)^n per sample, unless X is Bernoulli); actually, by Markov's inequality on 1 - X, P(X = 0) <= 1 - alpha, so for n i.i.d. variables P(all X_i = 0) <= (1 - alpha)^n is indeed maximized when X_i ~ Bernoulli(alpha), meaning an exact distribution-free test could achieve level (1 - alpha)^n, except the text justifies this by saying 'a true FAP just above alpha gives n units without a false acceptance with a probability close to (1 - alpha)^n', which is only true when the per-unit distribution is concentrated on {0, 1} rather than general distributions in [0, 1] (e.g. a constant X_i = alpha gives zero false-acceptance units with probability 0). Clarifying that the worst-case distribution in [0, 1] is the two-point Bernoulli(alpha) distribution would make this argument rigorous.
5. Selection-dependent sample size n in positive-detection frames — Major
where fc denotes the car detections of condition c in frame f , a frame f belongs to c (f ∈ c) if fc is not empty, and a frame without accepted detections in fc contributes af = 0.
By defining a frame f as belonging to condition c only if fc is non-empty, the set of frames and the denominator n for the near and far distance conditions depend on the trained detector's pre-threshold detections. In a SOTIF context, conditioning the sample space on frames where the detector fires—rather than all operational frames where objects may be present or absent—means A_c(tau) does not reflect a fixed operational exposure unit across different detectors or retraining runs.
xPeerd (AI review)
AI Review (xPeerd — DAReview)
Summary
This manuscript addresses an important and technically specific safety-validation problem for ML-based LiDAR perception in automated driving: whether an uncertainty threshold that satisfies a performance target in one condition can be transferred to another condition while preserving that target. The paper is strongest when it clearly formulates this as a statistical applicability problem under ISO 21448/SOTIF and when it distinguishes threshold validity from AUROC-based separability. It also contributes a practically relevant negative finding: sequence-level independence requirements are so demanding that the available KITTI and nuScenes data are insufficient to verify target satisfaction under the tested conditions .
The manuscript’s main strengths are conceptual rigor, explicit research questions, and honest reporting of limitations and non-results. The most significant weaknesses concern external validity, operational interpretation of the safety target, construct choices that may materially affect conclusions, and the practical usefulness of “applicable” outcomes when retention is poor. The work is coherent and methodologically serious, and I do not see obvious paper-mill traits or internal incoherence suggestive of fabrication. However, several claims of practical relevance would benefit from sharper qualification, stronger sensitivity analysis, and clearer linkage between the detector-level metric and downstream safety implications .
Potential Major Revisions
- Clarify the practical safety meaning of the chosen target and strengthen the link from detector-level FAP to system-level safety.
The paper states that “The performance target of this paper bounds the false acceptance proportion (FAP), the share of wrong detections among the accepted detections of a frame averaged over frames, by a limit α” and also notes that “every rejected correct detection is a car that the motion planner does not receive” . Later, the manuscript itself acknowledges a key limitation: “A safety argument that uses the threshold therefore needs a second target on missed objects” . This is a major issue because the manuscript’s core positive/negative applicability conclusions are based on one-sided control of accepted wrong detections, while the main practical harm of conservative thresholding may be missed true objects.
The paper should therefore do more than mention this in discussion. It should operationalize the limitation earlier and more centrally. In particular, when the manuscript concludes that “a transferred threshold with an applicable outcome at 10−2 rejected most correct detections” , this strongly suggests that applicability with respect to FAP alone may be safety-theoretically insufficient or even misleading. I recommend reframing the contribution as a partial assurance method, not a sufficient basis for SOTIF evidence, unless accompanied by a second tested criterion on retention/recall. At minimum, the abstract, introduction, and conclusion should explicitly state that the proposed test does not by itself establish acceptable perception safety.
- Better justify the construct validity of “wrong detection” and assess sensitivity to alternative matching rules.
The manuscript openly states that “A wrong detection is defined by the annotations and the overlap criterion, so boxes on annotated cars that miss the overlap criterion and boxes on unannotated objects count as wrong” . It also notes that “This paper uses an IoU of 0.5 for both datasets, whereas the KITTI benchmark uses 0.7 for cars and nuScenes matches by the distance between box centres” . These are not minor implementation choices; they shape the label space that drives both threshold selection and transfer conclusions.
This concern is amplified by the earlier acknowledgment that, in an illustrative KITTI frame, “an unannotated object under the box of the wrong detection cannot be excluded” . Because the entire argument hinges on false acceptance behavior, the paper should include a sensitivity analysis using at least one alternative correctness criterion per dataset. Without that, readers cannot determine whether the reported not-applicable or not-determined outcomes reflect actual threshold-transfer failure or annotation/matching artifacts. This matters especially because the paper aims to support safety analysis rather than only benchmark evaluation.
- Strengthen the independence argument with either empirical dependence estimates or a clearer validation design rationale.
A central thesis is that frame-level results are optimistic because “Frames of one recorded sequence… are not independent of each other” and that “Counted in recorded sequences, as the dependence between frames requires, neither validation set is large enough to show a target limit to be met” . This is an important and likely correct concern. However, the manuscript currently moves from observed temporal dependence plausibility to a hard sequence-level independence treatment without quantifying the middle ground.
The discussion admits this as future work: “It will estimate the dependence between frames of one recorded sequence to state the number of effectively independent units” . Given how central this issue is to the paper’s main conclusions, this should not be deferred entirely. Even a partial analysis—e.g., block dependence diagnostics, effective sample size approximations, or sensitivity of decisions to block length—would materially improve the reproducibility and interpretability of the results. Otherwise, the manuscript’s strongest empirical message becomes: frame-level evidence is too weak, sequence-level evidence is too sparse, and therefore the practical calibration regime remains unresolved.
- Reassess the novelty claim with more precise scoping and avoid overstatement.
The manuscript claims that “None of these works, and no other study in LiDAR object detection found in a documented search… tests whether a threshold selected under one condition meets its limit under another” . Table 1 is helpful and the novelty may well be real in the narrowly defined sense of finite-sample tested transferability of an uncertainty threshold across conditions . However, the current wording is broad and should be scoped more carefully.
Because adjacent literature addresses threshold adaptation, uncertainty calibration degradation under shift, and condition-aware assurance, the manuscript should distinguish:
- threshold transfer testing,
- condition-specific threshold selection,
- uncertainty robustness under covariate shift,
- and safety-case decomposition across situations.
The paper’s contribution appears strongest not as “first condition-transfer study” in a broad sense, but as a formal test framework for transferability of a threshold controlling per-frame false acceptance proportion under stated error probability and independence assumptions. That narrower novelty claim would be more defensible.
- Provide fuller justification for the operationalization of conditions, especially rain and distance.
The paper states that “Far distance and rain are treated as potential TCs” and that a rain scene is identified if the nuScenes text description contains “rain,” while “The label gives no rain intensity and does not separate falling rain from a wet road” . For distance, “the far condition is defined by the detector output, not by the distance of an annotated car,” and “near and far share frames” .
These are understandable compromises, but they limit causal interpretation and may entangle condition labels with detector behavior. In particular, output-defined distance conditions create a risk that condition assignment partly reflects model error rather than environment state. Similarly, coarse rain labels and location/traffic confounding are already acknowledged: “Rain and clear scenes also differ in location and traffic… and are not shown to cause” the observed effects . The manuscript should sharpen its claims accordingly and present the results explicitly as condition-associated rather than condition-caused changes unless stronger controls are added.
- Improve the treatment of operating characteristics beyond AUROC.
The manuscript makes the valuable point that “AUROC measures separation only” and does not capture prevalence or spread over frames . It also cites literature noting that precision-recall analysis can be more informative in imbalanced settings . Since the target is fundamentally about accepted-error proportion, the empirical evaluation would be stronger if precision-recall style summaries or retention-versus-FAP curves were made more central than ROC/AUROC.
This is especially relevant because the manuscript reports that in rain “the area under the receiver operating characteristic curve did not decrease” while not-applicable outcomes could still arise at the exploratory limit . That mismatch is one of the paper’s most interesting findings. It deserves a more systematic visualization and explanation, not only narrative treatment.
- Expand reproducibility details around model training and fixed predictions.
The manuscript commendably provides code and “predictions fixed in advance” via a repository . Still, the training description is brief: “Each ensemble consists of M = 4 PointPillars networks,” trained with OpenPCDet, with KITTI trained for 10 epochs and nuScenes “for about 10 of its 20 epochs” . For a paper where uncertainty estimates depend on ensemble diversity and training configuration, “about 10” is too imprecise. The manuscript should specify exact stopping criteria, random seeds, hardware or software determinism assumptions, and whether model selection used validation information that could interact with the threshold-selection procedure.
Potential Minor Revisions
- AI-use statement requires stricter delimitation of epistemic responsibility.
The paper states: “Generative AI tools were used to assist with the literature screening, the analysis and figure scripts, and the writing of the text” . Because the article is from 2026 and explicitly reports AI assistance, this deserves closer scrutiny. Assistance with “analysis and figure scripts” is more epistemically significant than language polishing. The manuscript should specify which tasks were AI-assisted, what verification procedures were applied, and whether any generated code affected statistical testing or plotting decisions. As written, the statement is transparent but under-specified.
Estimated percentage of AI-generated content: 10–20%.
Highlighted likely AI-assisted or AI-smoothed passages:
- The abstract is unusually uniform in cadence and highly compressed across several linked claims, e.g., “This paper presents an uncertainty evaluation in which the threshold is transferred into a triggering condition…” .
- The concluding synthesis is polished in a manner consistent with assisted summarization, e.g., “A performance target met under one condition therefore counts as evidence under a triggering condition only after a test there…” .
- The declarations section explicitly confirms AI-assisted writing and analysis support .
Assessed epistemic impact:
- Likely low for prose quality alone.
- Potentially moderate if AI contributed materially to analysis or figure scripts without independent validation logs.
- This does not indicate misconduct, but greater procedural transparency would improve trust.
- Some phrasing is dense and could be simplified for readability.
Examples:
- “The test examines whether the target is met under the condition of use, not whether it is required there” is precise but cognitively dense on first reading .
- “For a given true FAP, only one of the two erroneous outcomes is possible” may benefit from an intuitive explanation or short example .
- “The bound is necessary, not sufficient” would be clearer if followed immediately by a practical implication sentence .
- Terminology should be harmonized earlier and more explicitly.
The distinction between FAP and pooled share is important, but the explanation around Eq. (2) is easy to miss. The manuscript should foreground this distinction earlier because several later interpretations rely on it . Similarly, “retention” becomes important to the practical meaning of applicability, but it is introduced after the reader has already formed an impression of the target’s adequacy.
- Figure interpretation could be made more self-contained.
Figure 5 appears central, especially to the claim that AUROC may remain stable while FAP-relevant transferability worsens . The caption is informative, but the body text should more directly walk the reader through one concrete example of how similar ROC operating behavior can coexist with poor target transferability.
- Formatting and typography issues.
- There are visible line-break and hyphenation artifacts in extracted text such as “rang- ing,” “de- tections,” and “fol- lows” in multiple sections, including the abstract and contributions . If these are typesetting artifacts only, they may be acceptable in the PDF layout, but the final camera-ready version should still be checked carefully.
- Mathematical notation and symbols should be checked for consistency across prose and figures, especially α, δ, c, and c′ where accessibility could be improved .
- The exploratory limit notation “0.05†” and “0.1†” may confuse readers unless the dagger notation is explained immediately at first appearance in the main text and repeated in figure captions where used .
- Citation framing should distinguish preprints from archival evidence.
The related-work section cites several preprints alongside archival publications, including arXiv works from 2024 and 2026 . This is acceptable in a fast-moving area, but the manuscript should be careful not to present preprint claims with the same evidentiary weight as peer-reviewed standards or established datasets.
- One table merits stronger interpretation.
Table 5 is useful because it translates each applicability outcome into a downstream action, but those actions mix detector-supplier and vehicle-manufacturer perspectives . The manuscript should state more explicitly whether the proposed framework is intended primarily for component assurance, integrator decision-making, or both.
Recommendations
-
Reframe the manuscript’s claim from “testing applicability of a performance target” to “testing one necessary detector-level condition for safety evidence across triggering conditions.” This would better match the manuscript’s own admission that a second target on missed objects is required .
-
Add a sensitivity study for:
- IoU threshold choices,
- distance-based matching for nuScenes,
- alternative condition definitions where feasible,
- and perhaps different ensemble-vote rules. This would substantially strengthen construct validity and reduce the risk that conclusions are driven by operational definitions rather than robust threshold-transfer effects .
-
Quantify temporal dependence rather than only contrasting frame-level and sequence-level extremes. Even a supplementary effective-sample-size analysis would greatly improve the paper’s practical value .
-
Make retention a co-primary reporting metric throughout the main results, not only a downstream caution. The manuscript’s own findings show that an “applicable” threshold may still be operationally unattractive if it withholds too many correct detections .
-
Tighten novelty wording to the precise methodological contribution: a finite-sample applicability test for transfer of uncertainty thresholds across conditions under explicit independence assumptions .
-
Expand the reproducibility appendix with exact training and threshold-selection details, explicit seed lists, and a compact end-to-end protocol description from raw predictions to final applicability outcomes .
-
Add one concise threat-to-validity paragraph to the abstract or conclusion summarizing the three most important limitations:
- detector-level metric does not fully capture safety,
- sequence-level independence requirements limit demonstrability,
- and condition definitions include unavoidable confounding .
Overall, this is a serious and coherent manuscript with a meaningful methodological contribution and commendable transparency about limitations. Its main revisions should focus on sharpening validity claims, reducing overinterpretation, and making the safety significance of the statistical results more explicit.
XSci (AI review)
Summary
ISO 21448 requires that a perception element meet its performance targets across triggering conditions, but a rejection threshold selected with a distribution-free method such as Learn then Test (LTT) carries its guarantee only for data distributed like the selection data. The authors propose an applicability test: a threshold is transferred unchanged into a condition of use, a two-sided Hoeffding-Bentkus interval on the false acceptance proportion there returns applicable, not applicable or not determined, and a zero-failure bound (Eq. 5) gives the minimum number of independent units an applicable outcome needs. The test is applied to four-member PointPillars ensembles on KITTI and nuScenes across changes of distance, weather, and dataset-plus-detector, with two uncertainty indicators and three target limits. Counted in frames, not applicable outcomes appear at the target limit 10^-2 after the change of dataset and detector and, on some splits, after a change of distance; counted in recorded sequences, only 5 of 240 outcomes are determined at all, and thresholds with an applicable outcome at 10^-2 retained at most 29% of correct detections.
Contribution and Significance
The contribution is not a statistical method, and the authors say so (Sect. 1.2): the test composes an LTT selection, a confidence interval on held-out data and the rule of three. Its value lies in turning a question SOTIF practice leaves implicit into one with a decision procedure attached, and in two consequences the field should adopt. An applicable outcome means little alone, since Eq. (3) shows the limit is met whenever few frames hold an accepted detection, and the measured retentions make the price concrete. And Eq. (5) converts a limit into a recording requirement before any data are collected - 299 units to select and 368 to test at alpha = 10^-2, delta = 0.05 - yielding the most useful result here: 150 nuScenes scenes and 45 KITTI drives cannot show any target limit to be met. Table 5, mapping each outcome to a measure and a responsible party, is usable in a safety process.
Table 1 places the work correctly against the closest prior art, and I found no counterexample: Lee et al. [34] and Li et al. [35] select a threshold per condition, Kumar et al. [33] report abstention per severity, Salay et al. [31] and CUBICS [32] estimate per-condition performance, and none tests a limit in a second condition with a finite-sample guarantee. What is new is narrow but correctly scoped; against that, the empirical yield is thinner than the framing suggests.
Strengths
- The design is pre-registered: delta, the three limits, the conditions, the per-frame FAP and the predicted outcomes were fixed in version control before any outcome was computed (Sect. 4.4), with code and per-run results public.
- The partitioning is careful: a separate grid part (10%) fixes the near/far boundary and the candidate grid, so neither selection nor test part influences them (Sect. 4.3, Fig. 3).
- Section 3.4 decomposes the four changes that can break a transferred threshold and shows AUROC is invariant to three of them - correct, and a useful corrective to the routine use of AUROC as uncertainty-quality evidence.
- The derivations check out: I verified Eqs. (1), (3) and (4), the identity h(0, alpha) = -ln(1 - alpha) behind Eq. (5), and the values 299, 368 and 2,995.
- Section 6.2 is candid, naming the frame dependence, the matching criterion, the undertrained nuScenes ensemble and the KITTI frames shared between near and far.
Recommendation
Major Revision for Reliability Engineering & System Safety. The question is well posed, the construction is correct, and the recording and retention requirements are findings the safety community needs. What stands in the way is that the empirical claims are drawn from the one evaluation the authors themselves declare invalid, while the evaluation they endorse returns almost nothing, and that the AUROC and target-limit claims are each supported by less than the text asserts. These are matters of analysis and framing, repairable within the manuscript's approach and without new data.
Major Concerns
-
The findings rest on the evaluation the manuscript rejects (Sects. 4.3, 5.1, 6.2). Section 6.2 states that "the frame-level evaluation overstates the evidence", and Sect. 6.1 that the unit "is the recorded sequence unless the independence of frames is shown". Yet the sequence-level evaluation yields only 5 determined outcomes of 240, all applicable and all at the exploratory limit 0.1 that defines no performance target (Sect. 4.4), while the answers to RQ1 and RQ2 come entirely from the frame-level run. That run has a further defect noted only in passing: with division by frame, frames of one sequence lie in both the selection and the test part, so the two are not independent, as Sect. 3.3 requires them to be. The third run (division by sequence, frames as units) is the only configuration free of that leakage, is reported in two sentences, and its one figure moves against the frame-level result - 0 of 112 same-condition not applicable outcomes becomes 5 of 113 - without comment. Report the third run in full as the primary analysis, and replace the collapse of a sequence to one mean with a method for clustered calibration data, or an effective-sample-size estimate: Dunn, R.J., Wasserman, L., Ramdas, A., "Distribution-free prediction sets for two-layer hierarchical models", J. Am. Stat. Assoc. 118(544), 2491-2502 (2023); Oliveira, R.I., Orenstein, P., Ramos, T., Romano, J.V., "Split conformal prediction and non-exchangeable data", J. Mach. Learn. Res. 25(225), 1-38 (2024).
-
At a target limit, only the control transfer fails (Sects. 1.1, 4.2, 5.1). The one not applicable outcome at 10^-2 on all five splits is the transfer of the u_s threshold from nuScenes to KITTI, which the authors designate the control and state "does not isolate a TC". Rain, the motivating triggering condition, gives no not applicable outcome at any target limit; transfers into far distance and into rain fail consistently only at 0.1. The remaining target-limit evidence is reported "on some splits" (one of five for near to far on KITTI, three of five for far to near on nuScenes), but the five splits re-divide the same frames, so a split fraction carries no error probability, as Sect. 4.3 itself says. The abstract and the answer to RQ1 thus let a reader take the control transfer and the split fractions as evidence about triggering conditions; what the data support is that no target limit was shown to be violated under one.
-
The AUROC claim rests on one thin case (Sect. 5.2, Table 4). Contribution 3 asserts that AUROC does not reveal the changes found in rain and between the datasets. For the dataset change the manuscript itself writes that the two AUROC values "describe two detectors on two datasets, so their difference is not a change of separation for one detector" - so that comparison cannot demonstrate a blind spot of AUROC, yet Table 4 records "shown by AUROC: none" for that row as though it could. For rain, AUROC moves from 0.844 to 0.856 with a 95% resampling interval of -0.030 to 0.051 over 24 scenes, so "the estimated separation did not weaken" overstates an interval straddling zero. The analytic argument of Sect. 3.4 does not need these cases: rest the claim on it, present Table 4 as illustration, and mark the dataset row not applicable in the "shown by AUROC" column.
Minor Comments
- The limits 10^-2, 10^-3 and 10^-4 are not derived from anything, yet RQ3's answer is arithmetic in alpha by Eq. (5); an illustrative vehicle-level derivation would show which part of that range matters.
- The per-frame FAP is chosen because "LTT bounds a mean of per-unit values in [0, 1]" (Sect. 3.1); a per-kilometre rate alongside it would let readers map alpha to a hazard rate.
- Sections 5.1 and 5.2 write "u_c" where "u_e" is meant, in both references to the far-to-near transfer on nuScenes.
- Table 2 uses A_c both for the FAP in c and for the FAP over all frames with a car detection (written A in Eq. 4), and the b_k, s_k row carries a stray "spatial and existence indicator".
- The near/far boundary is the median of the detector's own box distances (34.8 m, 21.4 m), so a check at a fixed physical boundary would show the outcomes are not an artefact of it.
- The nuScenes members were trained for about 10 of the configured 20 epochs and match only 0.398 of annotated cars after the vote (Table 3), so the rain pair describes an undertrained detector.
- Only the two indicators of the conference paper are evaluated; a baseline such as the merged box confidence would show whether the failures are specific to ensemble indicators.
- No computational cost is reported, although running M = 4 PointPillars members per frame constrains a run-time rejection rule.
- Section 3.3 requires delta to be divided among the outcomes a safety argument relies on, but the paper draws conclusions from several of its 240 outcomes at delta = 0.05 without that correction.
- Equation (5) covers only the zero-failure case, and the increase from "any unit with a false acceptance" is unquantified, although a recording plan needs it.