How Do Small Coding Agents Use Code Graphs and Embeddings?
Saurav Bellam
Abstract
Coding agents are given embedding search and code-graph tools on the premise that a repository's structure is something the model lacks and the tool supplies. We test that premise on SWE-bench Verified with small open-weight models, stratifying tasks by a deterministic hop depth (the shortest code-graph path from code the issue mentions to code the gold patch changes) that a companion paper defines and validates. First, we ablate a localization agent's tool set (grep only; +embedding search; +graph tools; all) on a depth-balanced 160-task split and on all 500 tasks for ten models (six Qwen3 sizes from 0.6B to 32B with five decodings each, an FP8 control, three Gemma 4 sizes; 79k episodes). Structured tools double grep-only accuracy for Qwen3-8B and Gemma 4 E4B; their gain peaks at 17-21 points at 1.7-4B and falls to 4-5 points at 32B (odds ratio 0.77-0.82 per doubling of parameters). From 8B up, about half of that gain is the form of the submission rather than finding the target; the fall from 14B to 32B is a fall in tool uptake at equal grep-only accuracy; and the same 4B size gains or does not gain from a tool depending on the checkpoint. Each extra hop multiplies the odds of correct localization by 0.50-0.67 at every size, and no tool set flattens that gradient: traces show the graph tools' gain is name lookup at hop 0 and one expansion at hop 1, and agents chain expansions in only 12-14% of episodes. A fixed repair model resolves 6-13 of 160 tasks from these localizations (oracle 26). Second, linear probes find direct call edges decodable under a within-anchor, degree-stratified protocol (AUC 0.74; 0.61-0.66 inside agent conversations), but below a lexical baseline (0.99), with two-hop edges at 0.57. Small agents carry a one-hop, largely lexical picture of their repository, in weights and in use. We release the agent, probes, logs and traces.
Review reports
Reviewer3 (AI review)
AI Review (reviewer3.com)
Comments (10)
1. GLOBALPROBE INHERIT AUC for pair probe: 0.77/0.81 vs. 0.77/0.80 — Editorial
In Section 6.2 (page 19), the text first states that on the GLOBALPROBE task, "INHERIT [reaches] 0.77/0.80 at later layers" (matching Table 16's best column for Qwen3-4B: .774 on Django and .805 on others). Two sentences later, when comparing the h_a-only control against the pair probe on INHERIT, the text writes "(0.79/0.80 vs 0.77/0.81)", inconsistently changing the pair probe's score from 0.77/0.80 to 0.77/0.81 (and quoting 0.79/0.80 for a-only, whereas Table 16 reports .743/.772 at layer 24).
2. Section 5.3 tool-use percentages come from Table 8 (subset), not 500 tasks — Editorial
Qwen3-32B calls find_entity in 39% of its T episodes and 44% of its ET episodes, graph_neighbors in 31% and 15%, and semantic_search in 34% of E and 8% of ET episodes, against 94%, 86% and 85% for Qwen3-8B and 98%, 87% and 40% for Qwen3-4B on the same 500 tasks (Table 8). It
In Section 5.3 (page 13), the prose states that Qwen3-32B's tool-use rates are compared "against 94%, 86% and 85% for Qwen3-8B and 98%, 87% and 40% for Qwen3-4B on the same 500 tasks (Table 8)." However, Table 8 (page 16) is explicitly captioned as reporting "pooled over three runs per model (156 tasks with function gold per run)" on the 160-task subset, not the 500 tasks. Furthermore, in Table 8, the third percentage for Qwen3-4B (ss in arm E) is .43 (43%), whereas the prose cites 40%, and for gn in arm T, Qwen3-4B is .90 (90%) while .87 (87%) is the value for arm ET.
3. Hybrid Qwen3-4B grep accuracy: 9.1% in text vs. 10.1% in Table 7 — Editorial
On the same 500 tasks with the same harness, Qwen3-4B-Instruct-2507 reaches 26.8% with grep alone and gains nothing from embedding search (Table 6); the hybrid Qwen3-4B reaches 9.1% and gains +17 . 6 points from it. The Instruct
In Section 5.4 (page 16), the prose states that on all 500 tasks, "the hybrid Qwen3-4B reaches 9.1% and gains +17.6 points from it [embedding search]." However, in Table 7 (page 14) for Qwen3-4B (hybrid), the G arm strict function Acc@10 is reported as 10.1% and the E arm is 27.7% (which gives the +17.6 difference: 27.7 - 10.1 = 17.6). A 9.1% G accuracy with a +17.6 gain would imply an E accuracy of 26.7%, conflicting with Table 7.
4. Total repositories in SWE-bench Verified: 12 vs. 11 non-Django — Editorial
both have 36 layers; layer 0 is the token embedding). Folds are repository-held-out: train on the eleven non-Django repositories and test on Django, and the reverse.
In Section 6.1 (page 19), the text describes the repository-held-out folds for the static probes (which run on all 468 bucketed tasks across the benchmark) as: "train on the eleven non-Django repositories and test on Django, and the reverse." However, Table 11 (page 27) shows that SWE-bench Verified spans 12 repositories in total (11 non-Django repositories plus Django). It is the 160-task evaluation subset (Section 4.1, page 8) that spans 11 repositories in total (Django plus 10 non-Django repositories).
5. Mismatch between localization metric and repair prompt filtering — Minor
Localization is only useful if a repair stage can use it. For each task and arm of A BL -8B we take the top-5 submitted strings that are exact function or class ids of the snapshot, show their source (up to 150 lines each) together with the issue to Qwen3-8B (greedy, non-thinking) with a SEARCH/REPLACE edit-block prompt (Appendix F), apply each block only if its SEARCH text occurs exactly once in the file, convert the result to a unified diff, and score it with the official SWE-bench harness 4.1.0
While localization is evaluated using top-10 predictions with forgiving prefix matching (and canonical rescoring in Section 5.4), the REPAIR stage strictly filters for the top-5 submitted strings that match exact node IDs without applying the canonical repair rules. This design choice artificially penalizes the G arm—dropping 109 of 156 episodes before the repair model is even called—and conflates localization discovery with uncanonicalized string formatting in downstream repair.
6. Linear log-scale model misspecifies inverted-U tool gain curve — Major
The tools’ gain over grep rises from 0.6B to 1.7–4B and then falls monotonically to 32B. ... A GEE model over the Qwen seeds, y ~ arm × log2(params) clustered by task, puts the trend at an odds ratio of 0.77 [0.75, 0.79] per doubling for E, 0.82 [0.80, 0.84] for T and 0.80 [0.78, 0.82] for ET (all p < 10^-40)
Fitting a single linear interaction term arm x log2(params) across all six model sizes (0.6B to 32B) misspecifies the functional form of the relationship, because the text and Figure 3b/c explicitly show an inverted-U curve where gains rise from 0.6B to 1.7B–4B before falling. Additionally, the monotonic decline from 14B to 32B is heavily driven by a single anomalous 32B checkpoint that underperforms the 14B model overall, making a single per-doubling odds ratio across the entire range potentially misleading.
7. Uncontrolled instruction prefix and dimensionality in WEAKEMB ablation — Minor
WEAKEMB replaces the embedder in arm E with a weak static model (model2vec potion-base-8M (Tulkens & van Dongen, 2024), mean of static token vectors, 256-d, no instruction prefix) for Qwen3-8B.
When swapping Qwen3-Embedding-0.6B for potion-base-8M in the WEAKEMB experiment, the retrieval instruction prefix is also removed and the embedding dimensionality drops from the base model's dimension to 256-d, slightly confounding the effect of static versus dense neural architecture with prompt formatting and vector capacity.
8. Underpowered comparison for claiming half of gain is retriever quality — Minor
Replacing Qwen3-Embedding-0.6B with the static potion-base-8M embedder lowers Qwen3-8B’s E arm from 26.3 to 21.2% function Acc@10 (paired: 12 tasks lost, 4 gained, p = 0 . 077 ) and file Acc@5 from 51.2 to 47.5% ( p = 0
The claim that half of the embedding arm's gain comes from retriever quality rests on a drop from 26.3% to 21.2% in WEAKEMB that does not reach conventional statistical significance (p = 0.077 on 16 discordant tasks) and causes no significant change in file Acc@5 (p = 0.39). The 50/50 attribution should be framed more cautiously given the small sample of discordant episodes.
9. Single-file context truncation omits cross-file intermediates in CALL2 — Major
For each of the 468 bucketed tasks we take the first gold-patched file at the base commit (one context per distinct file; 468 contexts, mean 4,668 tokens, truncated at 6,000), with a mean of 34.2 definitions each. Labels come from the whole-repository graph of Section 3.1, not just the file.
In the static probing setup (GLOBALPROBE/ANCHORPROBE), activations are extracted from a single file truncated at 6,000 tokens, while ground-truth CALL2 labels are derived from the whole-repository graph. If the intermediate function c in a two-hop call chain a -> c -> b resides in another file or beyond the 6,000-token cutoff, the model cannot compute the transitive link a -> b from the input context via self-attention, which could explain the near-chance CALL2 probe performance independently of the model's two-hop reasoning capacity.
10. Severe output truncation confounds multi-hop graph tool evaluation — Minor
episodes but 25% of hop-2 and hop-3+ episodes. A two-hop neighbourhood of a mentioned node contains hop-2 targets by construction, and the agents do ask for two hops ( hops=2 in 57% of Qwen3-4B’s and 95% of Qwen3-8B’s graph_neighbors calls), but the output is cut by the 4,000-character observation budget in 64% and 72% of calls respectively (a median of 23–24 edge lines is shown, so the 50-line cap never binds), and the agents almost never
In 64% to 72% of graph_neighbors calls, the 4,000-character observation budget truncates the output to a median of 23–24 edges. Because the tool expands via breadth-first search, hop-1 edges are printed first and hop-2 edges are disproportionately cut off before the agent ever sees them, making it difficult to disentangle whether the lack of hop-2+ improvement is due to agent reasoning limitations or the unranked BFS truncation design.
xPeerd (AI review)
AI Review (xPeerd — DAReview)
Summary
This manuscript studies whether code-graph tools actually help language-model agents localize bug-fix targets in SWE-bench Verified, and whether the benefit persists as graph distance (“hop depth”) increases. The paper’s core findings are coherent, clearly stated, and largely well supported by the presented experiments. It introduces a deterministic hop-depth labeling of the benchmark, shows that each additional hop substantially reduces localization success, and argues that structured tools mainly help with exact-name lookup and one-hop expansion rather than deeper multi-hop reasoning . The work is strengthened by multiple experimental strands: tool ablations, depth-stratified analysis, scale curves across model sizes, downstream repair tests, and representation probes .
The manuscript’s main strengths are empirical breadth, unusually explicit limitations, and strong reproducibility claims, including release of labels, code, raw results, and analysis scripts . Its main weaknesses are narrower external validity than some claims may imply, partial confounding between the hop label and graph-based tools, some overextension from probing evidence to behavioral interpretation, and a few places where the comparative framing versus prior work could be stated more cautiously . I do not see obvious paper-mill features or internal signs of fabrication; the manuscript is technically coherent and unusually transparent about failure cases and missing traces .
Potential Major Revisions
- Clarify and narrow the central claim about “structure” versus “tool design” The manuscript concludes that “The multi-hop structure that the deepest third of the benchmark requires is neither in these models’ weights nor, in practice, extracted from the tools we gave them” . This is a strong and interesting claim, but the manuscript itself presents evidence that the tool interface may be a major limiting factor. In the discussion, the authors write that “our T arm hands a 4–8B model raw edge lines truncated at 50 and leaves the chaining to it” and that “What is left is the model’s ability to use a neighbourhood listing to decide what to expand next” . That wording suggests the negative result may be partly a failure of the specific tool presentation rather than a general limit of graph tools or graph-augmented agents.
The revision needed here is conceptual precision. The title, abstract, and conclusion should distinguish more explicitly among:
- limitations of the tested models,
- limitations of the tested tool interfaces,
- limitations of this localization-only task formulation,
- limitations of static graph construction in Python.
At present, some statements read broader than the evidence fully supports. For example, “The graph tools’ benefit is exact-name lookup at hop 0 and one expansion at hop 1—they do not flatten the depth gradient” is defensible for the tested setup, but should be framed as a result for these implementations and prompt/tool protocols, not necessarily for graph-guided localization in general.
- Address the coupling between the evaluation label and the graph arm more centrally One of the manuscript’s most important limitations is currently acknowledged, but it should be elevated into the main validity discussion. The authors write: “The label and the T arm’s tools are built from the same graph, so the hop-0/hop-1 advantage of T is in part an advantage on the label’s own terms” . This is a substantive methodological issue, not a minor caveat. Because the same graph formalism helps define both the difficulty axis and the graph tool’s possible returns, the study may structurally favor graph-based performance at shallow depth while also constraining what counts as deeper structure.
The manuscript should revise the framing of causal claims accordingly. I recommend:
- moving this limitation from a late discussion paragraph into the main methods-validity discussion;
- quantifying, if possible, how sensitive the main depth-by-arm conclusions are under stricter label variants already mentioned in Section 3.5 ;
- explicitly separating claims that remain robust under alternate mention-resolution choices from those that may depend on the current graph definition.
Without this strengthening, the reader may overinterpret the depth analysis as more independent of the graph construction than it actually is.
- Temper inference from linear probes to representational absence The manuscript is careful in places, but still occasionally edges toward stronger conclusions than the probe design warrants. The discussion states that “Two-hop structure is essentially absent from the activations” and that “The multi-hop structure that the deepest third of the benchmark requires is neither in these models’ weights” . Yet the paper also acknowledges that “A linear probe is also a lower bound: a nonlinear readout, an attention-pattern analysis, or a causal intervention could find two-hop structure a linear probe cannot” .
That caveat is important enough to be integrated into the headline interpretation, not left as a qualification. The probe results strongly support a narrower claim: with the chosen representation and readout, direct-call information is linearly decodable after controlling confounds, whereas two-hop and inheritance relations are not robustly decodable beyond simple baselines . That is already a strong result. The manuscript should avoid letting “not linearly decodable here” slide into “not represented” without consistent qualification.
- Strengthen the treatment of partial and degenerate runs in the scale analysis The appendix notes that “Gemma 4 12B” is partial and degenerate, with looping behavior, high step-cap hits, and uncertainty as to whether this reflects “the checkpoint, of the gemma4 tool-call parser in vLLM 0.32.0, or of the session” . This is appropriately transparent, but the scale-curve narrative should more clearly isolate such runs from interpretive trends. Likewise, “Missing traces” are said not to affect metrics because results files remained complete ; that is plausible, but any trace-based mechanism claims should clearly state whether all relevant runs had complete trace availability.
I recommend revising the scale-curve section so the primary conclusions are explicitly derived only from stable runs, with unstable/partial runs visually and rhetorically separated. Otherwise, readers may infer a cleaner monotonic scaling story than the raw experimental conditions justify.
- Clarify the contrast with LocAgent to avoid overclaiming disagreement The manuscript states: “LocAgent ... reports that graph-guided agents degrade least with depth. Our graph arm degrades at least as steeply as grep alone” . This contrast is important, but the current wording risks implying a direct contradiction under matched conditions, which is not the case. The paper itself notes plausible differences in “model scale and agent design,” including larger models and “compressed subgraphs with relation-aware summaries” in LocAgent versus raw edge lines here .
This section should be revised so the disagreement is framed as a boundary condition comparison rather than a refutation. The current evidence supports: under this tool presentation and these models, graph tools do not flatten the depth slope. That is a valuable result. It need not be packaged as stronger-than-warranted general disagreement with prior graph-agent findings.
- Improve external validity discussion around benchmark, language, and task framing The manuscript does acknowledge that it uses “one benchmark in one language” and that “Python’s dynamic dispatch is invisible to our graph” . However, the implications of this are broader than currently emphasized. Since the task is localization-only, the results may differ materially for full repair pipelines, for statically typed languages where call graphs differ in fidelity, or for repositories where issue descriptions encode different levels of entity naming. The authors themselves write that “The localization-only formulation removes the interaction between exploration and editing that a full agent has” and that the repair model is weak .
The manuscript should convert these observations into a sharper boundary statement: the paper isolates an important bottleneck in repository navigation under a specific benchmark and graph formalization, but it does not yet establish that multi-hop structure is similarly inaccessible in broader software-engineering agent settings.
Potential Minor Revisions
-
Terminology and readability The paper is technically dense and frequently compresses multiple claims into long sentences. For example, the conclusion sentence beginning “We ablated the tool set of a localization agent and found that structured tools roughly double grep-only accuracy...” continues through many clauses and parenthetical qualifications . This is precise, but difficult to parse. Similar density appears in the contributions paragraph listing four contributions in one block . Shortening several high-information sentences would improve readability without reducing rigor.
-
Explain shorthand earlier and more consistently Abbreviations such as G, E, T, ET, LABELVAR, DEPTH GEE, SCALE CURVE, FORCE TOOL, and REPAIR are central to the argument but may impose high cognitive load, especially outside the methods section. Table-driven shorthand is efficient, but in narrative sections these labels should be glossed more often for readability .
-
Distinguish “discovery” from “submission form” earlier A notable and interesting result is that from 8B upward, a substantial part of the measured tool gain is “the form of the submission” rather than genuine discovery improvement . This distinction is important enough that it should be introduced earlier in the results narrative rather than mainly surfacing in later discussion/conclusion passages.
-
Table and appendix navigation The manuscript references many tables across the appendix. While the cross-referencing is technically adequate, the reading burden is high. Consider adding a brief roadmap sentence at the start of major results sections indicating which tables establish the main effect, robustness, mechanism analysis, and scale curve. This would improve navigability, especially around Tables 1, 3, 12, 13, 18, and 19 .
-
Typographic/style issues I did not detect many obvious grammatical errors from the retrieved text, which is a strength. The main style issue is visual and typographic inconsistency caused by small-caps/spaced labels in the extracted text, such as “S CALE C URVE,” “R EPAIR,” and “F ORCE T OOL” . This may partly be PDF extraction noise rather than source formatting, so I would not treat it as a substantive manuscript defect. Still, the final camera-ready version should ensure acronym formatting remains visually clean and consistent.
-
AI content analysis Estimated percentage of AI-generated content: low, approximately 5–15%.
Rationale: the manuscript shows features more consistent with expert-authored technical writing than generic AI generation. It contains nuanced caveats, self-limiting claims, precise methodological distinctions, and transparent reporting of negative or ambiguous results, such as “we report it and draw no conclusion from it” about the Gemma 12B partial run . It also includes field-specific methodological self-critique that is unusually disciplined, e.g., “A linear probe is also a lower bound” and the warning that a less careful study could have overclaimed from global pair-probe AUC .
Possible AI-like signals are limited to occasional overcompressed prose and repetitive rhetorical templates in contribution/limitation summaries. These do not amount to strong evidence of machine-generated text. I do not identify any clearly “obvious AI-detected sections” from the manuscript excerpts reviewed. The epistemic impact of any limited AI assistance, if present, appears negligible relative to the manuscript’s substantive technical contribution.
Recommendations
-
Reframe the headline contribution more precisely: Present the main result as evidence that, in this benchmark and tooling setup, graph-structured assistance improves shallow localization more than deep multi-hop reasoning. This is both accurate and impactful.
-
Make the validity architecture more explicit: Bring the graph-label coupling, localization-only setup, static-graph assumptions, and benchmark/language limits into a concise “threats to interpretation” paragraph in the main paper, not only later discussion .
-
Tighten the representational claims: Replace any wording implying definitive absence of multi-hop structure in model weights with wording tied to the actual evidence: not robustly linearly decodable under the chosen controls and not behaviorally exploited by the tested agents .
-
Sharpen the comparison to prior work: Frame the LocAgent contrast as a meaningful divergence under different modeling and tool-presentation conditions, not a blanket contradiction .
-
Separate stable findings from suggestive findings: In the scale-curve and cross-model discussion, isolate partial or degenerate runs and make sure headline claims rely only on the stable subset .
-
Improve exposition: Reduce sentence density in the conclusion and discussion, and add brief reminders of the meaning of G/E/T/ET in prose near the main result transitions.
Overall, this is a strong and intellectually serious empirical paper with valuable negative findings, good reproducibility practice, and credible methodology. My main recommendation is not to change the experimental core, but to narrow several top-level claims so they match the strongest defensible interpretation of the evidence.
XSci (AI review)
Summary
Coding agents are given embedding search and code-graph tools on the premise that repository structure is something the model lacks and the tool supplies. The authors test that premise on SWE-bench Verified with small open-weight models. They label all 500 tasks with a deterministic hop depth from issue-mentioned code to gold-patched code (186/128/78/76 at hop 0/1/2/3+) and show that each extra hop multiplies the odds of correct function-level localization by 0.47–0.55 after controls. They then ablate the agent's tool set on a depth-balanced 160-task split and on all 500 tasks for ten models (79,228 episodes): structured tools roughly double grep-only accuracy at 4–8B, the gain peaks at 1.7–4B and falls to 4–5 points at 32B, and the graph arm's depth slope is never flatter than grep's. Linear probes then find direct call edges decodable (AUC 0.74) but below a lexical baseline (0.99), with two-hop edges at 0.57.
Contribution and Significance
The core contribution is a controlled, depth-stratified measurement of what structured tools buy a small coding agent, backed by a released depth labelling of SWE-bench Verified. Three results matter. The tool advantage falls by about a fifth of its odds per parameter doubling (0.77–0.82, Section 5.4), and the fall from 14B to 32B happens at identical grep-only accuracy (32.1 versus 32.1%) because the larger checkpoint stops calling the tools (find_entity in 96% of T episodes at 14B, 45% at 32B). From 8B up, about half the measured gain is the form of the submission: canonical rescoring moves the G arm from 32.1 to 42.1% at 14B while ET moves 2–3 points. And which tools help is a property of the checkpoint, not the size — Qwen3-4B-Instruct-2507 gains nothing from embedding search while the hybrid Qwen3-4B gains +17.6 points. Papers reporting localization gains from structure should now have to rule out the form effect.
Relative to LocAgent (Chen et al., 2025a), which introduced the hop-distance analysis and reported that graph agents degrade least with depth, what is new is a deterministic released label, a one-tool-at-a-time ablation on identical tasks and harness, and the reversal at depth for 4–8B models. The probing contribution is narrower than presented: its one positive result is close to definitional (Concern 2), so Part 3's content is essentially negative.
Strengths
- The depth label is interrogated rather than asserted: call resolution barely moves the bucket (κ = 0.961–0.981), while the strict mention extractor leaves 311 of 500 tasks without a mention — a caveat released with the data.
- The scale curve (Tables 7, 19) is unusually complete: four arms × six Qwen3 sizes × five decodings on all 500 tasks, with an FP8 control at 14B that rules out precision as the cause of the 32B result.
- Every tool line carries the nodes it exhibits, so discovery and conversion are separable (Table 8) and first-reveal is attributable per tool and bucket (Table 15, Fig. 4).
- Section 6.2 reports the global pair probe (AUC 0.785) that one-sided probes (0.709) and in-file in-degree (0.847) defeat, and builds the protocol in response.
- Numbers reproduce: I recomputed the Fig. 2 prose from Table 13 and the abstract's ranges against Tables 1, 7, 9 and 18.
Recommendation
Major Revision for ICLR. The contribution is of a kind ICLR publishes and the evidence is large and honestly reported, but three central claims carry confounds the authors' own data can close: the hop-2+ negative is not separated from the output truncation that cut two thirds of the two-hop queries; the positive probe result is near-definitional given how INVOKE edges are built; and the depth axis is not controlled for how much code the issue names.
Major Concerns
-
The hop-2+ negative is confounded with the observation budget (Sections 4.3, 5.5, 5.8, 7). Section 5.5 reports that the agents do ask for two hops — hops=2 in 57% of Qwen3-4B's and 95% of Qwen3-8B's graph_neighbors calls — but that "the output is cut by the 4,000-character observation budget in 64% and 72% of calls respectively (a median of 23–24 edge lines is shown, so the 50-line cap never binds)". A two-hop neighbourhood contains hop-2 targets by construction, as the authors say; if only about 23 breadth-first edge lines survive, the hop-2 edges are precisely the ones removed. That graph_neighbors is the first revealing tool in only 17% of hop-2 episodes is therefore consistent with the tool never having shown the target, independent of any reasoning failure. This bears on the central synthesis: Section 7 concludes that "What is left is the model's ability to use a neighbourhood listing to decide what to expand next", offering FORCETOOL and BUDGET as having excluded the alternatives. They do not exclude this one — BUDGET varies the step cap (30→50) and context (32k→40,960), not the truncation, and no run in Section 4.5 varies it. A run of arm T at a much larger observation budget, or with the two-hop output compacted so hop-2 edges are reachable — the tool design recommended in Section 7(ii) but never tested — would resolve it.
-
The positive probe result is near-tautological given the graph construction (Sections 3.1, 6.3). An INVOKE edge is created "for every ast.Call whose callee is a bare name or an attribute", resolved by name, so a→b exists essentially only when b's name occurs in a's body; the lexical baseline confirms it at 0.993 and 0.987 on the same groups. The strict grouping stratifies on the anchor and the in-degree bin of b but not on the lexical cue, so the probe's task reduces to "which identifiers appeared in a's body" — a fact about tokens, not about the repository graph. The manuscript concedes this in passing ("a degraded copy of information that is literally in the text of a"), yet the abstract, contribution (iii) and conclusion still present direct-call decodability as a finding and the protocol as one "that a node-level shortcut cannot pass". It does not pass its own standard here; where the cue is removed by design (CALL2, 0.569/0.573) or present but non-definitional (INHERIT, pair 0.619/0.629 below b-only 0.688/0.674), nothing is decodable. The per-pair scores are saved, so the fix is cheap: stratify on the lexical cue too, and report AUC inside the name-present stratum.
-
Depth is not separated from how much code the issue names (Sections 3.3, 3.4). Mentions fall across buckets — 8.56, 6.48, 7.15, 3.67 at hop 0–3+ — so deeper tasks are also tasks whose issues name less code. Eq. (3) controls for arm, repository, number of gold functions and log patch size, but not mention count or issue length, the most direct alternative explanation of the gradient. The claim that "depth is not a proxy for repository, patch size or number of targets" is accurate but does not address it, and depth is then read as structural distance throughout, including in Section 7's benchmarking advice. The strict_mentions variant points the same way: dropping most mentions moves the odds ratio toward 1 (0.817 [0.632, 1.056], p = 0.12 for Qwen3-32B). Adding mention count and issue length to Eq. (3), and reporting the per-hop odds ratio within mention-count strata, would settle it.
Minor Comments
- The label and the T arm's tools are built from the same graph, so the hop × arm interaction of Table 12 should be refit under each label variant, not only the default.
- In REPAIR, G produces 47 valid-id prompts against 89–113 for the structured arms, so part of the 6-versus-13 difference is id formatting; rerun with canonical rescoring applied first.
- The REPAIR arm ordering rests on one greedy run with discordant counts of 8/1, 7/1 and 9/2 (p = 0.039–0.070) and should be stated only as "the structured arms beat G".
- The 32B anchor is a single FP8, non-thinking checkpoint the authors report is weaker than 14B, and the FP8 control exists only at 14B; a second 30B-class checkpoint would help.
- Thinking is disabled for every Qwen3 hybrid although chaining expansions is a reasoning step, so one thinking-enabled 8B run of arm T is worth reporting.
- Table 2's Setting column and Table 13's Qwen3-8B BUDGET row appear offset by one, and "hop 34" (Section 5.1), "grey only" (Findings) and the CONVPROBE heading need fixing.
- The abstract says the call representation "remain[s] so inside agent conversations (0.61–0.66)", but on the Django fold of Table 10 the pair probe at 0.605 [0.513, 0.690] overlaps b-only at 0.511 [0.423, 0.594]; Section 6.4's "nearly do" is the accurate wording.
- Table 4's intervals bootstrap over tasks only despite between-run standard deviations up to 3.4 points (Table 5), and Holm correction covers only the 18 SCALECURVE tests, which matters for marginal results such as ET−G at 4B, +5.3 [+0.0, +10.5].
- A fifth "canonicalised grep" arm would turn the claim of Section 7(iv) into a result.
- Two directly relevant works are absent from the reference list: Siru Ouyang et al., "RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graph", ICLR 2025 (arXiv:2410.14684), and Yuntong Zhang, Haifeng Ruan, Zhiyu Fan and Abhik Roychoudhury, "AutoCodeRover: Autonomous Program Improvement", ISSTA 2024 (arXiv:2404.05427).