XSci

Straight to the Product: What Online Retailers Miss When Shoppers Ask AI First

Michael Iannelli, Alan Ai

Submitted by Michael Iannelli

Published November 19, 2026 · Version v1, November 19, 2026 · DOI 10.66977/xsci.2611.003f

Information Retrieval, Human-Computer Interaction, Computers and Society

Abstract

When people ask an AI assistant what to buy, what happens next at the store? We link the same people's assistant conversations to their later browsing at twenty large online retailers. People go to the product the assistant named: among shoppers who had not visited the store in the past week, visits to the named product within a day rise from 0.02 to 2.07 per 100 answers (+2.0 points [1.7, 2.4]), and on days with a product-page visit at the store, 44% show a product from their own answer, against 1.5% for a matched answer given to someone else. The result replicates in a separate five-month period. Few arrive through the assistant's link: 89% of visits carry no sign of it, and 54% follow a web search, the same share as their usual visits. When a search comes just before the first visit, it names the answer's product 28% of the time (1% in earlier weeks), as if the search box were an address bar. The chat names the product; the store logs a search. Once there, shoppers keep browsing. We propose product follow-through, the share of answers followed by a visit to the product they named, as a way to evaluate shopping assistants and agents by what people do next: it needs no referral data or provider logs and can be computed for each assistant.

View PDF

Review reports

xPeerd (AI review)

November 19, 2026

AI Review (xPeerd — DAReview)

Summary

This manuscript, titled Straight to the Product: What Online Retailers Miss When Shoppers Ask AI First, studies whether AI shopping-assistant answers influence subsequent retailer browsing behavior. The core claim is that users often visit the product named by the assistant, but retailers usually do not observe the assistant as the referral source because most visits arrive through search or direct navigation rather than explicit AI links. The paper is clearly framed, the main behavioral measure (“product follow-through”) is intuitive, and the replication period strengthens the descriptive consistency of the reported pattern. The abstract reports a rise in visits to the named product “from 0.02 to 2.07 per 100 answers” and states that “89% of visits carry no sign” of the assistant link . The manuscript’s main strengths are its practical relevance, product-level measurement idea, and repeated comparison designs across baseline weeks, swapped answers, and matched moments . Its main weaknesses are limits to causal identification, lack of reported sample sizes due to commercial sensitivity, partial dependence on LLM-based labeling, and underdeveloped discussion of external validity, platform heterogeneity, and statistical transparency . I find the work promising and potentially publishable after substantial revision, but in its present form several methodological and reporting choices prevent a fully confident assessment of reproducibility, robustness, and inferential strength.

Potential Major Revisions

  1. Causal language should be tightened and more consistently aligned with the observational design.

The manuscript repeatedly uses language implying directional effect, for example: “People go to the product the assistant named” in Section 3.1 and “the answer shapes which product people go to see” in Discussion . Yet the paper later acknowledges: “Limitations. This is an observational study” and states that the interpretation depends on an assumption about what would have happened without the answer . This tension is important. The current framing sometimes reads as if causal identification were established, whereas the evidence is better described as strong behavioral association under several comparison schemes. The manuscript should systematically revise causal verbs such as “shapes,” “move,” and “lead to” unless supported by a clearer identification strategy. The Discussion does note that “Intent formed just before asking could break it” , but this issue deserves earlier and more explicit treatment in the Introduction, Results, and Abstract.

  1. Reproducibility is materially limited by omission of sample sizes and panel counts.

Section 2 states: “We do not report panel size, per-analysis user counts or per-cell sample sizes; these are commercially sensitive” . This is a serious barrier to scientific evaluation. Without denominator counts, the reader cannot assess statistical power, subgroup stability, exposure concentration, or practical heterogeneity across retailers, assistants, and user segments. Confidence intervals are reported, but intervals alone are not a substitute for sample-size transparency. At minimum, the manuscript should provide masked or binned counts, ranges, effective sample sizes, counts of answers retained after each filtering stage, and store-level or assistant-level exposure distributions. Otherwise, the claimed robustness is difficult to verify independently.

  1. The construction and validation of the key labels needs fuller methodological disclosure.

The paper relies on model-based judges for answer type and product matching. It explains that “Both judges are classification tasks run on one underlying model” and that the product-match judge uses page titles and URLs, counting variants as matches . Validation is helpful but still incomplete. The manuscript reports that on a human audit, “the judge’s matches were confirmed in 93% [86, 97] of 90 after an answer and 54% [39, 68] of 39 in the earlier-week baseline” and that answer-type agreement was “75% [65, 82]” on a class-balanced sample . These numbers suggest nontrivial labeling uncertainty, especially in the rare-event baseline. Because the main effect compares 2.07% versus 0.02%, even modest systematic classification asymmetry could matter. The paper should provide confusion matrices, class prevalence, examples of false positives/false negatives, judge prompts or rubric equivalents, and sensitivity analyses under stricter match definitions.

  1. The selection of the “main group” appears post hoc and needs stronger justification.

The Limitations section candidly states: “We chose the main group after first results; its store-visit rise (+5.1 points) is smaller than over all answers in which the assistant named the store (+10.5)” . This is a nontrivial concern. Post-results subgroup definition raises the possibility of researcher degrees of freedom, even if the chosen estimate is more conservative. The manuscript should clearly enumerate all alternative cohorts considered, explain why the final analytic group was selected, and report full results for the broader and narrower populations in the main text or appendix. A specification-curve or multiverse-style robustness display would substantially improve credibility.

  1. The paper’s inferential framework would benefit from more explicit modeling details.

The manuscript reports many bootstrap intervals and within-person comparisons, but the statistical framework remains somewhat terse. For example, the paper says that “every main estimate uses the full eligible panel” and intervals come from “a bootstrap that resamples people” . However, the reader is not told enough about clustering structure, repeated observations per person, imbalance across assistants, dependence across stores, or whether estimates are weighted by answer count, person, or store. The sentence “the named-product rise is 2.3 points with each person counted once” is useful , but it appears only as a robustness note rather than part of a fully specified estimation framework. The revision should state the exact unit of analysis for each figure and estimate, the bootstrap design, and whether multiple-comparison concerns were considered given the many subgroup and route analyses.

  1. External validity and scope conditions should be discussed more rigorously.

The manuscript covers “two English-speaking markets,” “twenty large retailers,” and February–October 2026 . This is useful but narrow. User behavior may differ sharply for smaller retailers, mobile-app shopping, closed marketplaces, non-English markets, or product categories with different consideration cycles. Similarly, the paper combines multiple assistants but reports only a limited assistant-level comparison: “follow-through rises 2.2 points [1.8, 2.7] after ChatGPT answers and 1.0 [0.5, 1.6] after Gemini answers” . The revision should discuss whether observed effects reflect assistant market share, answer style, user demographics, category mix, or retailer search optimization. More decomposition is needed before readers can generalize the findings.

  1. The integrity and coherence of the scientific paradigm are broadly acceptable, but some paper-mill or fabrication risk markers should still be addressed through transparency.

I do not see strong paper-mill traits such as incoherent jargon, fake citations, or conceptually empty claims. The manuscript is internally coherent, and its examples, figures, and limitations sections align reasonably well. However, one potential trust issue is that several important evidentiary details are withheld as “commercially sensitive” while the paper makes precise quantitative claims . That is not evidence of fraud, but it does increase verification risk. Transparency-enhancing steps such as de-identified aggregate appendices, detailed codebook disclosure, and a fuller audit trail for label validation would reduce that risk considerably.

  1. The practical interpretation of “product follow-through” needs sharper boundaries.

The paper proposes “product follow-through” as “the share of answers naming a specific product that are followed, within 24 hours, by a visit to that product’s page at the named store” . This measure is useful, but its meaning may be overstated if interpreted as recommendation quality or persuasion. A visit could reflect curiosity, comparison shopping, prior intent, or dissatisfaction. The Discussion says it is “a behavioral measure of a recommendation” , which is fair, but readers may infer more than the metric can support. The revision should distinguish between attention capture, product recall, navigation assistance, and genuine recommendation success, and explain what the metric does not measure, especially since “We see visits, not purchases” .

Potential Minor Revisions

  1. Typographic, wording, and style issues

The manuscript is generally polished, but several wording choices could be improved for precision or style consistency.

  • In the abstract, the phrase “as if the search box were an address bar” is vivid but somewhat journalistic for a research abstract . Consider replacing it with a more formal formulation.
  • In Section 3.2, “Our shopper did not click the assistant’s link” and similar recurring narrative sentences rely on the running-watch vignette . The device helps readability, but it is overused and occasionally blurs the shift between illustrative narrative and empirical result.
  • In Limitations, “Intent formed just before asking could break it” is too compressed and colloquial for a formal limitations statement . Specify what “it” refers to and rewrite more explicitly.
  • In Section 3.1, “Recommending goes with far more store visits than mentioning” is understandable but stylistically informal . A more formal wording such as “recommendatory answers are associated with substantially more store visits than mere mentions” would read better.
  • In several places, “rise” and “difference” are mixed across absolute-point and ratio interpretations, which may confuse readers. For example, “rise 2.8 times,” “about 24 times,” and “+2.0 points” appear in close proximity . Standardize whether effects are described as percentage-point changes, multiplicative factors, or both.
  1. Formatting and presentation issues
  • Page numbering is not visible in the extracted text, which makes precise review anchoring harder. The final version should ensure page numbers are clearly printed throughout.
  • Figure-heavy result presentation is clear, but the main text would benefit from at least one compact numerical summary table consolidating the principal outcomes now distributed across Figures 1–3 and prose .
  • The product-follow-through definition box is effective, but it should include a clearer note on how ambiguous product titles, bundles, or SKU variants are handled .
  • References appear adequate, but several cited works are arXiv preprints from 2026. The paper should distinguish more clearly between peer-reviewed and non-peer-reviewed prior work where relevant .
  • The ethics statement is present and useful, but it would benefit from clearer disclosure of institutional or provider governance, if any, beyond “terms” and aggregate-only reporting .
  1. Minor methodological clarifications
  • Explain more explicitly why 14, 21, and 28 days earlier were chosen as the baseline comparison windows rather than, for example, 7-day lag or rolling matched windows .
  • Clarify whether the “first answer that names the store” may omit later stronger recommendations in the same conversation and how that affects measurement .
  • Clarify whether multiple store mentions in one answer are excluded or how they are handled.
  • Clarify whether searches on external engines and internal site search are always distinguishable in the logs.
  • Clarify whether the “same kind of request” used for matched-answer pairing is operationalized by human coding, model coding, taxonomy rules, or embedding similarity .
  1. AI content analysis for this post-2021 manuscript

Estimated percentage of AI-generated content: low to moderate in the writing itself, likely around 10–25%, but likely higher in workflow assistance for coding/editing because the paper explicitly states: “AI coding assistants helped write the analysis and figure code and edit the text” .

Highlighted AI-detected sections:

  • The prose is unusually uniform, compressed, and rhythmically repetitive in a way consistent with heavily AI-assisted copyediting, especially in the repeated vignette transitions such as “Our shopper did not click the assistant’s link” and “Once on the watch’s page, our shopper looked at another watch” .
  • The bullet-pointed practical takeaways in “What a retailer can do” have a polished managerial tone that could reflect assisted drafting, though they are also plausible as human-authored applied writing .
  • The manuscript’s explicit use of model-based judges for classification is methodological AI use, not text-generation evidence by itself .

Assessed epistemic impact:

  • The disclosed AI assistance for code/text editing is not inherently problematic because the authors explicitly assume responsibility for the claims .
  • The more important epistemic issue is not prose generation but analytic dependence on LLM-based judges. That directly affects measurement validity and therefore the credibility of the paper’s contribution.
  • I do not see obvious hallmarks of fully machine-generated scholarly fabrication. The manuscript has coherent argument structure, nontrivial design choices, meaningful limitations, and targeted domain contribution. The main risk is methodological opacity, not synthetic authorship.

Recommendations

  1. Reframe the paper more conservatively as strong observational evidence of answer-associated browsing behavior, unless stronger causal identification is added.

  2. Add a transparent appendix with:

  • flow of sample construction,
  • answer counts and user counts in masked/ranged form,
  • store/assistant/category distributions,
  • full label-validation details,
  • robustness checks under alternative match and cohort definitions.
  1. Report an expanded robustness suite:
  • alternate baselines,
  • alternate follow-through windows,
  • stricter and looser product-match rules,
  • exclusion of ambiguous product pages,
  • assistant-specific and retailer-specific heterogeneity,
  • person-level weighting versus answer-level weighting.
  1. Make the post hoc cohort-selection issue fully explicit and provide specification-curve style evidence showing the principal result is not an artifact of analytic choice.

  2. Separate three claims more clearly:

  • assistants influence product attention,
  • retailers under-measure assistant influence in referral analytics,
  • shoppers continue broader browsing after arrival. These claims are related but not identical and should not rely on the same inferential rhetoric.
  1. Improve reproducibility by disclosing as much as possible without violating commercial constraints. If raw counts cannot be shown, provide coarsened ranges, normalized counts, synthetic examples, or third-party audit language.

  2. Expand the limitations section to discuss generalizability, unobserved intent, panel-selection bias, and the consequences of measuring visits rather than purchases.

  3. Consider adding one pre-registered or prospectively specified follow-up study, especially since the manuscript itself states that “The causal test is to randomize which of several suitable products an assistant names and measure the visits that follow” . That would convert a promising observational contribution into a much stronger empirical program.

Overall, I judge the manuscript as coherent, relevant, and potentially important, but not yet methodologically transparent enough for unreserved acceptance in its current form. Major revision is warranted.

XSci (AI review)

November 19, 2026

Summary

The paper asks what shoppers do at online stores after an AI assistant recommends a product. Using an opt-in panel in two English-speaking markets recording both browsing and conversations with five assistants, the authors link each person's answers to their later browsing at twenty large retailers, comparing each shopper to their own visits 14, 21 and 28 days earlier, to matched answers given to other people, and to matched no-answer moments. Visits to the named product rise from 0.02 to 2.07 per 100 answers (+2.0 points [1.7, 2.4]), and on days with a product-page visit at that store 44% show a product from the shopper's own answer against 1.5% for someone else's matched answer. Meanwhile 89% of post-answer visits show no sign of the assistant's link, 54% follow a web search that names the answer's product 28% of the time, and these visits view two or more products slightly more often than the shopper's other visits (31.4% against 26.2%). All headline results reproduce in a held-out five-month period, and the authors propose "product follow-through" as an evaluation measure for shopping assistants.

Contribution and Significance

The core contribution is measurement: this appears to be the first study linking the same individuals' assistant conversations to their cross-site browsing at the product level, and showing that the influence is largely invisible in the store's own logs. The own-answer versus swapped-answer contrast is the right design here, and the 42.5-point gap it produces is large and stable. The second finding — the chat names the product, the store records a search — matters most to the field: it identifies a structural blind spot in how e-commerce traffic is attributed and quantifies it (11 of 100 visits carry the link; 81% of the extra visits still do not), complementing Kaiser and Schulze [9], who see only referral-carrying sessions.

Relative to the closest prior work the advance is real but narrower than the framing suggests. Iannelli and Ai [6] already showed on panel data that recommended brands get more searches and site visits; the increment here is resolution and the within-store journey, not the existence of the effect. The second claimed contribution, the metric, is weaker than presented: a definition plus one application, and the claim that it "needs no referral data or provider logs" understates that it needs a panel linking conversations to cross-site browsing — an asset rarer, and more tightly held, than referral data.

Strengths

  • The held-out replication is genuine: the February–June analysis plan was committed before those data were pulled (§2) and the same code reused, and the estimates reproduce closely (+42.5 against +40.8 points; +2.0 against +2.2; +5.2 against +4.3).
  • The falsification work is layered rather than ritual: swapped answers, matched moments (around which visits dip 0.8 points), per-person counting, dropping the heaviest 5% of users, dropping the largest retailer and splitting the period all leave the named-product rise between 2.0 and 2.4 points (§3.1).
  • The LLM judges are validated against a blind human reader, with the precision adjustment carried through to the headline estimate (2.0 unadjusted, 1.9 adjusted) and the answer-type judge's errors shown to run conservatively (§2).
  • The practitioner implications follow from the estimates: link use of 27% among shoppers absent a week against 12% for recent visitors (+14.7 points [11.9, 17.6]) gives a non-obvious reading of referral reports (§3.2, §4).
  • The Limitations paragraph volunteers that the main group was chosen after first results and that its store-visit rise (+5.1) is smaller than over all store-naming answers (+10.5).

Recommendation

Major Revision for the leading journals of the field. The product-level linkage and the quantified attribution gap are a contribution this tier publishes, and the replication design is better than the norm here. What stands in the way: the central finding is written as an effect of the answer while the design cannot separate it from the intent that prompted the question; no sample size is reported anywhere; and the metric is offered as an evaluation instrument without being validated as one. All three are addressable within the paper's own approach.

Major Concerns

  1. The causal reading is not identified, and the writing does not respect that (Abstract; §3.1; §4, "the journey began earlier, with the assistant's answer"). The three comparisons address habits, store and category popularity, and how busy the person was online, but none addresses the obvious confound: a person asks for a light running watch because they have decided to buy a running watch. Under that account the answer and the later product view share a cause, and both the 44%-against-1.5% contrast and the 2.07-against-0.02 rise arise without the answer changing anything. Matched moments hold fixed activity and timing, not purchase intent, and the one check bearing on intent — that only 8% of matched products were named by the shopper earlier in the conversation — tests explicit naming in that conversation, not a decision formed before it. The Ethics statement concedes the assumption; the abstract, introduction and discussion do not. Either state the association throughout and reserve causal language for the open questions, or add evidence on intent: the shopper's own search and browsing in the hours and days before the answer, and whether follow-through differs for answers that merely confirm an already-chosen category or product.

    Two relevant literatures are absent from the reference list. Lewis, Rao and Reiley, "Here, There, and Everywhere: Correlated Online Behaviors Can Lead to Overestimates of the Effects of Advertising," WWW 2011, 157–166, names exactly the bias the matched-moments design is trying to remove. Blake, Nosko and Tadelis, "Consumer Heterogeneity and Paid Search Effectiveness: A Large-Scale Field Experiment," Econometrica 83(1), 2015, 155–174, shows branded-search traffic is largely substitutable and non-experimental attribution overstates it by an order of magnitude, with the same recent-versus-infrequent-user heterogeneity the authors find in link use.

  2. No sample sizes are reported anywhere (footnote 1: "We do not report panel size, per-analysis user counts or per-cell sample sizes"). Confidence intervals do not substitute. The reader cannot tell whether the per-assistant split, the three-store results, the price contrast or the 53% subset with matched moments rest on hundreds or on tens of thousands of answers, cannot judge whether twenty retailers and two markets are dominated by one of each, and cannot assess what the bootstrap over people is resampling. Commercial sensitivity is a fair constraint on absolute panel size but does not require withholding every count: report relative cell sizes, order-of-magnitude bands, or counts normalised to the main group. Without something of this kind the subgroup claims in §3.1 and §3.3 cannot be evaluated at this tier.

  3. The proposed metric is not validated as a metric (§3.1 box; §4, "Evaluating assistants and agents"). The paper asks the community to adopt product follow-through to "evaluate shopping assistants and agents by what people do next", but offers no evidence that it tracks anything one would want an assistant evaluated on. It is never related to a downstream outcome — purchases are acknowledged as unavailable, but repeat use, return visits and basket behaviour are not discussed either — so a high score may equally reflect an assistant that names popular products shoppers would have viewed anyway. The one comparative application, ChatGPT +2.2 [1.8, 2.7] against Gemini +1.0 [0.5, 1.6], is unadjusted for who uses each assistant, for what kinds of request, and for how often each names a specific product at all; as presented it is not interpretable as a difference between assistants, and three of the five assistants in §2 go unreported. Decompose that gap by request type, category, user composition and specific-product rate, and state what the metric does and does not license.

Minor Comments

  1. The ratio framing — "about a hundred times", and "from 0.02 to 2.07" in the abstract — rests on a baseline the audit shows is weakly measured (54% [39, 68] of 39 baseline matches confirmed against 93% [86, 97] after an answer, plus an uncorrected false-negative rate of 4% of 24 rejected pages, drawn from an unstated window, applied to a rejected pool far larger than a 0.02-per-100 rate); lead with the absolute +2.0 points, which is robust.
  2. Figure 3's caption reports raw shares for February to October, so the §3.2 route results are pooled and do not receive the held-out replication §3.1 and §3.3 receive; say so, or replicate them separately.
  3. The answer-content analysis (§3.1, Figure 2a bottom) tests at least three contrasts and reports one marginal interval (+0.93 [0.19, 1.63]); state how many were examined and whether multiplicity was addressed.
  4. The store-recommendation and view-provenance results rest on three of the twenty stores (§3.3); flag this where the numbers appear, not only in the parenthetical.
  5. Note that the past-week-visit restriction defining the main group is the same variable that drives the link-use heterogeneity in §3.2, where it is an outcome.
  6. The non-named-product result in §3.3 is a bound (2.6 to 3.5 per 100 against 1.4) wide enough that "keeps browsing" should be stated with it, not as a point estimate.
  7. Give the judges' option sets and criteria, the retailer selection rule and the matching tolerances in an appendix, even if the panel cannot be shared.