How XSci Finds Strong Preprints, and How We Know It Works

A demonstration and validation of the AI pipeline that surfaces strong preprints for expert review

About XSci

Science runs on trust, but the machinery meant to produce it is failing. Peer review is slow, opaque, inconsistent, and increasingly gamed, and a rising flood of AI-generated research is set to overwhelm a system that was already straining. Today no one is the credible, independent arbiter of research quality. We think there should be one.

XSci is building a credit rating agency for science: a certification layer that combines AI with expert peer review so researchers, institutions, and AI developers can assess research quality quickly and at scale. Much of that work is deciding what "quality" even means, and how AI and expert judgement combine into a defensible rating.

Why we're publishing this

XSci runs an ongoing effort to find strong research among recent preprints. An AI pipeline ranks new preprints by quality, and we invite the authors of the strongest to our human review service, where domain experts in the same field are compensated to review the paper in depth. This report explains that AI ranking and shows how we validated it, so the authors we invite can see the selection rests on a tested, transparent process rather than guesswork. The 1,000-paper run described here is a starting point; the effort is ongoing as new preprints keep appearing.

The pipeline in this report is the ranking agent that surfaces candidates at scale, and it is deliberately lighter than the reviews an invited author can then use. XSci offers two of those: a per-manuscript AI review, a more thorough agentic workflow with live novelty search and deeper checks than this ranking pass, and the human review service described above.

The approach: a two-stage quality funnel

We rank preprints the way a careful program committee would: quickly screen everything, then spend real effort only where the decision is close.

The fast pass scores every paper on a transparent rubric. The deep pass re-examines only the papers near the cut, where a single fast score is too noisy to trust, using head-to-head comparison against papers of known quality.

Stage 1: The fast pass (transparent composite score)

Every paper is read by a calibrated frontier language model (Claude Sonnet 4.6) and scored on a signed rubric: strengths add points, concrete flaws subtract them.

Positive dimensions (each scored 1–10, with 2–4 lines of evidence quoted from the manuscript):

Negative points: the model lists concrete, evidence-grounded flaws, each with a category and a severity penalty from −1 (minor) to −10 (fundamental). There is no fixed number of flaws to hit and the model reports only the ones it can back with evidence, anywhere from none to several.

composite = 0.5·significance + 1·rigor + 1·evidence + 1·clarity + 0.5·literature
            + capped_negative_penalties

Two deliberate design choices:

• Significance and literature grounding are down-weighted (0.5×). Both depend on knowledge from outside the manuscript: how novel a contribution is, and whether prior work is engaged with honestly, can only be judged against the wider literature. That is the hardest thing to get right from text alone, especially without a live novelty search, so these two count for less than rigor, evidence, and clarity, which can be checked directly against the paper itself.
• Flaw penalties are capped: no single flaw can subtract more than −5, and total penalties are floored at −12. This prevents one harshly-worded objection from sinking an otherwise strong paper. It guards against a documented failure mode of automated review: a recent large-scale expert study of AI reviewers (Kim et al., 2026) found that, unlike human reviewers, LLMs take an overly critical stance on minor issues.

Every score is grounded in quotes from the paper. The model reads the manuscript as submitted, and we add no external signal to it: no venue, acceptance status, citation counts, or author-reputation lookups. (On what the model might still recognise from its own training, see the limitations below.)

Stage 2: The deep pass (re-ranking the boundary)

A single fast-pass score is reliable enough to separate clearly-strong from clearly-weak work, but it is too noisy right at the cut line. That's exactly where accept/reject decisions are made, so that's where we spend more compute.

For the band of papers around the cut, the deep pass runs head-to-head, full-paper comparisons:

• Full papers. Each side is compared on up to ~30k tokens of the complete manuscript (not a summary or a lossy abstract). The median paper fits entirely, and about 80% are compared in full; the longest ~20% are truncated after the main text, where the cut falls in appendices and reference lists. A full comparison uses only ~6% of the model's 1M-token context window, so comparisons are never constrained by context length. How much of each paper we feed in is a choice about comparison quality and cost, not a limit we're forced into.
• Anchored to known quality. Each boundary paper is compared against anchor papers with known outcomes: real ICLR accepts (reviewer rating ~6) and rejects (~4), plus a sparse set of comparisons against other candidates.
• Both orderings. Every pair is judged in both directions (A-vs-B and B-vs-A) to cancel position bias.
• Verified flaws as context. Flaws found in the fast pass are independently re-verified. Confirmed flaws are fed into the comparison as context rather than as blind deductions.
• Bradley-Terry ranking. Head-to-head wins are converted into a latent quality score via a regularised Bradley-Terry model (the same statistical model used to rank chess players and sports teams), with regularisation to keep the ranking stable even when a paper wins or loses all of its matches. The score is centred at 1.0, an average paper, so anything above 1.0 wins more of its match-ups than it loses.

Does it actually work? Three independent validations

Method is only worth as much as its evidence. We ran three separate checks against external ground truth.

1. Are the flaws we flag real? On par with frontier AI, above human reviewers

We evaluated our flaw-detection against PeerReview Bench, a benchmark of papers with expert-annotated review points. The question: when our system claims a paper has a flaw, how often is that flaw actually valid?

To compare fairly, one common grader (Claude Opus 4.8) judged every reviewer's flagged items by the same standard. On that footing our pipeline's flags hold up 97% of the time, on par with the frontier AI reviewers (94-95%) and well above the human experts (49-73%). We would not read the small margin over the other AI systems as us being better: they are different, and some earlier, models, so an edge of a point or two proves little. The takeaway is simply that our flags are as reliable as a frontier AI reviewer's, and more reliable than a human's. We favour precision over volume: the system raises fewer objections than a human, but the ones it raises are much more likely to be valid, so as an author you are far less likely to be penalised for a complaint that isn't actually correct.

Why do the AI reviewers, ours included, come out ahead of the human experts on this metric? Precision measures one narrow thing: of the criticisms a reviewer raises, how many are correct, significant, and backed by evidence in the text. Human reviewers tend to judge a paper more holistically and raise a wider, more varied set of concerns, including judgement calls and higher-level questions that a strict correct-and-evidenced rubric scores conservatively. However, AI reviewers tend to converge on concrete, text-grounded points that the rubric rewards. So AI looks strong on this specific measure, but that is not the same as being the better reviewer. The benchmark's own authors reach that conclusion, positioning AI reviewers as complements to, not substitutes for, human experts (Kim et al., 2026), and noting that AI reviewers carry weaknesses humans do not, such as limited subfield knowledge and an overly critical stance on minor issues. Human experts bring what this metric cannot capture: judging whether a contribution actually matters, deep command of the field and its prior work, and accountability for the decision. That is precisely why our funnel uses AI to surface and flag, and human experts to do the review that follows.

The GPT-5.2, Claude Opus 4.5, and Gemini 3.0 Pro rows are reviews those models wrote. The benchmark collected them and had expert scientists annotate each one (Kim et al., 2026), so we did not generate them. To compare all reviewers on the same footing, every reviewer's flagged items are then scored by one common AI meta-reviewer (Claude Opus 4.8), deliberately a different and stronger model than the one that produced our items, so we are not marking our own work. The meta-reviewer sees the paper and each flagged item and verifies it against the text on three axes (is the criticism correct, is it significant, is its evidence sufficient), counting an item as valid only if all three hold. Figures are per criticism, not per paper, over the same 20 benchmark papers. These rates are best read as a ranking rather than exact scores: an AI grader is more lenient than human annotators (the benchmark's own experts scored these reviewers in the 30-56% range), so every rate here sits higher, but what the comparison establishes is the order the reviewers land in.

2. Does the score track real quality? Accepted above rejected 86% of the time

We tested whether the fast-pass composite can distinguish papers that were accepted from those that were rejected at a top venue, using ICLR 2017 accept/reject decisions (n = 120; 60 accepted, 60 rejected).

• AUC (area under the curve) = 0.864. Given a random accepted paper and a random rejected paper, the composite scores the accepted one higher 86% of the time.
• Accepted papers averaged +11.2 against +6.9 for rejected papers (n = 60 per group), with individual scores spread roughly ±2.8 around each mean. That spread is why the two distributions overlap in the middle rather than splitting cleanly, and it is exactly what the AUC of 0.864 captures.

In the figure, the bars show the actual paper counts in 1.5-point bins. Composite scores fall on a 0.5-point grid, so the smooth curves are a kernel-density estimate that interpolates between those discrete values.

One pattern in the distributions stands out: the rejected papers look bimodal, clustering near 6 and again near 9 to 10, while the accepted papers form a single peak. If it holds up, it tells an intuitive story. The rejected papers may genuinely be two populations: clearly-weak papers (the lower cluster) and strong-but-rejected borderline papers (the upper cluster, which overlaps the accepted range). Those borderline rejects are exactly the papers the deep pass has to fight over at the cut. We note it only as an observation, though, and confirming it would take a larger sample and a formal test.

We also checked a harder, different cut: 60 recent ICLR 2024/25 papers, all of them accepted, with full text recovered from arXiv. With no rejects here, this isn't an accept-versus-reject test. Instead, we correlate our composite with each paper's mean reviewer rating. The signal is positive but weak (Spearman rho about 0.25), which is what you'd expect: among papers that all cleared the acceptance bar, genuine quality differences are small and the reviewer ratings themselves are noisy. A clean accept/reject separation on the 2017 set, plus a weak-but-positive correlation on the harder accepted-only set, is the pattern a real quality signal produces, not a second identical result.

3. Does the deep pass earn its keep? A 1000-paper run

For this study we ran the two-stage funnel over a single, recent slice of the literature: the 1,000 most recently submitted arXiv preprints in Computation and Language (the cs.CL category, i.e. natural-language-processing research), fetched newest-first at the time of the run. This is one field and one time window, not a random or cross-disciplinary sample, so the numbers below describe how the funnel behaves on that slice rather than on all of science.

• 999 of 1,000 papers were successfully scored by the fast pass.
• At the cut line (the top 100 of 999 scored, our target top ~10%), the deep pass re-examined a 150-paper band straddling the boundary (ranks 25 to 174, roughly 75 places on either side of the rank-100 cut, wide enough to catch any paper that could realistically move across it) and changed 26% of the final shortlist: 26 papers the fast pass had ranked below the cut won their way into the top 100, displacing 26 it had placed there. Several papers with higher raw composite scores lost their head-to-head matches and fell below papers with lower raw scores. That is the deep pass doing its job: it spends its effort precisely where the fast-pass score is least reliable, at the cut line, and leaves the clear-accept and clear-reject papers alone. The composite is also coarse right there: 20 of the band papers share the exact same score at the cut, so the fast pass cannot tell them apart and its rank-100 boundary among them is essentially arbitrary. Breaking those ties is exactly what the deep pass does.
• Sanity check passed cleanly: the known-good anchor papers scored ~2× higher than the known-bad ones (0.78 / 0.57 vs 0.24 / 0.21), with no overlap, confirming the head-to-head ranking is measuring quality, not noise.

The full shortlist of all 100 papers surfaced by this run is listed in the appendix at the end of this report.

What this ranking means for you as an author

• Every judgement is evidence-backed. Each strength and each flaw comes with specific quotes from your paper. There is no unexplained black-box verdict.
• Fewer false complaints. We optimise for precision: when the system flags an issue, that flag holds up as reliably as a frontier AI reviewer's, and more often than a human reviewer's. It stays narrow by design, sticking to concrete, text-grounded points. The more subjective, judgement-based concerns (novelty, significance, field context) are where human experts add the value, and the diversity of what they raise is a strength they own. That is the review your paper receives next.
• Borderline papers get a real second look. If your paper sits near the cut, it isn't decided by a noisy number. Instead it earns a full-paper, head-to-head re-evaluation against work of known quality, with the ability to be promoted.

Honest limitations

We would rather you trust us because we're candid than because we oversold:

• These results come from a simplified pipeline. To run economically across 1,000 papers, this study streamlined the AI review we run on individual submissions. The review a single manuscript actually receives is more thorough than the version measured here.
• Text-only, no figures. The pipeline reads the manuscript text, including tables and figure captions, but does not see the figures, plots, or images themselves. For figure-heavy empirical work, evidence or a flaw that lives only in a plot can be missed.
• No live novelty search in this run. To keep the large-scale run economical we did not perform live web search, so novel-versus-derivative judgements rely on each paper's own reference list. The per-manuscript production review does run this search.
• Benchmark labels are imperfect. Peer-review accept/reject is itself noisy at the margin. We read our numbers against that ceiling rather than treating human labels as perfect truth.
• This is a validated prototype, not a verdict. Our scores are a rigorously-tested signal of quality to help surface strong work, not a replacement for peer review or a final judgement on any individual paper.
• The model may recognise older papers. A language model can recognise a paper it encountered during training and, in principle, recall where it was published or how it was received. This does not affect the 1,000-paper run: those preprints are recent enough to postdate the model's training data. But some of the older papers used for validation and as known-quality anchors (for example the 2017 ICLR set) may predate that cutoff and could be recognised, so we treat this as a caveat on those specific validation numbers rather than on the ranking itself.
• Where AI review reaches its limits, human review begins. Automated review has an inherent ceiling: at best it offers a signal, not a final verdict. That, and the limits above, are exactly the gaps a human expert closes, which is why the AI ranking is only the first step: the strongest work is passed to expert human review. This report doesn't measure the human review itself. But the limitations listed above are exactly the ones expert human review is there to cover.

Appendix: The 100-paper shortlist

These are the 100 preprints (the top ~10% of the 999 scored) that the two-stage funnel surfaced from the 1,000-paper cs.CL run described above. They are listed alphabetically by title, not by rank.