How entries are selected for expert review: the method note

1. What the selection is for

Up to 100 entries are sent to a paid expert. The selection decides which entries go, not whether a paper is good. The expert decides that, on the test in rule 6.3: no major error, and a genuine prospect of publication in the target journals after the kind of revision a journal would normally ask for. The selection tries to send the expert the entries most likely to pass that test. An AI report verifies and certifies nothing (rule 5.4).

2. The evidence for each entry

Two AI referee reports. Each entry’s current version is read, as a PDF, by two AI referees: state-of-the-art large language models from two different frontier AI labs, so that they share fewer blind spots. Both follow the same instructions: acting as a senior referee for a leading venue of the manuscript’s field, each judges whether the evidence supports the central claims, names the contribution before looking for problems, and judges novelty against the closest prior work rather than the manuscript’s own framing. Each weakness carries a certainty (demonstrated, likely or possible); only a demonstrated problem that no revision could repair may be called decisive. The referees do not see the target journal and read at most the first 80 pages. They are instructed to treat text in a manuscript addressed to reviewers or AI systems as part of the manuscript, not as an instruction, and to report it as an integrity concern.

These reports exist for the selection and its evaluation: they are not shown to the author, published or charged to the author. The AI reports an author orders on XSci are the author’s own and are not read by the selection: which AI reviewers an author chooses, or choosing none, makes no difference.

One meta-review. A third state-of-the-art model, the meta-reviewer, acts as a senior area chair or journal editor in the manuscript’s field, with the target in mind. It receives the title, the abstract, the declared discipline, the target journal (the one the author ranked first, or none), a grading of that journal from a published venue catalogue where XSci holds one, and the two reports without the referees’ own recommendations, so that it weighs substance, not another model’s verdict. It counts each distinct problem once however many reports raise it, gives more weight to criticism that points to evidence in the manuscript than to general assertions, and does not treat routine requests for further work as flaws unless a central claim depends on them. It is not told the threshold, the number of places or anything about other entries, and sees no human review.

The meta-reviewer decides from the reports, as an editor normally does. It may ask to read the manuscript only when the two reports disagree on a point that would decide soundness or whether the manuscript would be accepted, and the manuscript can settle it: for example, one report shows an error in a proof that the other checked and found correct. It then checks at most five named questions and raises no new issue. It never reads the manuscript to judge novelty or significance. In testing before launch it asked once in 130 papers.

3. The signals

Soundness, yes or no. “Yes” means no major error or fundamental flaw: no problem that would change a main claim or that needs more than minor revision to put right. A “no” is required to name a referee report, quote it and identify the claim, result or step at issue. A “no” that names and quotes nothing is treated as “yes” and recorded; a quotation that does not match a report is recorded for a person to resolve, and the “no” stands meanwhile. Soundness is kept apart from the score, so that one well-supported serious problem rules out an entry on that version whatever its other merits.

The score, in two parts, each on the same 0 to 100 scale, anchored on descriptions a referee would recognise (from strong reject through borderline to strong accept), with reasons.

• General. What the reports and the abstract establish about the contribution and the evidenced problems, and how likely the manuscript is to be accepted by a leading venue of its field after no more than minor revision: changes that need no new experiments, data, proofs or analyses and alter no claim or conclusion.
• At the target. The meta-reviewer’s own judgement of fit with the target journal (its scope, the novelty and rigour it expects, its audience) and how likely acceptance there is after no more than minor revision. With no target, or one the meta-reviewer cannot place or of doubtful standing, the manuscript is scored against the leading venues of its field (rule 4.3).

The selection uses the score at the target, adjusted for the target’s standing (below). The general score is kept as a check (an unexpected gap between the two is counted, without affecting the decision) and for analysis against the experts’ decisions.

The standing of the target. The same score means more at a leading journal than at a modest one. The meta-reviewer places the target in its field (leading; established but below the leading ones; another recognised peer-reviewed venue; not recognised; none), using a reference grading as a hint where XSci holds one and stating its basis. The program, not the model, then applies a fixed adjustment for each tier; among recognised venues, it is larger the further the target sits below the leading venues of its field. The sizes are published with the full method. In effect, an entry naming an ambitious target judged a plausible fit ranks above a similar entry with a modest target (rule 5.3), and rolling places go to candidates for the leading venues of their field (rule 5.1).

Confidence: high, medium or low. High when the reports agree on the main strengths and weaknesses and the decisive points are specific and evidenced; medium when some disagreement or vagueness would not change the decision; low when the reports contradict each other on a decisive point, the decisive points are vague, or the abstract suggests the reports missed the substance. Low confidence keeps an entry out of the rolling places, which are for entries the reports clearly place, but is not a rejection: the entry is still ranked after the close.

Ratings and flags. The meta-reviewer also rates novelty, rigour, clarity and fit, the four aspects rule 5.3 names. The ratings are recorded for analysis, not added into the score: the four aspects enter the decision through the two judgements above. Of the flags it may raise, only one changes anything: an instruction addressed to it in the abstract or the reports, or in the manuscript when it reads it, holds the entry until a member of XSci staff has looked at it and cleared it.

4. Rolling places (rule 5.1)

Normally once a day, the selection considers every entry whose current version has been scored. An entry qualifies when soundness is “yes”; its adjusted score reaches the bar; confidence is medium or high; it has a declared discipline; it carries no instruction flag that staff have not cleared and is not held by XSci over an integrity matter; its submitting author is verified (rule 3.1); and its preprint is public.

Qualifying entries are admitted in order of submission while places remain; the score does not affect the order. An entry’s place is the later of the time it was entered and the upload of the version that was scored (rule 5.5(a)). An entry held only for its author’s verification takes its place at the first daily round that finds the verification settled (rule 5.1). Before any place is given, a member of XSci staff checks the day’s list for errors (the paper, the recipient, the admission email, possible duplicates) and approves it; the check does not reorder the list or apply another standard, so admission stays automatic in the sense of rule 5.1.

An entry that does not qualify on a given day is not told and not rejected: it is considered again later and ranked after the close. When the 60 rolling places are taken, the daily rounds stop until the close, and in any case they end at the close. A place whose entry is later withdrawn is not given to anyone else.

The bar was reviewed once, on 5 October, as announced, under a rule set before entries opened: it would be raised if the rolling places were projected to fill too quickly, lowered if too slowly, and otherwise kept. A lower bar applies to every entry already scored, a higher one only to versions scored afterwards, so the review could cost no entry a place. Its outcome will be published with the full method.

5. Places allocated after the close (rule 5.2)

Entries close on 30 November at 23:59 Anywhere on Earth. The remaining places, 100 minus the rolling places taken, are allocated in the first days of December, on the version and target standing at the close (rule 5.5(b)).

Who is ranked. Every entry without a place, unless its soundness is “no”, it carries an instruction flag that staff have not cleared, XSci is holding it over an integrity matter, it has no score or no discipline, its preprint is not public, or its submitting author was not verified at the close (under rule 3.1 such an entry is not an entry). Low confidence does not exclude an entry, and the rolling bar does not apply. XSci may nonetheless set a minimum score for these places; any minimum used will be published with the full method.

Categories. The 19 discipline categories of the platform’s research-field list, from Physics & Astronomy to Interdisciplinary and Other. An entry’s category is that of the first research field its author listed; a typed-in field not on the list counts under Other, and an entry with no field takes the manuscript’s subject area as recorded on XSci. Categories with fewer than five eligible entries are pooled with Other.

Shares. Each category’s share is proportional to its number of eligible entries without a place. It receives the whole number of places in its share, and places left over go to the categories with the largest fractions; no category receives more places than it has eligible entries. If fewer eligible entries remain than places, the extra places stay unfilled.

Ranking within a category. By adjusted score; ties go to higher confidence, then to the earlier place in the order of submission. The ranking only selects: it is not published and produces no placings.

6. Revised versions and changed targets (rule 5.5)

• Before admission a revised version replaces the earlier one. It gets its own two reports and meta-review, its score replaces the earlier one whether higher or lower, and its place in the order runs from its upload. Until it is scored the entry is not admitted on the earlier version.
• A changed target: the same version is meta-reviewed again against the new target, and its place in the order is unchanged. Switching back to a target already scored reuses that score.
• While the rolling places are full, revisions are not scored as they arrive; the version and target standing at the close are. A version uploaded after the close is not scored for selection.
• After admission nothing is rescored, and a revision never costs a place (rule 5.5(e)). A rolling-place author has five days from the notice of admission to post a revision, never past the close; the expert is ordered on the version standing when that window closes, which may be one the selection did not score. A post-close place is assessed on the version standing at the close (rule 5.5(c)). An expert review the author already bought counts as the entry’s one review (rule 4.4).

Revising in response to a report is never held against an entry (rules 5.5 and 11.1(a)), and since the selection’s own reports are not shown, no revision can be written against them.

7. What the method does not see, and other limits

• It inherits the referees’ blind spots. What both referees miss, the meta-reviewer misses; it reads the manuscript only to settle a decisive disagreement between them.
• What AI reports cover least. In a comparison of official human reviews with three AI review systems on 70 conference papers (Nguyen, Hao, Elazar and Tan, Benchmarking Agentic Review Systems, arXiv:2606.19749), human reviewers wrote 39% of the comments but 80% of the paper-level criticism, of novelty, scope and motivation, and humans and AI largely flagged different passages. That is the judgement journals accept and reject on, and the selection’s evidence is thinnest there.
• ”Sound” is not “verified”. In the same study no system caught every error deliberately inserted into papers: the best caught about seven in ten, some about a quarter. The study did not test the referees used here. A “yes” means no referee demonstrated a serious problem, not that none exists.
• A “no” is strict, and final for that version and target. The soundness test allows no more than minor revision, which is stricter than the expert’s test in rule 6.3 (the revision a journal would normally ask for), and a “no” keeps that version out of both the rolling places and the ranking after the close. Only a revised version, or a change of target, is assessed afresh.
• The benchmark. The method was tested before launch on computer-science papers submitted to one leading conference (ICLR 2025), in their versions from before the decision, with the conference’s decisions as labels. On 80 held-out papers the adjusted score ranked a randomly chosen accepted paper above a randomly chosen rejected one roughly four times in five. That is an upper bound, since the models may have seen those papers and their reception; 80 papers leave wide uncertainty; and one conference stands in for the leading venues of every field. Real entries, and their scores, differ from the benchmark’s, and how well the method carries over to other disciplines is unmeasured.
• Venues without a reference grading are classified from the meta-reviewer’s own knowledge, which may misjudge a specialised journal.
• Run-to-run variation. The models cannot be made fully repeatable: a second reading of the same input can give a somewhat different score, and near the bar a different decision, so scores are not recomputed on request.
• Other limits. Referees read at most the first 80 pages. When a model’s safety system declines a manuscript, the call is retried once, then sent to an earlier model from the same lab; every case is recorded. Two of the three models come from the same lab.
• What the expert decisions can show. Experts review only selected entries: their decisions show how often the selection’s choices are confirmed, not how many strong entries it passed over.

8. What stays private, and what will be published

The selection’s referee reports and meta-reviews, individual scores and the post-close ranking are not published. Selection is not appealable (rule 5.4).

With the final results we will publish the full method, including the threshold value, the outcome of the 5 October review and the adjustments for the target’s standing, with the evidence behind them; the breakdown of entries and awards by discipline; and where the process went wrong and should be improved.

Questions: competition@xsci.com.