Does a Governed Confidence Score Know When a Text-to-SQL Answer Is Right? An Exploratory Study on BIRD and a Pre-registered Confirmation on Spider
Raghu Venkat, Tricha Anjali
Abstract
Conversational analytics lets people ask questions of records spread across many systems and get answers in plain language. It is useful only if each answer can be trusted and explained. A production system of this kind attaches a confidence score, the Interpretive Evidence Score (IES), to every answer it writes over a database, and uses it to show the answer (PASS), show it with a warning (FLAG) or withhold it (ABSTAIN). The score's weights, caps and thresholds were set by design. This paper is an audit of that design. On the 1,534 questions of BIRD dev, answered by GPT-5.1 with a benchmark-adapted production prompt and five samples per question, the designed score separates right from wrong answers weakly (AUROC 0.624, 95% CI 0.595-0.652) and withholds 35.7% of correct answers. Taking the score apart showed that nearly all of its ranking signal comes from agreement among the samples, and that most of it is lost where agreement is combined by a minimum with a checklist that drops to zero on any unsupported claim, including claims in commentary. We froze a rule in which agreement with the shown answer is the score and deterministic checks act as vetoes and caps, and pre-registered a test on Spider dev before any Spider answer existed. The primary hypothesis held: AUROC rose from 0.566 to 0.725 (paired difference +0.159, 95% CI +0.097 to +0.221), and the share of correct answers withheld fell from 49.0% to 8.8% with PASS accuracy 95.7%. Two results narrow the claim. Under one result-matching policy, the designed entropy and the share agreeing with the shown answer rank answers almost equally well, so the gain comes from how agreement enters the score and not from the statistic. And whether adding checks to agreement helps depends on how correctness is defined: on the same Spider answers, agreement alone ranked higher under model-judged labels and the rule with checks ranked higher under execution match, while the checks always cost unqualified coverage. A pre-registered empty-result check did not meet its effect-size criterion on Spider. Finally, agreement cannot resolve a business definition that is absent from the evidence, which motivates an evaluation of checks built from expert-approved definitions. Correctness here means the core answer to the question asked. Labels in this version are model-judged; a blinded two-person audit is not yet complete, and all correctness-based results, including the AUROC comparisons, remain subject to it. It will be reported, with the results recomputed on the adjudicated labels, in a revised version.