Evidence Quality, Not Evidence Volume: A Proposed Framework and Pilot Protocol for Grading Third-Party Security Evidence in Vendor Risk Assessments
Lorenzo Novas
Abstract
Third-party risk management (TPRM) programs routinely collect large volumes of vendor security evidence — SOC 2 reports, ISO/IEC 27001 certificates, penetration-test attestations, and security questionnaires — yet assessors have no shared, validated method for grading the quality of that evidence. In practice, review effort is allocated by document presence ("did we receive a SOC 2?") rather than by evidentiary strength ("does this SOC 2 cover the service we are buying, for the period we care about, with exceptions that matter to us?"). This paper proposes the Evidence Quality Grade (EQG), a five-dimension ordinal scoring framework — scope congruence, temporal validity, independence, exception relevance, and verifiability — that produces a single comparable grade per evidence item. We specify the framework precisely enough for independent replication, state three falsifiable hypotheses about its effects on assessor agreement and review time, and present a complete pilot protocol (60 de-identified evidence items, 6 assessors, crossover design) that a practitioner team can run inside a single organization without external funding. If the hypotheses hold, EQG offers TPRM programs a defensible way to spend scarce review effort where the evidence is weakest, rather than where the paperwork is thickest.