Nine questions an auditor will ask about your extraction pipeline
Every check below comes from a measured finding in a public benchmark across 2,280 SEC filings. Nothing here is invented, and nothing asserts a number we did not measure.
Before you can answer any of them
1. Can you re-run last quarter’s extraction and get the same answer?
Not approximately. Byte-for-byte, or with a stated tolerance you chose deliberately. If the model version, the prompt, or the temperature has moved and you cannot reconstruct what produced a given output, you do not have a pipeline, you have a history. Everything below assumes you can re-run.
Whether the output means anything
2. Do you measure content correctness separately from format validity?
A schema confirms an answer is shaped like a date. It cannot confirm the date is right, or that a date belonged in that field. These are different properties and only the first is cheap to check. Most pipelines check only the first.
3. Would you notice if the pipeline started returning nothing?
In our benchmark one of the cheaper models returned clean, correctly-typed JSON on every long document, with every field empty. It passed schema validation on the first attempt and was among the cheapest to run. On any evaluation scoring cost and format it looked like the best model in the set.
4. Does your validity rate vary by document type, and do you know by how much?
Ours did, enormously: 63% of 8-K filings produced valid output against 96% of earnings calls, same models, same run. A single blended accuracy number hides exactly this. One model overflowed context on 85 of 150 documents and simply produced nothing for them.
Whether your measurement means anything
5. Do you know your noise floor?
Run the same model on the same document five times with identical settings and measure how often it agrees with itself. Until you have that number you cannot interpret any other number. In our run, self-agreement was 1.000 on a board roster and 0.707 on a narrative field, same model, same document, same settings.
6. When two models disagree, do you know how much of that is real?
Cross-model agreement on our hardest field was 0.185, which reads as models fundamentally disagreeing about what a document says. Set against a 0.707 self-agreement floor, most of that “disagreement” is one model disagreeing with itself. Disagreement between two instruments is uninterpretable without knowing how much each disagrees with itself.
7. Have you separated fields that are hard from fields that are under-specified?
When a model cannot agree with itself across identical runs, the usual cause is not the model. It is that the field has no determinate answer in the document. A human analyst asked the same question twice a week apart would answer differently too, and neither answer would be wrong. No amount of model shopping fixes a field that was never well posed.
Whether your controls are load-bearing
8. Do you know what your configuration flags do to the answer?
We toggled extended reasoning on and off, same model, same documents. Agreement between the two settings was 0.836 on 10-Ks. A flag many teams treat as a cost-or-latency decision changed the output roughly one time in six.
9. Is the number you watch most closely the one that tells you least?
Across five identical repetitions, cost varied by less than a tenth of a cent. It was the single most stable number in the run and it carries no information about whether any answer was right. Everything unstable was downstream of it.
If you answered no to 5, start there. Every other measurement is uninterpretable until you know your noise floor, and it is the cheapest thing on this list to establish: one model, one document, five identical runs.
The findings above come from a public benchmark across 2,280 SEC filings. Read the full benchmark, or see what an independent audit measures.