Poor men can't afford cheap shoes — Joshua Cook Consulting

There is an old line that a poor man cannot afford cheap shoes. The cheap pair wears through in a season and he buys the next, and over a few years he has spent more than one good pair would have cost and walked on failing soles the whole way. The good shoes were the economy. He could not see it at the counter, where the only visible number was the price.

Document extraction has arrived at the same trap, and cheap models have made it easy to walk into.

In the benchmark I ran across 2,280 SEC filings, one of the less expensive models returned clean, correctly-shaped JSON on every long document I gave it, and on those long documents the fields came back empty. Every field present and correctly typed, the structure valid against the schema on the first attempt, and nothing inside.

On any evaluation that scores cost and format, that model looks like the best deal in the set. It is among the cheapest to run and it never produces malformed output, so it passes the checks most pipelines actually apply. A team watching throughput and valid-JSON rate would promote it. They would be buying the cheap shoes.

Three checks — cost, format, and content — applied to the same empty output. Cost and format pass while content fails, two green checks hiding one red. — The same output under three checks. Cost and format both pass; only content fails. Two green checks can hide the one that matters.

Format validity and content correctness are different properties, and only the first is cheap to check. A schema confirms that an answer is shaped like a date. It cannot confirm that the date is right, or that a date belonged in that field at all. A pipeline can run for months returning well-formed output that has quietly drifted from the documents it is meant to be reading, and every automated check it passes stays green the whole time. The soles are gone while the shoe still looks fine on the shelf.

This persists because the alternative is genuinely hard. Verifying that an extracted value is correct means comparing it against something you trust more than the extractor, and for most real tasks that reference does not exist yet. That problem is the subject of the pieces that follow.

So the question worth asking is whether you own any instrument that would notice the day your pipeline started returning nothing. Cost will not tell you, because every model is cheap now, and valid output will not tell you, because most models produce it. The poor man's mistake was trusting the one number he could see at the counter. Most extraction teams are making the same one.

Read what an instrument like that measures, or the full benchmark.