A green check is a statement about the check

Every serious failure my systems hit last month had the same shape. A check passed. The check was real, the code was right, and the thing it was supposed to prove was false.

Four from one month.

A deploy pipeline reported SUCCESS. The build succeeded, the deploy step succeeded. The service returned 403 to every request. An --allow-unauthenticated flag had not taken, because the deploying service account could not set the service IAM policy. Nothing in the output said the flag was ignored.

A nightly job summarised each day's work. One night its API key was invalid and every call returned 401. That failure collapsed into "no items found." The job recorded the day complete with zero entries, and every retry afterward skipped the day as already done.

A command added one secret to a running service. It used --set-secrets, which replaces rather than adds, so it removed the twenty-two other secrets that service depended on. The command succeeded. Nothing reported dropped mounts. Writes failed for ten hours before anyone connected the two events.

A merged pull request fixed a bug in a script. Six hundred tests passed. The fix existed only in the copy installed on the machine, never in the repository.

There were tests in all four cases. They ran and they passed.

Capability is not limit

In each one, the check exercised the path where things work and concluded something about the path where they do not.

A colleague hit the cleanest version. He was handed an access token and wanted to know whether it was properly scoped. He used it to write something, the write succeeded, and he concluded the token had the narrow permissions he expected. It was unscoped and carried full administrative access. His test showed the token could write. It could not show what the token could not do, because he never asked it to do anything forbidden.

A check that only exercises the allowed direction cannot distinguish a bounded system from an unbounded one. Both pass.

Test the direction you are worried about. We now verify credential scope by attempting something that must fail. When we installed a token-issuing capability last week, the meaningful test was not that it issued a token. It was that it refused four requests: one for an out-of-scope resource, one escalating its own role, one omitting a required boundary, and one granting a permission it does not hold. Any of those succeeding would have meant the boundary was cosmetic. All four refused.

Verify at the destination

A check tends to run close to the thing being changed. The claim being made is usually about somewhere further away.

The deploy pipeline verified that the deploy command exited zero. The question anyone cared about was whether a user could reach the service. A service IAM policy sits between those two facts, and the check sat on the wrong side of it.

We took six attempts to see this one. We were shipping updates to a mobile application and confirming each by checking that the update channel had advanced. It had advanced, every time, and the reports were accurate. The question was whether the change had reached the device in someone's hand. A channel timestamp cannot answer that.

We embedded a build identifier into the application, visible on screen. One glance settled what six confirmations had not, because the check now ran at the destination.

Measured reasons and reconstructed ones

A correct instruction can carry an invented explanation.

We wrote a deployment note telling engineers to verify a DNS change by asserting response content rather than a status code. The instruction was right. The reason attached to it, a specific claim about what an unconfigured hostname returns, had never been measured. Someone measured it later and the reason was wrong. The instruction survived unchanged.

That is harder to catch than a wrong instruction, because reviewers check instructions. A plausible reason attached to a correct rule passes review, and then someone builds on the reason.

A rule that survives repeated corrections to its reasoning is not thereby well founded. It may mean only that nobody has tested the rule, just the stories attached to it.

I did the same thing to myself while writing this. I checked whether some tokens carried an expiry, read the field as absent, and reported that our system was issuing permanent credentials. The field was absent from the response. The value was there. The check I ran could not tell those two cases apart, which is the failure this piece is about, committed while describing it.

Three habits

Ask what your check would report if the thing you fear were true right now. If the answer is the same thing it reports today, it is not a check.

Test the refusal. A boundary is real once you have watched it turn something away.

Run the check at the destination rather than the origin. The served response, not the deploy log. The far end of the queue, not the queue.

When you write down why a rule exists, note whether you measured the reason or reconstructed it.