Thirty-nine green runs, and no preview was ever uploaded

A red build is annoying, but it is honest. It tells you something is wrong and roughly where. The failure we have learned to fear is the green build that checked nothing, because it does more than fail to help. It trains everyone to trust it, and the longer the streak, the deeper the trust.

Here are the best specimens from our own repos, all real, all found by us, most of them found embarrassingly late.

Thirty-nine green runs of a skipped job

This website has a workflow called preview-deploy. Its first 39 runs were all green. Then, on September 20, it went red, and the natural reaction was “what broke?”

Nothing broke. It had never uploaded a preview.

The skip was designed, not accidental. A step called Check Cloudflare secrets looked for credentials that did not exist yet, and when they were missing every later step was told to stand down. The run even printed a notice saying no preview was uploaded. Opening one of the old successful runs showed what that meant in practice: the secrets check reported success, then 11 steps were skipped, including Upload preview version, which is the entire point of a job named preview-deploy, and the preview copies of our content-leak gates. (Our main CI workflow kept its own leak gate, which did run.)

The day the secrets landed, the guard flipped, the job ran its real steps for the first time, and it hit a wall that had been there all along: You cannot upload a new version of a Worker that does not yet exist.

So the bug was not that the job skipped. Degrading gracefully when a credential is missing is fine, and this one even said so in a notice nobody reads. The bug was that a run which skipped 11 of its steps finished with the same green check as a run that did the work. Those two outcomes should never share a color.

A job called Verify that verified production

On September 19 we audited the CI on our four demo apps. Each had a check called “Quick Verify (all PRs)”. It ran a ledger format check, a self-test of the review gate, and a smoke test that curled the URL of the deployed demo.

It did not run the unit tests. It did not typecheck. It did not lint. It did not build. On at least one demo, it was the only workflow there was.

The one behavioral step it did run pointed at production, which on our setup does not contain the PR’s code, since merging does not deploy. The check could still go red, and sometimes did, when production or the ledger had a problem. What it could not do was go red because of the PR. A PR that broke the build, or turned all 139 of one demo’s tests red, would have merged with a green check next to its name.

The portal had a subtler version. Its real tests lived in a separate CI workflow that did run on every PR, so it was never untested. But its Quick Verify had three hollow layers stacked up:

1. It smoke-tested the deployed site, not the PR.
2. It used an assertion type called redirects_to that was documented and never implemented. The script printed a “skip” line for each one and moved on, and curl -L had already followed the redirect and thrown away the Location header anyway.
3. Its “home page” assertion looked for sign-in copy, which only appeared because -L had silently followed the redirect to /login. The test named one page and checked another.

None of this is visible from reading the config, because the config reads fine. You have to read what each check targets, not what it is named. A check pointed at production reports production’s health. It is not a gate on your branch, however confident its name sounds.

The suite that ran a third of itself

One of our apps had an npm script along the lines of node --test src/**/*.test.ts, with the glob unquoted. On Windows, the shell passes that string through untouched and Node expands it fully. On the Linux CI runner, sh expands it first, and without globstar the ** matches one directory level.

So CI ran 22 of the 67 test files we had when we found it, on every run since tests were added to CI. The tests that never ran in CI included the ones for redaction, auth and outbound-data rules, which is roughly the list you would pick if you were trying to skip the important ones. Local runs on Windows saw all 67 files (a local failure on a commit CI had just called green is how we found it), so the people running tests by hand saw the whole suite and CI saw a third of it. Both were telling the truth about different things.

The fix was quoting the pattern. The lesson was asserting the test count in CI. A green check that does not say how many tests it ran is a horoscope.

Zero entries, zero problems (a near miss)

Our ledger checker validates the change-log entries in a repo. Reading its code, we found that it returned an empty list, and therefore passed, when the ledger directory did not exist at all. At least one of our container builds excludes docs/, where the ledger lives, so a check run inside it would have passed while looking at nothing.

As far as we can tell from the run history, that never actually happened: we found no run of the checker against a missing or empty directory. We caught this one by reading, before it produced a false pass, and fixed the copies of the script across our repos.

It now fails on zero entries, and fails on a missing directory with a different message that names the exact path it looked in. A search that finds nothing because it was looking at nothing must not look like a search that found nothing because everything is clean. “0 matches in 4,812 bytes” is a result. “0 matches” by itself tells you nothing.

Two smaller ones that bite anyway

GitHub Actions runs run: steps on Linux with bash -e but without pipefail. So npm test | tee log exits with tee’s status, and tee is very good at its job. A failing suite passes. Declaring shell: bash turns on pipefail.

And a race: one repo gated a job on the PR not being a draft and cancelled in-progress runs per PR number. A push while the PR was a draft (run A, which skips) and marking it ready a second later (run B, which does the work) could finish with A cancelling B. The newest result would be A’s skip, a skipped required check counts as passing, and the PR would show as clean with nothing run. We found it before it bit a merged PR. The fix was putting the draft state into the concurrency key, so a draft run cannot cancel a ready one.

The rule we kept

When you check that main is green, open the step list of a passing run. Failing runs get attention for free. And when a guard exists, delete it once on purpose and watch something go red. If nothing does, the coverage was decorative. We have done that often enough now that a suite surviving its own guard’s deletion no longer surprises us. It still annoys us.