A gate that only ever passes is no gate

Every team I have worked with has a CI pipeline that goes green. Very few can tell you what it would take to make it go red on purpose.

That distinction sounds academic until you watch an AI coding agent write both the implementation and its tests. The agent is optimizing for a green run. It will get one. The tests it writes will assert things that are true, and will keep asserting them long after the behavior underneath has drifted, because nothing ever required those assertions to be load-bearing.

So the discipline I ended up with is one rule: a gate has to prove both halves. It must fire when it should, and stay quiet when it should not. A check that only ever passes is indistinguishable from a check that is not wired up.

Proving the second half

The cheap way to prove it is to break the code deliberately and confirm the test notices. Mutation testing, done by hand, one mutation at a time.

Here is a real one. My résumé generator binds a section heading and its first entry together so a page break cannot strand a heading at the foot of a page. That worked. What I had not noticed was that the same wrapper also held every bullet beneath it, which made the whole block unbreakable, so a long entry could not fit anywhere in the remains of a page and jumped whole to the next one. The result was four and a half inches of white space on page one.

After fixing it I wrote tests, and then I attacked them:

M1 bind the whole list inside the wrapper  (the original bug)  -> KILLED
M2 drop the CSS that closes the seam                           -> KILLED
M3 split even a single-bullet entry                            -> KILLED
M4 drop a bullet from the continuation                         -> KILLED
M5 leave the role line outside the wrapper                     -> KILLED

Five deliberate breakages, five caught. Only after that did I believe the tests.

The failure mode nobody warns you about

On an earlier round I ran the same exercise and reported a survivor. A mutation went in, the suite stayed green, and I went looking for the hole in my coverage.

There was no hole. The mutation had referenced a line of code that no longer existed, so it applied nothing. I had tested an unmodified program and called the result a gap.

A mutation that cannot apply looks exactly like one that was killed. Both produce a green run and no diff in behavior. If you are going to work this way, the mutation step has to fail loudly when the edit does not land, or the whole exercise quietly reports whatever you were hoping to hear.

Why this matters more now

None of this is new. What is new is the volume. When a person writes a test, the act of writing it is itself a small check on whether the assertion means anything. When a machine writes ten thousand of them, that check is gone, and you are left with a suite whose green light means only that nothing threw.

Verification has to move from the reviewer's judgment into the build. Not because people are careless, but because inspection does not scale to the amount of code now being produced, and inspection was never good at catching things that are absent.

The question worth asking about any suite you inherit is not "does it pass?" It is: when did this last fail, and what made it fail? If nobody can answer, you do not have a gate. You have a light that is on.