Shipped a feature Friday. Woke up Saturday to three broken integrations.

Friday evening I shipped a refactor to the part of our stack that decides whether a draft is allowed to publish. Saturday morning, three integrations were quietly broken. The deploy logs were green. The health endpoint was green. The work product was missing.

What broke

The refactor added a final voice scan at the publish boundary. The scan worked: it correctly flagged drafts that had been marked clean under an older scanner. The bug was in the rejection path. The code that should have re-queued the rejected drafts for rewriting was wrapped in a try/catch that swallowed errors. The drafts kept being rejected. Nothing was ever re-queued. Posts sat in "scheduled" forever, the dispatcher refused them every minute, the system reported success on each refusal.

Three scheduled posts had not gone out by 9am Saturday. Reading the logs took 20 minutes because the failure mode was perfectly silent.

What I learned about observability

Every refusal in an autonomous pipeline needs three fields in the log: (a) the reason, (b) the artifact id, (c) what the system did about it. The third field is the one most teams skip and it is the one that turns a silent loop into something a human can fix.

Without it, you see refusals firing on schedule and assume the system is working. With it, you can answer the only question that matters: did anything change as a result?

What I learned about Friday deploys

The standard advice is "don't deploy on Friday." That is not actually the lesson here. The lesson is that any change that touches an autonomous loop needs a post-deploy health check that watches the OUTPUT, not the deploy logs.

Did every agent that was supposed to ship work in the next 12 hours actually ship something? If the answer is no, roll back before the silence compounds. The post-deploy check takes one minute and would have caught this on Friday night instead of Saturday morning.

The takeaway

When you build autonomous systems, you stop being able to trust green checks. The system can be green and broken at the same time because the unit of work is no longer the request and response. The unit of work is whether something happened in the world that should have happened. Every observability decision flows from that.

Three integrations broken for 14 hours is a cheap lesson. Two weeks of silent rejection would have been a brand incident.