Change windows for machines: the CI build that unlocked, deployed, and relocked prod by itself

My CI pipeline files its own change tickets now.

This week an ordinary website release did something I have wanted from infrastructure for twenty years: the build requested a change window, production hosts unlocked themselves, the deploy ran, integrity checks verified what changed, the hosts locked again, and the change record closed with the evidence attached. Zero human touches. I stared at the audit trail for a while.

The premise: locked means locked

AlertKick production hosts sit SSH-locked by default - not firewalled-with-exceptions but locked, enforced by the agent at the host, with no standing allowlist. I wrote about our auto-block banning the ops IP; the conclusion there was explicit beats exempt. If deploy tooling can always get in, “locked” is marketing. So nothing gets in outside an approved window, and deploys ask permission the way a human operator would:

1. The pipeline creates a change request: title, description, window, target hosts.
2. On approval, the window opens: unlock commands flow to the target agents, scoped to the window.
3. Agents unlock within seconds; the deploy runs.
4. The pipeline completes the change; agents relock.
5. Verification queries the file-integrity trail for the window and attaches what changed to the record. In-window changes attribute to the change request; out-of-window changes alert.

An abandoned change self-relocks when its window expires - the failure mode of a crashed pipeline is locked hosts, never open ones.

What it felt like the first time

The first end-to-end run was a real deploy: create, approve, start, both target agents unlocked in under fifteen seconds, the deploy succeeded on all hosts, complete, relock in seconds, verification returned clean - zero unexpected files changed. The question every audit and every incident begins with, what changed in production, when, and who authorised it, has a record for an answer instead of an archaeology project across shell histories.

The incident that made it trustworthy

Two days in, the wrapper handed me a gift. During a deploy, the API credential the script used was revoked mid-run (a key rotation happening in parallel; single-founder problems). The script’s HTTP helper did not treat HTTP errors as failures, so the “complete change” call failed silently, the script exited green, and both production hosts sat unlocked for twenty minutes until I noticed.

A green build with production open is the worst outcome the system can produce - worse than a failed deploy, because nothing summons a human. The hardening that shipped generalises to any script that closes any security control:

• Fail on HTTP errors. The difference between a silently ignored 401 and a loud one was one curl flag.
• Retry the close. Completing the change, the step that relocks, retries with backoff, because it is the step you least want lost to a blip.
• Verify the control re-engaged. The script now polls until every host actually reports locked again. Trusting the API’s 200 is trusting a message about the state; the host’s own report is the state.
• Fail the build on relock failure even when the deploy succeeded. Deliberately. A red build gets investigated; a green one never does.

The pattern is portable to anything that opens a hole and promises to shut it.

Why this matters beyond neatness

Every compliance framework asks for change management, and most change management is theatre - tickets written after the fact, describing roughly what someone remembers doing. This loop inverts it: the change record is not documentation about the deploy, it is the mechanism of the deploy. Access exists only because the record exists, and attribution, scope, and evidence become properties of the pipe.

And then there is the reason I keep returning to: increasingly, the hands on the keyboard are not human. Agents and pipelines were the actors in everything above. “An AI deployed it” is exactly as acceptable as “some engineer did it, not sure which”: unacceptable, unless the actor has an identity, an authorisation, and a log. A change window does not care whether the requester has a pulse, which is what makes it the right control when pipelines and AI agents are the actors. Machines get the same deal as people.

Honest edges

Break-glass is still the open problem: the unlock path runs through the platform, so if the platform itself is down hard, opening a window needs an out-of-band answer (a locally-verifiable signed token or a documented provider-console path), and that work is not finished. Relock failure currently fails loudly and pages me, which is correct but is an alarm, not a remedy. I would rather ship the loop with its edges named than pretend it is finished; that honesty is the point of the whole system.

If your deploys go through a hole that is always open, the lock is a decoration. Give the machines a change window.