The day our own fail2ban banned the ops IP

Last week AlertKick’s SSH auto-block, a fail2ban-style feature where the agent bans brute-forcing source IPs at the host firewall, banned me. Deploys to staging started failing with SSH connection timeouts, and I burned an evening learning what our own automation looks like from the receiving end. Every team shipping automated defensive response eventually runs this experiment; here are the results so you can skip it.

The incident

Deploy jobs to two staging nodes began timing out on SSH while the applications on those nodes kept serving traffic normally. Healthy apps and dead SSH pattern-matched to “node half-wedged” just well enough to send me down the wrong road.

The auto-block feature had banned the ops workstation’s IP, the machine that is simultaneously the Ansible source, the deploy runner’s egress, and my terminal. Something had tripped the failed-auth threshold from that IP, the agent did exactly what we shipped it to do, and dropped our own operations traffic at the firewall.

Then I made everything worse in ways worth listing precisely:

I probed port 22 to “check” the nodes. A quick connection sweep across the fleet to see which hosts were reachable. Port probes from a banned IP look exactly like scanning - because mechanically they are scanning - so the sweep re-triggered detection, banned me on an additional node, and reset the ban clock on the first ones. Diagnosing the outage extended the outage.

Ping told me nothing. ICMP is filtered on these hosts, so “no ping” supported the dead-node theory without being evidence of anything.

I hard-reset a healthy server. Convinced one node was wedged, I reached for the provider API and power-cycled it. It came back fine - because it had been fine. Its only problem had been that it couldn’t hear me, and rebooting a machine for that is the ops equivalent of shouting louder at someone who has muted you.

The actual resolution required doing nothing: bans self-expire after an hour. Waiting was the fix, and every impatient retry before that pushed the fix further away.

What the design got right and wrong

Right: bans expire. The single most important property. A self-expiring ban converts “locked out of production” into “locked out for under an hour”. If our bans had been permanent-until-removed, with removal requiring the very SSH access that was banned, the recovery story becomes console access and rescue images. Every autonomous defensive action needs a bounded lifetime or a recovery path that does not depend on the resource it just blocked - ideally both.

Wrong: no liveness signal that wasn’t itself suspicious. Once SSH is banned and ICMP is filtered, the only innocent health checks are the application-level ones - health endpoints, provider API status. I had those; the failure was procedural, not technical. Under mild panic I reached for the muscle-memory tools, and the muscle-memory tools were all indistinguishable from attack traffic. Now it is written down: never probe SSH to check a host, read the app-level signals first.

No allowlist, on purpose. The obvious fix is exempting trusted ops IPs from auto-blocking. We chose not to. A standing hole for specific IPs is a standing target - compromise the trusted box, or its address, and the entire defence waves you through. It also rots: IPs change, exemptions outlive their reasons. Instead the direction is making legitimate operational access explicit rather than exempt - deploys announce themselves through a declared maintenance window that opens access, does the work, and closes it, rather than being whitelisted past the guardrail forever. Explicit beats exempt: it leaves an audit trail, and it expires.

The checklist for auto-response features

1. Every automated block gets a TTL. No exceptions shipped in v1.
2. Recovery must not require the blocked resource. Prove it by rehearsal, not by assertion.
3. Give operators an innocent liveness signal, and write down which signals are not innocent from a flagged source.
4. Prefer explicit, expiring access (windows, declared maintenance) over standing exemptions.
5. Assume the first serious triggering of the feature will be your own team, and treat that as the acceptance test.

We ship software that auto-blocks attackers, so getting banned by it is embarrassing - and it is also the feature demonstrating it does not play favourites. I have had worse evenings for worse reasons. The staging environment now defaults to a more permissive posture while production keeps the strict one, the runbook says “wait out the ban” in bold, and the unnecessary reboot is immortalised here as a warning to future me.