Notes · Open source

A dropped SSH session is not a dead roll

I audited hammerdown against the worst way a rolling restart can fail. It found three brakes I'd left off, in a tool I'd just published under the line “with the brakes on.”

Vitaliy Mogilevskiy · October 27, 2026 · 9 min read

Somewhere around 4 a.m.

You're halfway through a major-version roll when the jump box drops your SSH session. You reconnect, run the same command to resume, and hammerdown says:

ERROR: process lock: lock is held: process/roll (holder: {"host":"jump1","pid":4242,...};
  remove /home/you/.hammerdown/locks/process__roll.lock if stale)

Your session died, so the roll must have died with it, and the tool is telling you how to clean up. You remove the file and run the command again.

But the roll from pid 4242 never died. It's still running on the jump box, and now there are two of them.

That error message is real. I wrote it, and it shipped in hammerdown's first public release, v0.1.0, the release I announced in the launch post as a way to roll a fleet “with the brakes on.”

The operator isn't the bug

It's easy to blame the person who deleted the file. Don't: they did exactly what the tool told them to do, with the information it gave them.

When I'm not sure, I stop and ask someone, and I've done this long enough that a held lock I can't explain scares me. What took me too long to see is that my stop threshold is not the stop threshold of the people who run my tools. Some of them are newer, some are covering a rotation they didn't train for, and all of them are tired at 4 a.m., working to a deadline someone set last quarter.

The tool, meanwhile, knew things the operator didn't. It knew which host held the lock and which pid, and it could have checked whether that pid was still running. Instead it handed the riskiest decision in the whole roll to the person least equipped to make it, and phrased it as a suggestion.

The lock wasn't the real brake

Rewording the message was the easy part. The harder question was what happens when the lock gets removed anyway, by someone with root, a cleanup cron, or a lock service that loses its state. What does the second copy actually do?

hammerdown already had shard locks, so two copies can never work on the same shard at the same moment. I assumed that made it safe. It doesn't, because a plan is a snapshot. Each copy reads the fleet once, at startup. By the time the second copy gets a shard's lock, the first may already have failed that shard over, but the second copy's snapshot still says the old primary is the primary and the new primary is a replica. So it restarts the new primary.

I wrote a test to prove it. A fake topology server returns fresh copies on every read, the way a real one does, and two batches share one set of locks:

snapB, _ := tp.Nodes(ctx)
planB := Plan(snapB, f) // B's view: n0 is every shard's primary

snapA, _ := tp.Nodes(ctx)
newBatch().Run(ctx, Plan(snapA, f)) // A rolls everything, failing every primary over

newBatch().Run(ctx, planB) // B runs its stale plan

The fake records the role each node really had when it was restarted. Without a re-read, B's log is a list of restart ... (really primary), and every line is a primary going away mid-traffic.

The humbling part is that an earlier version of this tool, which I also wrote, didn't have this bug. Right before restarting a replica, it refreshed the node's metadata and refused if the node had become a primary. The check was three lines. It got lost when I ported the idea into hammerdown, and nothing failed, because nothing in the test suite ever ran two copies at once.

The bug I found while fixing it

The opening scene assumes the roll survived the dropped session, the way it does when it runs inside tmux or under nohup. While testing the new lock messages, I tried the other case: a roll still attached to the session when the session drops.

When that happens, the shell sends SIGHUP to everything running in it. hammerdown handled Ctrl-C carefully: in-flight work finishes, nothing new starts, and the locks are released. It didn't handle a hangup at all. The process died wherever it was, mid-restart or mid-failover, with no chance to finish or clean up, and it left the process lock and the shard lock behind.

So v0.1.0 lost either way. If the roll survived, the error told you to delete a lock that was still in use. If it didn't, the roll had died halfway through a shard, and the same message gave you no way to tell which case you were in.

The fix was one more signal in a list. A hangup now drains the roll, exactly like the first Ctrl-C:

received hangup (terminal gone?): draining - in-flight operations finish, locks are released
( worker=01 ) ... stopped: stopped before failover of users/-80:db1: interrupted: received hangup
released process lock process/roll

The in-flight restart finished, the failover never started, and both locks were released, so the next operator finds nothing to delete. Draining is graceful, but it still stops the roll, so keep running long rolls inside tmux or screen.

What changed

The hangup fix shipped in v0.2.0 (PR #6) together with three new brakes, all with tests.

1. Re-read the shard under its lock. Once a worker holds a shard's lock, it reads that shard again from the source of truth. If any node's role, state or running build no longer matches the plan, it leaves the shard untouched:

SHARD users/-80   shard changed since it was planned: users/-80:db2 was replica, now primary
                  (skipped untouched; rerun to replan)

2. Check roles before every node operation. These are the three lines I lost in the port, back where they belong: in the processor, right before each restart and the failover.

if err := job.CheckRoles(ctx); err != nil {
	return err // a replica became a primary since we last looked: don't touch it
}
return p.Ops.Restart(ctx, n)

This also catches something two-instance testing never would: an automatic failover in the middle of a shard.

3. A lock never asks you to judge its holder. The lock now checks for itself whether its holder's pid is still running, and says so:

ERROR: process lock: lock is held: process/roll by pid 4242 on jump1 (running),
  since 2026-10-27T11:02:17Z, last heartbeat 8s ago
the holder (pid 4242 on this host) is RUNNING. Do not remove the lock: reattach to it
  (tmux/screen), stop it with Ctrl-C there, or wait. 'ps -p 4242' shows it

Releasing a lock is now a command, hammerdown unlock, and it refuses while the holder is running. When the holder really is gone, it requires --reason and records who released the lock, and why, in unlock.log. If the holder is on another host, it also requires a stale heartbeat and --confirm-host <that host>. There's no --force.

Isn't that dummy-proofing?

Reed Hastings has a line I agree with: “People tend to think that they need a process for everything, and once in a while you hear 'We're going to dummy-proof it.' But if you dummy-proof the process, you only get dummies to work there.” (Stanford GSB, 2014)

None of these brakes takes a decision away from the operator, who still picks the slice, the workers, the speed and when to stop. What the brakes add are facts the operator can't see from a terminal at 4 a.m.: whether that pid is alive, whether the shard changed, whether the lock changed hands. The operator in the opening scene didn't lack judgment. They lacked one fact, and the tool had it.

Guard what the operator can't see. Trust them with what they can.

What code can't fix

I'd like to end with “and now it's safe,” but it isn't. Better brakes make the next incident smaller. Making incidents rarer takes three things no code can do.

A runbook is a reminder, not training. I've written detailed runbooks for tools like this, covering how to start a batch, where to announce it, what to watch and when to stop. A runbook can tell you a held lock is dangerous, but it can't make you feel it at 4 a.m.; only repetition does that. A new operator should run dry runs, then canaries, then a real roll with someone who knows the tool's sharp edges sitting next to them, and get signed off before running solo. Scary becomes boring through reps, not reading. Most of ops still treats this as optional tribal knowledge, which is why leadership has to make it mandatory.

Count trained operators per timezone. If only one person in a region can run the tool, that knowledge is one vacation away from gone, and the schedule will still say the roll continues tonight. Someone will pick it up anyway. Two is one, and one is none.

The deadline is not a safety input. Nobody remembers that an upgrade slipped a week; everyone remembers the outage. A deadline set in last quarter's planning meeting doesn't know who's on call tonight, who's trained or what's already degraded. When those disagree, the deadline moves.

Try to break it

The two-instance test is roll/stale_test.go, and the lock tests are in lock/lock_test.go. If you run hammerdown against a real fleet, run them against your backend's fakes too. And if you've seen a failure mode these brakes wouldn't catch, open an issue: those stories are how the next brake gets built.

If you're running your own fork, hardening it against exactly these failure paths is now part of HASHJOIN's Adopt review: two instances, a stale plan, a dropped session, and a lock that asks you to delete it.

Want hammerdown running on your fleet?

HASHJOIN builds the backend for your database stack in a fixed-price, three-week build ($30,000), then keeps it current with support from $2,000 a month. Already running the open-source version? Adopt reviews and hardens your fork.