A dropped SSH session is not a dead roll

Vitaliy Mogilevskiy · October 27, 2026 · 9 min read

Somewhere around 4 a.m.

You're halfway through a major-version roll. The jump box drops your SSH session. You reconnect, run the same command to resume, and hammerdown says:

ERROR: process lock: lock is held: process/roll (holder: {"host":"jump1","pid":4242,...};
  remove /home/you/.hammerdown/locks/process__roll.lock if stale)

Your session died, so the roll died with it. The lock is stale. The tool is telling you how to fix that. You remove the file and run the command again.

That message is real. I wrote it, and it shipped in hammerdown's first public release, v0.1.0. In the post announcing it, I described the tool as rolling a fleet “with the brakes on.”

The roll from pid 4242 never died. It kept going on the jump box, and now there are two.

The operator isn't the bug

It's easy to read that story and blame the person who deleted the file. Don't. They did what the tool told them to do, with the information it gave them.

When I'm not sure, I stop and ask someone. I've done this long enough that a held lock I can't explain scares me. Here's what took me too long to see: my stop threshold is not the stop threshold of the people who run my tools. Some of them are newer. Some are covering a rotation they didn't train for. All of them are tired at 4 a.m., with a deadline someone set last quarter.

The tool knew things the operator didn't. It knew which host held the lock and which pid. It could have checked whether that pid was still running. Instead, it handed the riskiest decision in the whole roll to the person least equipped to make it, and phrased it as a suggestion.

The lock wasn't the real brake

So I fixed the message. Then I asked a harder question: what if the lock gets removed anyway? Someone with root, a cleanup cron, a lock service that loses its state. What does the second copy actually do?

hammerdown already had shard locks, so two copies can't work on the same shard at the same moment. I assumed that made it safe. It doesn't, because a plan is a snapshot. Each copy read the fleet once, at startup. The second copy's snapshot says the old primary is still primary and the replica is still a replica. The first copy has since failed that shard over. When the second copy finally gets the shard lock, it restarts the new primary, because its plan says it's a replica.

I wrote a test to prove it. A fake topology server returns fresh copies on every read, the way a real one does. Two batches share one set of locks:

snapB, _ := tp.Nodes(ctx)
planB := Plan(snapB, f) // B's view: n0 is every shard's primary

snapA, _ := tp.Nodes(ctx)
newBatch().Run(ctx, Plan(snapA, f)) // A rolls everything, failing every primary over

newBatch().Run(ctx, planB) // B runs its stale plan

The fake records the role each node really had when it was restarted. Without a re-read, B's log is a list of restart ... (really primary). Every one is a primary going away mid-traffic.

Now the humbling part. An earlier version of this tool, which I also wrote, didn't have this bug. Right before restarting a replica it refreshed the node's metadata, and it refused if the node had become a primary. The check was three lines. It got lost when I ported the idea into hammerdown, and nothing failed, because nothing in the test suite ran two copies at once.

The bug I found while fixing it

While I was testing the new lock messages, I sent a running roll the signal your shell sends when an SSH session drops: SIGHUP. hammerdown handled Ctrl-C carefully. In-flight work finishes, nothing new starts, locks are released. It didn't handle hangups at all.

So a dropped session killed the process on the spot. That could happen in the middle of a failover, and it left both the process lock and the shard lock behind. In other words, hammerdown produced the exact situation in the cold open by itself: a lock held by a session that really is dead, next to a message suggesting you delete it.

The fix is one signal in a list. A hangup now drains the roll like the first Ctrl-C:

received hangup (terminal gone?): draining - in-flight operations finish, locks are released
( worker=01 ) ... stopped: stopped before failover of users/-80:db1: interrupted: received hangup
released process lock process/roll

The in-flight restart finished. The failover never started. The locks are gone, so the next operator finds nothing to delete. Run long rolls inside tmux or screen anyway: draining is graceful, but it still stops the roll.

What changed

Three brakes, all with tests, shipped in v0.2.0 (PR #6).

1. Re-read the shard under its lock. After a worker takes a shard's lock, it reads that shard again from the source of truth. If any node's role, state or running build no longer matches the plan, it skips the shard untouched:

SHARD users/-80   shard changed since it was planned: users/-80:db2 was replica, now primary
                  (skipped untouched; rerun to replan)

2. Check roles before every node operation. That's the three lines I lost in the port, back where they belong: in the processor, right before each restart and the failover:

if err := job.CheckRoles(ctx); err != nil {
	return err // a replica became a primary since we last looked: don't touch it
}
return p.Ops.Restart(ctx, n)

That also catches what two-instance testing never would: an automatic failover in the middle of a shard.

3. A lock never asks you to judge its holder. The lock checks for itself whether its holder's pid is still running, and the error says so:

ERROR: process lock: lock is held: process/roll by pid 4242 on jump1 (running),
  since 2026-10-27T11:02:17Z, last heartbeat 8s ago
the holder (pid 4242 on this host) is RUNNING. Do not remove the lock: reattach to it
  (tmux/screen), stop it with Ctrl-C there, or wait. 'ps -p 4242' shows it

Releasing a lock is now a command, hammerdown unlock, and it refuses while the holder is running. When the holder really is gone, it takes --reason and writes who released it, and why, to unlock.log. If the holder is on another host, you also need a stale heartbeat and --confirm-host <that host>. There's no --force.

Isn't that dummy-proofing?

Reed Hastings has a line I agree with: “People tend to think that they need a process for everything, and once in a while you hear 'We're going to dummy-proof it.' But if you dummy-proof the process, you only get dummies to work there.” (Stanford GSB, 2014)

None of these brakes takes a decision away from the operator. The operator still picks the slice, the workers, the speed and when to stop. What the brakes add is facts the operator can't see from a terminal at 4 a.m.: whether that pid is alive, whether the shard changed, whether the lock changed hands. The operator in my cold open didn't lack judgment. They lacked one fact, and the tool had it.

Guard what the operator can't see. Trust them with what they can.

What code can't fix

I'd like to end with “and now it's safe.” It isn't. A tool is only as safe as the last incident it caused. Better brakes buy you a smaller incident. Two things buy you fewer of them.

A runbook is a reminder, not training. I've written detailed runbooks for tools like this: how to start a batch, which channels to announce it in, what to watch, when to stop. A runbook can tell you a held lock is dangerous. It can't make you feel it at 4 a.m. Repetition does that. A new operator should run dry runs, then canaries, then a real roll with someone who knows the tool's sharp edges sitting next to them, and get signed off before running solo. Scary becomes boring through reps, not through reading. Slow is smooth, and smooth is fast. Most of ops still learns this as optional tribal knowledge, so leadership has to make it mandatory.

Count trained operators per timezone. If one person in a region can run the tool, that knowledge is one vacation away from gone. The schedule will still say the roll continues tonight, and someone will pick it up. Two is one, and one is none.

The deadline is not a safety input. Nobody remembers that an upgrade slipped a week. Everyone remembers the outage. A deadline set in last quarter's planning meeting doesn't know who's on call tonight, who's trained, or what's already degraded. When those disagree, the deadline moves.

Try to break it

The two-instance test is roll/stale_test.go, and the lock tests are in lock/lock_test.go. If you run hammerdown against a real fleet, run them against your backend's fakes. If you've seen a failure mode these brakes wouldn't catch, open an issue. Those stories are how the next brake gets built.

If you're running your own fork, hardening it against exactly these failure paths is now part of HASHJOIN's Adopt review: two instances, a stale plan, a dropped session, and a lock that asks you to delete it.