The hire you can't evaluate
Sooner or later, most engineering teams need to hire someone who will know more about operations than anyone who interviews them: the first Staff SRE, the engineer who will own the database fleet, the platform lead who will decide how everything gets deployed. The panel is strong product engineers, and the loop they know how to run is a coding round, a system-design whiteboard and a culture chat.
That loop tests the part of the job that's getting cheaper. AI writes decent code now. What it doesn't do is notice that a shared dependency will take down a whole region, stop a rollout at the first odd metric, or decide what not to build. That judgment is what you're hiring a Staff engineer for, and a loop that can't see it ends up hiring the best storyteller.
At Slack I ran the General Tech & Automation round for Senior and Staff database infrastructure candidates, more of them than anyone else on my team. This is that round, rebuilt in my own words, so a panel without deep operations experience can still get a real signal. It takes 45 minutes and needs no whiteboard.
Let them pick the system
Open with one request:
“Pick the system you've owned that best shows how you automate operations. We'll spend the whole time on it.”
Letting the candidate choose sounds generous. It's actually the hardest version of the interview. There's no unfamiliar domain to blame and no trick question to recover from. It's their best ground, chosen by them, so whatever depth they show here is the most depth they have.
Read the resume beforehand and have a second system in mind in case the first one turns out to be thin. Then spend the remaining 40 minutes going down, not across. One system explored to the bottom tells you far more than five systems described at the top.
Five questions, in order
The order matters. Each question builds on details the candidate has already committed to, which makes the later ones hard to bluff.
1. “Walk me through it, down to where it runs.”
Ask for the architecture, then keep drilling until you hit bedrock. What runs where? How does a component get onto a host: an image, a package, a config management run, an operator? How many hosts, shards or clusters? How much data, and how often does it change?
Strong candidates get more specific as you go. They name the runtime for each piece and quote numbers that hang together: forty clusters, three replicas each, a few terabytes on the largest one. Be wary when a whole layer collapses into a single word like “the cloud” or “Kubernetes” and stays there, or when the numbers can't be reconciled with each other.
2. “Where does it break first?”
Then stop talking. This question usually needs a long pause, and filling the silence for them ruins it.
You're listening for a named single point of failure and its blast radius: what fails, what else fails with it, and how they know. The best answers come with a story about how they found out, often from a drill or an incident that went sideways. Generic answers (“networking,” “the cloud provider”) and answers about someone else's component are weak.
If the candidate can't find it, a small nudge is fair: “What happens when this component restarts?” But write down that you nudged. A Staff engineer who finds the weak spot unprompted and one who agrees with it once it's pointed out are different hires.
3. “What would you change, and why hasn't it happened?”
The second half is the important part. Everyone has a list of improvements. What separates senior engineers is that they can explain the constraint that's kept each one from happening: migration risk, a team that owns the other side, an SLA that rules out downtime, a quarter's worth of work for a modest gain. They'll usually mention an alternative they considered and why they rejected it.
“I'd rewrite it in a better language” with no cost attached is a yellow flag. So is a list of changes that never touches the fragile part they just named.
4. “Which parts did you build?”
Ask it late, on purpose. By now the candidate has described the system in detail, so you can see whether the details they've given are details of their own work.
Strong candidates draw a clean line: “the original design predates me, I built the scheduler and the pre-flight checks, a teammate owns the operator.” That precision is a good sign in itself. Be careful when the answer moves to the edge of the system, such as a reporting job, a cleanup task or a billing integration, after twenty minutes of talking about its core. That's someone who worked near the system, not on it, and it's a pattern I saw more than once behind strong-looking resumes.
5. “How do you rebuild component X? Should that be automated?”
Pick the most stateful piece they mentioned, usually a database node or a coordination service, and ask how they'd replace it from scratch. Then ask whether that procedure should be automated.
A strong answer knows the real procedure, how long it takes and what goes wrong partway through. On automation, either answer can be right. What matters is the reasoning: how often it happens, how risky it is, how many people can do it by hand today, and what a half-finished run leaves behind. “We'd just redeploy it” for something that holds data means they've never had to do it.
What to listen for between the answers
Some of the most useful signals don't belong to any one question.
- Adoption. Ask how much of the fleet runs through the tool today. A self-service platform that has provisioned a handful of clusters in a fleet of a thousand is a design, not yet a track record. That's not disqualifying, but it changes what you can conclude.
- Scale over familiarity. Listen for choices made because they fit the scale, not because they were comfortable. A candidate who knows an approach won't scale and picked it anyway, out of habit, is telling you how they'll choose on your team.
- Manual seams. Every automated system has a step a person still does by hand, like an approval or a policy agreed over chat. Strong candidates know where theirs are and have an opinion about them. Candidates who don't know where they are usually haven't been on call for the system.
- Clarity under depth. The engineer you want at 3 a.m. gets clearer as the questions get harder. If answers get vaguer, longer and less structured the further down you go, that's the signal, whatever the resume says.
When the project can't carry the interview
Sometimes the chosen system turns out to be small, early or mostly someone else's. Don't spend the remaining 25 minutes punishing the candidate for it. Change the frame:
“Suppose you were building this from scratch, without the constraints you had. How would you do it?”
Now you're testing design depth without the safety net of their own history. Some candidates come alive here because their last job never let them work at the level they're capable of. Others can only describe what already existed. Both are useful to know, and the switch is fair to the candidate either way.
If they did a take-home or have open-source work, spend five minutes on it. Don't grade the code again; ask about a choice they made and what they'd do differently at ten times the scale. Whether they say “I knew this wouldn't scale, here's why I chose it anyway” or “I hadn't considered that” tells you a lot.
Grade against your scale
The most common grading mistake is comparing the candidate to an imaginary ideal, or to the biggest company on their resume. Grade them against your environment: your fleet, your team and how much mentorship you can actually give.
I used five outcomes: strong yes, yes, yes with reservations, not for this role, and no. The middle three do most of the work. “Proficient at moderate scale and can grow into ours with some mentorship” is a hire, if you have the mentor. “Deep expertise in a niche, will need help with networking and provisioning” is a hire with a plan. And “not for this role” is not the same as “no”: it's often the right answer for a strong engineer interviewing for the wrong job.
Write it up the same day
Notes go stale within hours, and memories turn into impressions. Write the assessment while the conversation is fresh, and keep it to evidence:
- Recommendation, graded against your scale, with one paragraph on why
- The system and their role in it, including what they built versus what the team built
- Evidence for each question: what they said, not adjectives about it
- Where they'd need support in your environment
- What this round couldn't assess, and two or three questions for the rest of your loop
“Strong communicator” is an adjective. “Explained how a replacement replica gets its data, and when they'd choose a base backup over restoring from the archive” is evidence. Write nothing you wouldn't be comfortable showing the candidate, and nothing that isn't about the job.
Here's a sample assessment written in this format for an invented candidate.
Mistakes interviewers make
- Talking too much. If you're speaking more than a quarter of the time, you're interviewing yourself.
- Rescuing, then crediting. Nudging is fine. Forgetting you nudged, and writing “identified the single point of failure,” is how weak candidates get through.
- Grading the logo. A famous employer tells you the candidate was near hard problems, not that they solved them. Question four exists for this.
- Grading tool familiarity. Someone who has run Postgres at scale can learn your MySQL fleet. Someone who has only clicked through a managed console probably can't learn either.
- Asking trivia. Flags, defaults and port numbers measure memory. Ask what they'd do, then ask why.
None of this needs a senior operations engineer on the panel. It needs an interviewer willing to keep asking “and then what?” until the candidate hits bottom, and to write down honestly where that was.
If you'd rather have someone who has run this round more than twenty times do it for you, that's what the Pager Test is: a calibration call, this interview, a written assessment by the next business day and a debrief, for a fixed $2,500 per candidate.
HASHJOIN Field Notes
New tools and field notes on running production fleets: rolling upgrades, incident forensics, and Claude-augmented ops. About once a month, no spam.
Hiring a senior ops engineer?
The Pager Test is this round, run by someone who has done it more than twenty times: a calibration call, the interview, a written assessment by the next business day and a debrief. $2,500 per candidate.