How to Vet a Software Developer: A Practical Framework

Why most technical assessment fails

The dominant failure in technical hiring is not that companies assess too little. It is that they assess the wrong things confidently. Algorithm puzzles, whiteboard exercises and trivia questions are all easy to administer and easy to score, which is precisely why they persist, and they correlate weakly with the thing you actually care about: whether this person will produce maintainable work in your codebase over the next year.

The second failure is unstructured interviewing. A conversation with a candidate feels informative and is almost entirely noise. Without a written rubric agreed in advance, an interviewer's assessment drifts toward candidates who communicate in a familiar style, who happen to share their background, or who they simply spoke to on a good day. This is well documented, and yet unstructured interviews remain the norm almost everywhere.

The framework below is not novel. It is the boring, evidence-supported version: assess real work, structure the interview, score independently, and then check the scoring. It takes more effort to set up and rather less effort to run, and it produces decisions you can actually defend afterwards.

The four-stage framework

1. 01

   Start with a work sample

   The strongest single predictor is a sample of the work itself. Ideally review real shipped code they can share; where they cannot, use a short scoped exercise resembling actual work rather than a puzzle. Assess what a good reviewer on your team would: boundaries, naming, error handling, test discipline, and whether the next engineer could change it safely.

2. 02

   Run a structured interview against a written rubric

   Write the rubric before you meet anyone. Define what a weak, adequate and strong answer looks like for each question. Ask every candidate the same core questions in the same order. This feels rigid and is the single highest-return change most teams can make to their hiring process.

3. 03

   Score independently, then discuss

   Each interviewer records their scores before any discussion happens. Discussing first produces convergence on whoever spoke most confidently rather than on the evidence. Divergent scores are useful signal about the candidate and about the rubric, and that signal is lost entirely when people anchor on each other.

4. 04

   Check references with the right question

   Reference calls that only confirm employment dates are theatre. The useful question is what changed after this person left. The answer is difficult to prepare for, and it reveals whether the person was genuinely load-bearing or merely present in the room.

What to assess in a work sample

• Do the module boundaries make sense, or is everything reachable from everywhere else?
• Are the names accurate, or do they describe what the code did two refactors ago?
• Is error handling deliberate, or is every failure either silently swallowed or immediately fatal?
• Do the tests describe behaviour, or do they assert implementation details that break on any change?
• Could a new engineer safely change this in six months without doing archaeology first?
• Is there evidence of judgement about what not to build, and not only about what to build?
• Where the code is bad, does the candidate know it is bad and can they say why?

The stage almost everyone skips

Calibration review is unglamorous and it is the reason standards hold over time. A single reviewer drifts, reliably and invisibly: toward candidates who resemble them, toward whoever they interviewed most recently, toward whatever topic they happened to be interested in that month. Nobody is exempt from this, and simply being aware of the bias does not remove it.

The correction is cheap. A second reviewer scores the same evidence against the same written rubric without seeing the first reviewer's conclusion. Where the two disagree materially, the candidate is re-interviewed rather than argued about. Over the course of a year, this is the difference between having a hiring bar and having a hiring mood.

It also produces a second benefit that teams do not anticipate: disagreements reveal defects in the rubric itself. If two competent reviewers consistently score the same answer differently, the rubric is ambiguous and should be rewritten. That feedback loop is how an assessment process actually improves over time rather than merely persisting unchanged.

Part of the Vetting & Technical Assessment cluster · Read the pillar page