Design an Eval Harness

An eval harness is a test runner that has stopped pretending every question has an answer. Everything hard about it lives in the third state.

The recipe

Say before you type: "Run every case, catch the throw, bucket the verdict into pass, fail, or unknown, and never let unknown touch the pass rate."

Three fields on the class and you are done: the cases, the optional scorer, the last run's rows.

Phase 1: a throw is a data point

The loop is a try around one call:

try {
  got = fn(evalCase.input);
} catch (err) {
  threw = true;
  error = err instanceof Error ? err.message : String(err);
}

Say it out loud when you write it: a case that throws is a failing case, not a broken harness. It scores false, it keeps its name, it carries the message, and the twelve cases behind it still run. A harness that dies on case three tells you nothing about cases four through fifteen, which is the only reason you built it.

Deep equality is your own five lines. Reach for JSON.stringify on both sides and you have quietly decided [undefined] and [null] are the same answer.

function sameValue(a: unknown, b: unknown): boolean {
  if (Array.isArray(a) || Array.isArray(b)) {
    if (!Array.isArray(a) || !Array.isArray(b)) return false;
    if (a.length !== b.length) return false;
    return a.every((v, i) => sameValue(v, b[i]));
  }
  return Object.is(a, b);
}

And keep run() a read. If you push into this.results, the second call reports fifty cases for a suite of twenty five.

Phase 2: the third state

A model judging a model returns three things, not two: yes, no, and I cannot tell. Fold the third into false and your dashboard says the system is broken when it is only unmeasured.

So pass is true | false | null, and the rollup is three counters. The only line that matters:

const judged = passed + failed;
const passRate = judged === 0 ? null : passed / judged;

Unknown leaves both sides of the fraction. One pass and one failure sitting next to eight unknowns is a pass rate of 0.5, not 0.1. And when nothing was judged, the answer is null. Not 0, which reads as a total failure, and not NaN, which reads as a bug in your code and eventually becomes one when someone renders it.

Two ordering rules to state before you are asked:

• A throw outranks the scorer. There is no output for a judge to read, so it never gets called.
• The scorer is scorer(got, expected). Most cases are symmetric, so a swapped signature survives every test you would write by hand.

Phase 3: the gate

compare(baseline) diffs the last run against a prior one. Two lines carry the whole idea:

if (now === false && before !== false) regressed.push(name);
else if (before === false && now === true) fixed.push(name);

Read them against the third state. null -> false regresses, because it started failing and you now have evidence. false -> null is not a fix, because losing your ability to judge is not the same as passing. true -> null is neither. Unknown is an absence of evidence, and absence of evidence moves nothing in either direction.

Then the part most people skip. Walk the intersection of the two runs and you have written a gate that goes green when someone deletes the failing test. Report both set differences:

• missing: in baseline, gone from this run.
• added: new here, absent from baseline.

stable is true only when regressed and missing are empty. A vanished case is lost coverage, not a pass. New cases never break the gate, or nobody would ever add one.

What this is really testing

Whether you know that an eval suite is a measurement instrument. Every shortcut here trades a correct answer for a confident wrong one: 0 instead of null, unknown folded into failure, a diff that ignores what disappeared. Name that out loud and the rest of the interview is easy.