We ran the same prompt, against the same model, on the same inputs, twice. The two runs disagreed with each other on 11 of 124 predictions, roughly 9%.
Nothing was broken. No prompt had changed, no model had been swapped, no input had drifted. That disagreement is just what the model does, and if you are shipping LLM features without knowing your own version of that number, every eval result you have ever celebrated or panicked over is suspect.
We build production LLM systems, and every one of them eventually needs the same thing: a way to know whether a change made the system better or worse. So we build eval frameworks alongside them, wire them into CI, and live with them through prompt revisions, model upgrades, and refactors. Evals are how we make the understanding checkpoint mechanical instead of aspirational. Some of what we learned confirms the standard advice. The lessons below are the ones we have not seen written down, and they transfer to any team shipping LLM features. The numbers throughout are real measurements from one production system we built and instrument; your numbers will differ, but the method for finding them will not.
The short version
- Run your eval twice before you trust it once. Our two identical runs disagreed on ~9% of predictions, a spread bigger than the improvement we were trying to measure.
- A pass/fail threshold sitting inside that spread is a coin flip wearing a merge gate's clothes. We had one. This post is about what we replaced it with.
- Noise rotates between items. Real regressions fail the same items every run. If you only store aggregates, you cannot tell the two apart.
- Record real model responses once, then replay them through your real production code in CI. Deterministic, free, and loud when stale.
Lesson 1: Measure your LLM eval noise floor before anything else
Your noise floor is the disagreement your system shows between runs of an identical configuration: same prompt, same model, same inputs. Until you know it, no eval delta means anything, because you cannot tell whether a change in a metric came from your change or from the dice.
Here is the experiment that taught us this. We had revised a prompt and wanted to measure the improvement, so we recorded a fresh run. Then, to sanity-check, we recorded a second run of the identical configuration.
- The two identical runs disagreed with each other on 11 of 124 predictions (~9%).
- Agreement with our ground-truth labels was 0.895 in one run and 0.839 in the other, a spread of 0.056.
- The improvement we were trying to measure, the delta between the old prompt and the new one, was 0.032.
The run-to-run spread was nearly twice the size of the effect. Comparing one run of the old prompt against one run of the new one and calling the difference an "improvement" would have been reading tea leaves. We had been doing exactly that for weeks.
Two honesty notes that matter when you run this yourself. First, a handful of runs is not a variance estimate; it is a lower bound. The spread you observe is the least noise your system has, never the most, so treat it as a floor and do not dress it up in standard deviations. Second, temperature 0 does not rescue you. Providers do not guarantee reproducible outputs even at temperature 0 (batching, hardware, and routing effects all leak in), and most production pipelines do not run at 0 anyway.
How to apply it: before you trust any eval number, run your identical configuration at least twice and diff the item-level results. Publish the spread next to the metric, every time. Any improvement claim smaller than the spread is a claim about luck.
Lesson 2: A threshold inside your noise floor is a coin-flip merge gate
Once you know the noise floor exists, you start seeing what it does to pass/fail thresholds.
One of our metrics is a false-alarm rate: how often the system flags a problem that the ground-truth label says is not one. We had a blocking ceiling on it at 0.10: any run above the ceiling fails the change. Then we looked at four consecutive recordings of the same configuration on the same inputs: 0.048, 0.133, 0.048, and 0.114.
A spread of more than eight points, with no code or prompt change anywhere in sight. Our 0.10 ceiling sat inside that band. Which means the gate was not measuring quality at all. It was flipping a coin, and an engineer whose perfectly good change happened to land on run 2's dice roll would have "failed" it, while a genuinely harmful change could sail through on run 1's.
The tempting response is to iterate on the prompt until the rate comes down and the band clears the ceiling. We tried to reason through that and rejected it, for two reasons that generalize. First, at a few dollars per live run with an eight-point noise band, you cannot even tell whether an iteration helped or you just rolled better; you would be tuning by superstition. Second, on most tasks the errors are coupled: pushing a model to flag less trades tolerable over-flagging for intolerable under-flagging. We raised the ceiling above the observed band and demoted it to what it honestly is: a tripwire for catastrophe, not a precision instrument.
How to apply it: audit every blocking threshold you have against your measured spread for that metric. If the margin between baseline and threshold is smaller than the spread, the gate is theater. Either widen it into a tripwire or replace it with something deterministic (lesson 6).
Lesson 3: Real regressions are stable blind spots; eval noise rotates
The most useful diagnostic heuristic we found costs nothing to use.
In the two-identical-runs experiment, both runs missed exactly 2 of the 19 real issues in the label set. Same count; any dashboard would call the runs equivalent. But they were not the same two issues. The misses rotated.
That observation generalizes into the fastest way we know to tell a regression from a dice roll:
- A real regression is a stable blind spot. When a change genuinely breaks something, the same cases fail run after run, because the cause is in the change, not in the sampling.
- Noise is a rotating blind spot. Failures move between items, and the metrics drift in both directions at once: stricter on some items, looser on others. A real regression pushes one way.
The practical consequence is about what you store. If your eval only records aggregates, you cannot run this diagnostic at all. Keep per-item outcomes for every run, and when a metric moves, diff which items changed before you react. Ten minutes of item-level reading regularly saved us days of chasing phantom regressions, and once or twice it caught a real one hiding inside a "normal-looking" aggregate.
Lesson 4: Record and replay LLM calls at the model boundary
Everything above requires running evals often, and running evals often is only sane if most runs are free. The architecture that makes that true is old-fashioned: record and replay.
Route every model call through a single client interface, then wrap it. In record mode, each call's full context (model, system prompt, user prompt, key inputs) is hashed into a fixture key, and the model's real response is written to a committed, human-readable file. In replay mode, the same hash looks up the recorded response, and your entire production pipeline runs on top of it.
Three properties fall out of this, and they do all the work:
- Replay runs your real code. Not a re-implementation, not a harness-only approximation. The same functions that serve production run in the eval, with only the model call swapped for a recording. The common anti-pattern here is an eval harness that rebuilds a simplified copy of the pipeline; it drifts, and then you are measuring the harness, not the product.
- Replay is deterministic and free. No API key, no network, no spend. It runs on every pull request in CI, in seconds, offline.
- A miss is loud. If a code change alters what the pipeline sends the model, the hash changes, the recording is not found, and the run fails with instructions, instead of silently passing on stale data.
None of this came from an eval framework. The harness is in-house: plain TypeScript wrapping the provider's SDK, committed JSON fixtures and labels that are schema-validated on load (a malformed artifact fails the build instead of silently skewing a metric), and the same test runner the rest of the codebase already uses. It is a few hundred lines, and most of them are the checks, not the plumbing.
Recording is the only part that costs money, and not much: a full re-record of our suite is 25 model calls, between one and four dollars. Alongside the recordings, commit baselines that pin everything the run depended on: prompt version, exact model configuration, fixture list, a digest of the output, and the measured metrics. A baseline is a signed statement of "this is what the system did, under exactly these conditions." Every comparison in this post is a comparison between baselines.
Lesson 5: Re-record only when the model's inputs changed
The recordings raise the obvious question: when do you refresh them? The decision rule we landed on is the single most useful sentence in this post:
Re-record when the model's inputs changed. Replay when only your handling changed.
A prompt edit or a model upgrade changes what the model would say, so the recordings are stale and you pay the few dollars. But if the change is deterministic code downstream of the model (a scoring rule, a parser, a text normalization), the recorded responses are still exactly right, and replaying them measures your change in isolation, for free.
Getting this wrong is not just wasteful; it destroys the measurement. Re-recording re-rolls the sampling dice, and the noise you re-roll can be larger than the effect you are trying to see. You will conclude your fix helped when the dice did, or that it failed when the dice ate it.
The clearest example from our own work: we changed a pipeline so that quote verification decided credit, meaning the model had to support its judgments with quotes that actually match the source document. On replay, the false-alarm rate jumped from 0.048 to 0.114. The cause was not the model. Our document extractor emits markdown, and the quote matcher did not fold markdown bold, so when the model quoted the text as it read it (without the asterisks), the match failed and correct judgments were downgraded to false alarms. We fixed the folding rule and replayed again: 0.114 down to 0.095. The entire investigation, regression to root cause to verified fix, ran on the same 25 recorded responses. Zero API spend, zero sampling variance, exact attribution.
There is a deeper lesson inside that bug: the deterministic and stochastic layers interact. A strict matcher amplifies harmless variation in how the model formats a quote into a real metric swing, so some of what looks like model noise in a live run is actually your own pipeline's sensitivity to noise. Replay is the instrument that lets you tell the two apart, because it holds the stochastic layer perfectly still while you change the deterministic one.
One refinement we only got right on the second attempt: version pinning has to be per eval job, keyed to the prompt each job actually calls. Our first version had one global prompt version, so editing one prompt forced unrelated jobs to re-record too, re-rolling their variance for no reason. Too coarse and you pay for spurious re-records; too fine and a real regression hides behind a stale baseline.
Lesson 6: Deterministic checks gate merges. Statistical metrics report.
If accuracy thresholds are too noisy to block merges, what does block them? Our answer: everything that is decidable gets a deterministic check, and deterministic checks always block. In an LLM pipeline, far more is decidable than teams assume:
- Structural integrity. Every input got exactly one output, no missing or duplicated items.
- Grounding. Every citation the model makes resolves against the allowed source set. Every number it asserts appears in the source text it cites.
- Output constraints. Everything the model selected exists in the catalog it was choosing from.
- Configuration honesty. If the prompt version, model configuration, or recorded fixtures do not match the baseline, the run fails before a single metric is computed.
Two design rules make these checks trustworthy. First, never trust the pipeline's own metadata about itself: the eval re-derives the allowed sets and re-proves the grounding independently, because a bug that breaks the pipeline's citation logic would break its self-reporting the same way. Second, this is not an anti-LLM-judge position; it is a division of labor. Judgment calls (is this evidence good enough?) get measured statistically and reported. Decidable properties (does this citation resolve?) get proven, and proof can gate. That is the same discipline as the deterministic review gate in how we shipped a production app in days with AI coding agents. This post is about the layer that gate cannot reach.
Statistical metrics still matter; they gate differently, through two instruments that cover each other's blind spots:
- Coarse absolute tripwires, set outside the noise band, that catch catastrophe.
- A delta gate on rebaselining: the command that writes a new baseline refuses to bless metrics worse than the committed ones. The tripwire catches disasters; the ratchet catches erosion, because a metric can decay meaningfully while staying comfortably under any ceiling wide enough to survive the noise floor.
One more thing worth deciding early: know your asymmetry. In most systems, one error direction is expensive (a missed real issue) and the other is cheap (a false alarm a human dismisses in minutes). Pick the metric that guards the expensive direction, protect it hardest, and accept looseness in the cheap one. Trying to optimize both at once, inside the noise floor, is how teams quietly trade the harmless failure for the harmful one.
Lesson 7: Ground-truth labels are the hard 80% of LLM evals
Every eval conversation eventually arrives at the same place: the ground-truth labels are the hard part. Four constructions to check for mechanically, because each one produced a wrong conclusion before we caught it:
The unwinnable question. One case had a ground-truth answer that was not in the candidate set the model was shown. Recall on that case was zero by construction, and it looked exactly like a prompt failure until we traced it. Labels can only demand answers the system was actually offered. We now have a data test that fails the build if any expected answer falls outside its case's own candidate set: treat your labels like code, with tests.
The arithmetically impossible label. One truth row expected ten answers from a prompt that caps its output at eight. Full recall was impossible before the model said a word. Labels have to respect the constraints you imposed on the system yourself.
The case that should not be labeled. One case got two substantially different, individually defensible answer sets across two runs, over a space full of near-synonyms. We left it deliberately unlabeled: it still runs and still gets its deterministic checks, but it contributes no accuracy score, because forcing a single "correct" answer would inject noise into the aggregate rather than measure anything. An honest null beats a false certainty.
The self-graded label. Some labels start life seeded from the system's own earlier outputs, human-reviewed but self-derived. Those must stay report-only forever; a system must not grade itself into a merge gate. Only labels authored independently of the model (from source documents, by a human) earn the right to block merges. Labels earn the right to gate; they do not get it by existing.
And be honest about what your exam can see. Our hardest eval presents the model with a 45-option candidate set; production presents 835. Fewer distractors flatter precision, so the eval's precision is documented, inside the harness itself, as an upper bound on production precision, not a measurement of it. Writing that sentence down cost nothing and has prevented the number from being misused ever since.
Lesson 8: A small honest eval beats a big aspirational one
The eval that taught us everything above is small: 9 scenarios, 128 labeled rows, 25 recorded model calls. (The two-run experiment in lesson 1 covers 124 of those rows; the other 4 are a hand-authored synonym-recognition case that pins a specific behavior rather than scoring accuracy.) It caught real regressions, killed phantom ones, and settled prompt arguments that would otherwise have been decided by whoever argued longest.
Teams postpone evals because they imagine the destination is a thousand-case benchmark with statistical power. That framing gets it backwards. The value came from the discipline around the numbers (noise measured, baselines pinned, labels audited, gates matched to what they can actually decide), not from the size of the set. A small suite with those properties tells you true things about a narrow slice. A large suite without them tells you confident falsehoods about everything.
Start small, write down what the eval cannot see, and grow it only where a new case earns its place by covering something no existing case covers. "We don't have a benchmark yet" is not a reason to ship blind.
The playbook: how to set up LLM evals in CI, in order
If we were standing up evals on a new LLM product tomorrow, in order:
- Put a recordable seam at your model boundary. One interface, all calls through it. Record real responses; replay them through your real production code in CI.
- Measure your noise floor before anything else. Run your identical configuration at least twice, live. The spread you see is a lower bound, and until you know it, no delta means anything.
- Gate merges on decidable properties. Structure, grounding, constraints, config pinning. Deterministic checks block; statistical metrics report.
- Set tripwires outside the noise band, and add a delta gate. Absolute ceilings catch catastrophe; a rebaseline that refuses to bless a worse number catches erosion.
- Store per-item outcomes and diagnose by identity, not count. Stable blind spot: regression. Rotating misses: noise.
- Re-record only when the model's inputs changed. Otherwise replay, and get exact attribution for free.
- Audit your labels like code. No answers outside the candidate set, no labels that violate your own output constraints, no forced labels on genuinely ambiguous cases, no self-graded labels behind a blocking gate.
- Document what the eval cannot see. Every honest limitation you write down is a number that cannot be misused later.
And if you are not building this yourself but buying AI work, the list converts directly into vendor questions: What is your run-to-run spread, and are your quality bars outside it? Can you run your quality checks without calling the model? When a metric moves, can you tell a model regression from a code regression? A team with real answers is engineering the system. A team that has never measured its own noise floor is demoing it.
Frequently asked questions
Are LLM outputs deterministic at temperature 0?
No. Even at temperature 0, LLM providers do not guarantee reproducible outputs (batching, hardware, and routing effects all leak in), and most production pipelines run with sampling anyway. In our own measurement, two runs of an identical prompt disagreed on about 9% of 124 predictions.
How do you run LLM evals in CI without flaky builds?
Record real model responses once, then replay them through your real production pipeline on every pull request. The run is deterministic, offline, and free, so CI never depends on a live model call. Keep live-model evals on a schedule or on prompt changes, where statistical thresholds and human review apply.
What is a noise floor in LLM evaluation?
An LLM evaluation noise floor is the disagreement a system shows between runs of an identical configuration: same prompt, same model, same inputs. Any eval delta smaller than the noise floor cannot be attributed to your change. Measure it by running the identical configuration at least twice and diffing item-level results.
When should you re-record LLM eval fixtures instead of replaying them?
Re-record when the model's inputs changed: a prompt edit, a model upgrade, new input data. Replay when only deterministic code downstream of the model changed; replaying measures that change in isolation at zero API cost and without re-rolling sampling variance.
How many eval examples do you need for a reliable merge gate?
Enough that the gate's margin exceeds your measured noise floor for that metric. Small label sets support deterministic checks and coarse tripwires, not tight accuracy thresholds. Our own suite is 9 scenarios and 128 labeled rows, and it still caught real regressions. The discipline matters more than the size.
Should you build or buy an LLM eval framework?
We built ours in-house: plain TypeScript around the model provider's SDK, schema-validated JSON fixtures, and the test runner we already had. Buy tooling if it saves time, but own the methodology. The harness is the easy 20%; measuring your noise floor, auditing your ground-truth labels, and deciding which gates may block a merge are the hard 80%, and they stay your responsibility with any vendor.
Ready to ship AI features you can actually measure?
First Mate builds production AI systems with the eval infrastructure attached, so you know when they get better and, more importantly, when they get worse. See how First Mate staffs product-minded AI engineers, or start a low-risk two-week trial.
