How We Test AI-Generated Code Before It Ships

Most of QueueMate's code was written by AI agents. It's a free restaurant queue system we built in days in June 2026, and restaurants run real dinner services on it. So people keep asking us the same question: who checks the code?

Not the agent that wrote it. A different model designs the test cases, tries to break the build, and reviews every diff, and nothing merges until it signs off. On QueueMate, that checker wrote 554 test cases and filed 38 defects, including a severity-1 bug where a session-refresh file had never compiled or run at all. This post walks through the whole workflow with the actual artifacts, because the product is ours and we can show them.

The short version: how we test AI-generated code

To test AI-generated code, separate the checking from the doing: the model that writes a feature never judges it. A second model designs the test cases, weighted toward edge, negative, boundary, and concurrency paths. They run against an isolated stack on fresh data, findings are filed before any fix, and an independent review gates the merge.

• The agent that builds a feature never checks its own work. A different model plans the feature, designs the test cases, and reviews the code. A human engineer directs both and owns the result.
• On QueueMate, the checking model designed 554 test cases against the built product. Only 49 were happy path. Edge, negative, boundary, and concurrency cases outnumbered them six to one, because those are the cases humans forget to write down.
• Tests execute against an isolated environment per branch: its own app instance, its own Docker-backed database stack on its own ports, mock SMS and email gateways, and fresh data provisioned for the run.
• All 38 defects were filed before a single fix was applied, so every test case ran against the same version of the code.
• Nothing merges until the test cases have been executed and an independent review is clean, and neither one can stand in for the other.

The rule everything hangs on: the builder never grades its own work

Our workflow assigns different AI models to different roles, the way a real engineering team separates duties. One model owns checking: it plans the feature, designs the test cases, and reviews the finished code. A different model owns doing: it writes the implementation and executes the tests it is handed. The two never swap hats on the same piece of work.

Role	Owned by	Why it is separate
Feature planning, test-case design, code review	One model (the checker)	Designs the exam and grades the work. Never writes the implementation.
Code development, test execution	A different model (the builder)	Writes the code and runs the checks it is handed. Never decides what "correct" means.
Judgment calls, final sign-off	A human engineer	Directs both, owns the result, and breaks ties the models cannot.

A fresh chat session wouldn't be enough, because a model reviewing its own output carries the same blind spots into the review. If it misread the requirement while building, it misreads it the same way while checking, and agrees with itself. An independent checker never knew what the builder intended, so it can only read what the code actually does. Most of the bugs we catch live in that gap.

We're not the only ones who landed here. Spotify's chief architect, Niklas Gustavsson, has talked about the same pattern at their scale: their background coding agent hovered around a 20 to 30 percent success rate until an independent judging step was added to its loop, after which it reached roughly 80 percent. They've since retired that particular judge as the models improved, which if anything makes the point better. The mechanism changes. The need for someone other than the author to check the work doesn't.

Every feature moves through the same five stages, and each stage is a hard gate:

The rule that gives the gates teeth comes straight from the project's own workflow doc: a failing stage bounces the work back to development and then re-runs, and findings are never carried forward as follow-up tickets. "Cleared to merge" means the test cases were executed and the code review approved. Nothing merges on one of the two.

A single engineer directs this whole apparatus. If you want the org-chart view of how one person runs a team of specialized agents, we wrote that up separately in how one engineer built a full MVP with a team of AI agents. This post stays on the checking side.

Step 1: a model designs the QA test cases, and it is told what not to write

After a feature is built, the test-designer agent gets two inputs: the acceptance criteria, and a statement of everything the automated test suites already cover. Then it's given one instruction that does most of the work. Here's the actual preamble from one of QueueMate's generated case files:

**Already covered by automation — do NOT merely restate (vary or go deeper):**
- Pure (src/lib/queue/combine-suggestion.test.ts): ranking (fewest tables >
  tightest fit > id determinism), group discipline (no cross-group, no
  ungrouped members), silence when any single free table fits, the 4-table
  cap, FCFS assignment across the queue, no table promised twice, ...
- Board projection (...): occupied/held/already-combined tables excluded,
  longest waiter wins, ...
- Component (...): badge text, silence, one-tap joins+seats, SR announcement
  fires, ...

"Do NOT merely restate (vary or go deeper)" is the mechanism behind the claim that models find the edge cases humans forget. The designer is handed the happy path as already covered and is forced past it by construction. The first case it produced for that feature went after a gap: F-001, a three-table join tested end to end, with the designer noting that "automation only ever did two." Every automated test had combined two tables. Nobody had ever tried three. The model noticed because looking for gaps was the only move left to it.

That process runs at two scales: a per-feature pass while the product is being built, and a whole-product audit sweep afterward. QueueMate's audit sweep, run against the shipped build about a month after launch, produced 554 cases in seven suites:

Suite	Cases	What it goes after
A. Auth, venue lifecycle, security	185	Session handling, ownership guards, injection attempts, enumeration leaks
B. Guest journey	65	QR join, live status, malformed tokens, flaky-network behavior
C. Host board	105	Seating flows, undo races, multi-tab interleavings
D. Reservations, notify, purge	62	Timezone boundaries, notification gateways, the automated PII purge
E. Marketing, SEO, accessibility	98	Public pages, structured data, keyboard and screen-reader paths
F. Table-join suggestion	32	A later feature's delta sweep: deeper cases than its unit tests
G. Real-provider gate	7	Proving test runs can never reach live email or SMS providers

Happy-path cases, the kind that confirm the feature works when everyone behaves, are 9% of the suite:

What do the other 91% look like? Here's a P0 edge case from the reservations suite. QueueMate runs in the Philippines, so the designer went straight for the place timezone bugs hide: bookings on either side of midnight, where the calendar date in UTC disagrees with the calendar date in Manila.

### D-013 — Manila midnight day-boundary: 23:55 vs 00:15 grouping
- **Priority:** P0
- **Type:** edge
- **Preconditions:** Host browser/OS timezone set to Asia/Manila (the normal
  case); venue timezone Asia/Manila.
- **Steps:** 1. Create reservation "LateNight" for **today 23:55**.
  2. Create "EarlyBird" for **tomorrow 00:15**. 3. Inspect the Today and
  Upcoming tabs and their header counts. 4. SELECT the stored rows — confirm
  the UTC instants are 15:55Z today and 16:15Z today (Manila +8: 00:15
  tomorrow Manila = 16:15 UTC **today**).
- **Expected:** LateNight groups under **Today**; EarlyBird groups under
  **Upcoming → "Tomorrow"** — even though EarlyBird's UTC date is today's.
  This is the classic UTC-date-vs-venue-day defect probe: any appearance of
  EarlyBird under Today (or LateNight under a wrong day) is a FAIL.

The case doesn't just click through the UI. It pins down the exact UTC instants that should be sitting in the database and names the defect class it's hunting. A QA engineer with years of timezone scar tissue writes cases like this. Most teams don't have that person, and the teams that do can't get 554 cases out of them.

The concurrency cases go after the same class of thing, the interleavings nobody tests by hand:

### C-048 — Undo in tab A after tab B already seated the (restored) flow
- **Priority:** P1
- **Type:** concurrency
- **Preconditions:** Two host tabs. Party "Cross" waiting.
- **Steps:** 1. Tab A: remove "Cross" (undo toast up). 2. Tab B (before its
  refetch lands — act fast, or briefly offline tab B): tap **Seat** on
  "Cross". 3. Observe both tabs after settle.
- **Expected:** Tab B's seat fails ... and its optimistic seat ROLLS BACK:
  the table returns to free, "Cross" does not appear seated. Tab A may still
  Undo → a NEW equivalent row appears in both tabs. No state where the
  cancelled party occupies a table.

Two browser tabs racing each other through a remove, an undo, and a seat, with the network deliberately interrupted mid-race. The expected outcome is specified down to which toast appears and which optimistic update must roll back. There are 32 cases like this in the suite.

The generated cases also know when to hold back. When the model isn't sure whether a behavior counts as a bug, it says so instead of guessing: a case about a host re-adding a guest during an undo window ends with "record whether the UI gives the host any cue", filed as a product question rather than a defect. Ambiguity goes to humans. That restraint is what keeps the suite useful instead of noisy.

Step 2: execute against an isolated stack, on fresh data

Test cases are only as trustworthy as the environment they run in. Two rules govern ours.

Rule one: every branch gets its own world. Each unit of work happens in its own git worktree, and each worktree runs its own app instance plus its own database stack: Postgres, realtime, auth, and a local mail catcher, all running as Docker containers with a unique project id and their own port block. The QA sweep ran in a dedicated worktree on its own ports, with mock SMS and mock email gateways, while development continued on other branches in parallel. A destructive test wipes its own world. It holds no credentials for another branch's stack and none for production, so there is nothing else within its reach.

None of this is glamorous. Before any destructive database command, the agent has to check the config in its own working directory and prove it's pointing at its own stack rather than a shared one. When a branch merges, its stack gets torn down before the worktree is removed, while the project id is still unique. Rules like these do nothing on a good day, which is exactly why they're written down.

Rule two: fresh data, real browser, production build. Browser tests provision their own venues, parties, and reservations through the app itself, never a pre-seeded showcase dataset, because seeded data hides exactly the lifecycle bugs the cases are hunting. And end-to-end runs drive a production build of the app rather than the dev server, a rule that exists because a QA pass caught the dev-mode toolbar silently eating clicks on mobile buttons and failing two tests for reasons that had nothing to do with the product. Production builds have no toolbar, so rather than teach the tests to click around it, we removed the hazard entirely.

Isolation is also the honest answer to a question founders should ask any team using AI agents: what stops a test run from touching the real world? We learned this one the hard way, and the repository says so in plain sight. Early on, an end-to-end run attached to an already-running dev server whose local config still held real API keys. The inquiry-form spec mailed our own live inbox, and the guest-notify specs mailed fabricated example.com addresses that bounced. No customer ever received anything, but it was a real hole. The comment now sitting in QueueMate's Playwright config tells the story:

// NEVER let E2E reach a real provider. `...process.env` above carries
// whatever is in .env.local, and a developer's real RESEND_API_KEY there
// means the inquiry-form spec SENDS ACTUAL EMAIL to the live inbox — and
// the guest-notify specs mail fabricated @example.com addresses, which
// bounce and damage the sending domain's reputation. Both happened.
// Blanking these makes `emptyStringAsUndefined` (src/env.ts) resolve them
// to undefined, so the gateway factories return their Mock adapters.

The real fix went deeper than the comment: the application itself now refuses to construct live email or SMS providers unless it is genuinely running in production. The guard lives in the app, not the test harness, so a stray key in someone's local env file no longer reopens the hole. And the incident itself became test suite G: seven generated cases whose only job is to prove, on every sweep, that the guard still holds.

Step 3: findings are filed first, fixed second, and turned into regression tests

The execution phase has one discipline that sounds bureaucratic and turns out to be the whole ballgame. From the top of QueueMate's findings register:

All findings are filed here FIRST. No fixes are applied until every designed test case has been executed.

Fixing as you go contaminates the sweep: change the code mid-run and every case executed before the change tested different software than every case after it. Filing first also leaves behind an honest artifact, a severity-rated register of everything wrong at one fixed commit, which the fix work can then be planned against and measured by.

Here's what the full sweep produced on QueueMate, in numbers:

Measure	Count
Generated QA cases designed against the built product	554
Rated P0 (must pass before anything ships)	169
Edge, negative, boundary, and concurrency cases	291
Happy-path cases	49
Executed results: pass / fail / blocked / recorded N/A	~460 / ~43 / ~26 / ~25
Unique defects filed from the sweep	38
Defects fixed in the follow-up commit	27
Committed automated tests (unit + component), August 2026	~1,060 across 100 files, roughly double what shipped at launch
Committed end-to-end scenarios	97, each run on desktop and mobile profiles

The follow-up commit fixed 27 of the 38, dependency-ordered so root causes landed before symptoms. The written fix plan triaged the rest by name: six flagged for more careful sequencing, two deliberately deferred, one ruled working as intended. Those defects are the receipts for the whole process, so it's worth looking at what kind of bugs they were.

The worst one was invisible to every demo. A file responsible for refreshing user sessions sat one directory too high in the repository, where the framework never loads it. It had never compiled. It had never run. Sessions were quietly expiring early on a product that demoed perfectly, and four other filed findings turned out to be downstream symptoms of this one root cause. It's also a fair illustration of what per-feature checking can't do: every feature had passed its own gates while this sat underneath them all. Catching it took the whole-product sweep, designed by a model that had no hand in building any of it.

Real bugs got found more than once, independently. Suites are executed by separate agent runs that can't see each other's results. One defect, the login flow dropping the address you were trying to reach so you always landed on the dashboard instead, was filed independently by three different executors:

Independent rediscovery is the part of this setup we trust most. When three runs that share nothing but the codebase converge on the same defect, environment flukes and agent hallucination are off the table, and severity gets raised with confidence.

The sweep audits itself, too. The same pass that filed product defects also filed a process defect: the committed end-to-end suite was failing on the main branch with no product changes applied, which contradicted the project's own "done means green" bar. It would have been easy to shrug that off as expected noise. It was filed like any other finding. And when the checking model was later pointed at the QA artifacts themselves, it filed corrections against them, including a findings table whose bookkeeping disagreed with its own heading.

The last step is turning the interesting findings into permanent regression tests. This spec now lives in the committed end-to-end suite, generated from two findings about the guest's session token:

// M9 E2E — guest session-token edge cases (QA sweep: B-F-001, B-F-002).
//
// The guest's ONLY credential is the opaque `sessionToken` embedded in
// `/s/[token]`. These specs prove two boundary behaviours around that token:
//   - B-F-001: a malformed token (a NUL byte) must collapse into the same
//     generic "unknown token" outcome as any other unknown token — never a
//     500 — for BOTH the JSON status route and the guest page itself.
//   - B-F-002: once a party's status is genuinely terminal (seated), a guest
//     tab left open must stop polling `/api/status`.

test('GET /api/status/%00 resolves the generic 404, not a 500', ...)

That second behavior is a nice example of proving a negative: the test waits out the full polling interval and asserts that a finished guest tab makes zero further network requests, demonstrating the polling actually stopped rather than merely not having ticked yet. A bug found once in this codebase now has a test standing over it.

What actually blocks a merge in our AI code review workflow

It's worth being precise about which of these checks are hard automated gates and which are process we hold ourselves to, because teams blur that line all the time.

• On every push, CI blocks on the deterministic floor: type checking, linting, the unit and component suites (about 1,060 tests), and a full production build. Red means no merge, no exceptions.
• Before any merge, the agent loop enforces the rest: the end-to-end suite against an isolated stack, the QA case execution, the browser pass on freshly provisioned data, and code review by the checking model plus an independent second reviewer. Reviewers have to report the false positives they checked and cleared alongside their findings, which forces them to actually read the code rather than pattern-match on it.

The project's own workflow doc puts the underlying belief bluntly: tests alone are not verification. A green suite only proves the code does what the tests describe. Someone still has to ask whether the tests describe enough, whether the code should exist in that shape at all, and whether a human can actually use the thing. Those are the QA pass, the review, and the browser pass, and none of them can substitute for another.

Everything above assumes correct behavior is decidable, that a case either passes or fails. When the feature itself contains an LLM, output varies from run to run and pass/fail gates stop being honest, which is a different problem needing a different instrument. We wrote that one up in how we run LLM evals in CI without flaky gates.

What this costs, and why it is the product

None of this is free. The checking half of the loop, the 4,700 lines of generated case files, the findings register, the isolated stacks, the review gates, consumed real engineer and agent time on a build whose entire pitch was speed. Some of our engineers' most valuable output is now the code and artifacts that never ship: the harness an agent's work has to pass through before it counts.

We pay it anyway. AI agents made writing code cheap, so the code itself is no longer where most of the risk sits. The risk moved into verification, into understanding what was built well enough to stand behind it. Speed without a harness produces demos. The harness is what produces software you can run a dinner rush on, and it's most of what you're actually hiring when you hire AI software engineers rather than renting a code generator. QueueMate went through it and is live today at queuemate.tech, free for restaurants.

None of it gets us to zero defects either, and any team that tells you their process does is selling you something. What it changes is which defects survive. The ones that reach production are the ones nobody had a reason to look for, not the ones an independent checker would have caught in an afternoon.

If you're hiring anyone to build with AI agents, ask them four questions. Who checks the agent's work, and is it the same model that wrote it? What do the tests look for besides the happy path? Where do test runs execute, and can they reach production or real customers? What blocks a merge, and is it enforced or just agreed?

If you're weighing an AI-accelerated build and want this level of rigor on your product, start a low-risk two-week trial. We will show you the harness before we show you the demo.

Frequently asked questions

How do you test AI-generated code?

Separate the checking from the doing. A different model from the one that wrote the code designs the test cases from the acceptance criteria, deliberately weighted toward edge, negative, boundary, and concurrency cases. The cases execute against an isolated environment with fresh data, findings are filed before anything is fixed, and an independent code review gates the merge alongside the test results.

Can an AI agent write tests for its own code?

Mechanically yes, and those tests have value as regression protection. But an agent testing its own work inherits its own misunderstandings: if it misread the requirement while building, its tests encode the same misreading. That's why our test designer is a different model that never sees the builder's intentions, only the acceptance criteria and the finished product.

How do you stop AI agent test runs from touching production data?

Each branch of work runs its own copy of the application and its own Docker-backed database stack (Postgres, realtime, auth, mail catcher) with a unique project id and port block, plus mock SMS and email gateways. Tests provision their own data through the app. A destructive test can wipe its own world and nothing else: a test run holds no credentials for another branch's stack, and the app refuses to construct live email or SMS providers outside production.

Doesn't all this testing cancel out the speed of AI development?

It costs time, and it's still fast: QueueMate went from kickoff to a live production app in days with the per-feature loop applied throughout, and the deeper whole-product sweep ran afterward as a scheduled audit, without pausing development. Isolation is part of the answer, since parallel worktrees let QA sweeps run while development continues on other branches. The alternative is faster demos and slower products, because unverified defects surface after launch, where they're most expensive.

What does an AI code review workflow look like?

Two independent passes, both required. The checking model reviews a diff it never wrote against the acceptance criteria and the codebase's own conventions, and a second reviewer runs the same pass independently. Reviewers report the false positives they checked and cleared, not just their findings, which keeps the review rigorous instead of performative. A human engineer makes the judgment calls and owns the merge.

What happens when a test case fails?

The finding is filed in a register first, with severity and evidence, and no fixes are applied until every designed case has been executed, so the whole sweep tests one consistent version of the code. Then fixes land in dependency order, root causes before symptoms, and the notable findings are converted into permanent automated regression tests.

Do humans still review the code?

Yes. A human engineer directs the whole loop, makes the judgment calls the models cannot, and owns the result. The model review gate is in addition to that, not instead of it: a mandatory adversarial pass by the checking model plus an independent second reviewer before anything merges.