Testing LLM Applications: What "Functional" Means When the Response Isn't Deterministic
expect(response).toBe("Your order arrives in 3 to 5 business days") used
to be the most boring, reliable assertion in your test suite — until that
endpoint started answering with an LLM. Now the same test fails eight times
out of ten, not because the feature is broken, but because the model
returned “Your order will arrive within 3-5 business days,” a perfectly
correct answer that isn’t the exact string you wrote three weeks ago.
The feature works. The test has no way to say so. And that’s exactly the problem almost no QA team has solved yet.
What “functional” means when the response isn’t deterministic
It means replacing exact-match comparison with an eval: a test that applies grading logic to the model’s output to decide whether it meets a success criterion, instead of requiring it to match an expected output character for character. That’s how Anthropic defines it in its guide to evaluating AI agents: give the system an input, then apply grading logic to its output to measure success.
That shift in paradigm brings new vocabulary worth adopting as-is, because it’s what you’ll need to speak the same language as any team already testing LLMs in production:
- Task (or test case): a test with defined inputs and a success criterion — not necessarily a single correct answer.
- Trial: each attempt at that task. Since the model isn’t deterministic, you run several trials of the same case to measure consistency, not a single pass/fail.
- Transcript (or trajectory): the complete record of a trial — outputs, tool calls, intermediate reasoning, partial results.
- Grader: the logic that scores some aspect of the system’s performance on that transcript.
- Outcome: the final state in the environment at the end of the trial — the order was actually placed, the ticket was actually closed, the file was actually generated correctly.
With that vocabulary, the question stops being “does the output match what we expected?” and becomes “does the grader we chose measure what actually matters?” — which is a test-design question, not a string-comparison one.
Why most teams get this wrong
The most common mistake isn’t skipping tests for LLM features — it’s
testing them with the same binary criterion as always, forcing the model’s
response into an assertEquals that was never built for generated text.
The typical result: the suite stays permanently red, and someone eventually
deletes the test “because it always fails,” losing any real coverage on
that feature.
The opposite mistake is just as common and more dangerous: loosening the
test until it always passes — an assert len(response) > 0 dressed up as
coverage — without checking whether the response actually solves what the
user asked for. Anthropic sums up the fix in a line that applies just as
well to functional testing as it does to performance testing: no single
evaluation layer catches every issue; combine several methods and whatever
slips through one layer gets caught by another. It’s the same
defense-in-depth principle you already apply to performance and security,
applied to functional correctness.
Why this is your problem now, not in two years
Almost every product you’re building or maintaining today has, or will soon have, some feature answering through an LLM: support, search, summaries, content generation, internal agents. Most QA teams haven’t updated their functional-testing playbook for that yet — they keep writing exact-match assertions against a system that’s no longer deterministic, or they simply don’t test those flows because “you can’t.”
That’s a real window of opportunity: whoever solves a serious evaluation criterion for these features first ships with far less friction than whoever keeps fighting broken tests or, worse, ships without real coverage at all. In regulated industries (banking, healthcare, insurance) this isn’t optional — you need to show auditors a documented acceptance criterion, not “we eyeballed it.”
How we evaluate it
The flow we follow, adapted from Anthropic’s evaluation framework for AI agents, has five steps:
Flow adapted from “Demystifying evals for AI agents,” Anthropic Engineering Blog.
The grader is the piece that decides everything. Anthropic identifies three types, and in practice we combine all three depending on the feature:
| Grader type | When we use it | Strength | Limit |
|---|---|---|---|
| Code-based | State verification, pattern matching, static analysis | Fast, cheap, reproducible | Brittle against valid variations |
| Model-based (LLM-as-judge) | Quality rubrics, pairwise comparison | Scalable, captures nuance | Non-deterministic, needs human calibration |
| Human | Ambiguous cases, periodic judge calibration | Ground-truth reference | Expensive and slow to run on every build |
One detail almost nobody measures, straight from Anthropic’s guide: pass@k (the probability that at least one of k attempts is correct) and pass^k (the probability that all k attempts are correct) tell opposite stories. For an internal tool where it’s enough to work once, pass@k is the right metric. For a customer-facing feature, where the user expects consistency every single time, pass^k is the one that matters — and it’s usually much lower than the team assumes.
On the projects where we apply this, the real work isn’t writing the first eval — it’s what Anthropic calls “reading transcripts regularly”: when a trial fails, someone has to decide whether the model made a genuine mistake or the grader rejected a valid answer. That judgment doesn’t automate itself; it’s exactly the kind of test-design work we’ve been doing for years, applied to a new kind of system.
Does your product already ship an LLM feature without a clear acceptance criterion?
Let’s talk about designing the graders and eval suite for that feature, with a team that has done it before.