Evaluating agents with simulation: how we test a Procedure before it talks to a customer

A support team changes a Procedure in Pazau about every 40 minutes, averaged across our customer base. Someone tightens the refund window, adds a step for gift returns, or rewrites the tone for a market. Each edit is a change to production software that talks to thousands of customers a day. Most of those edits are made by operations analysts, not engineers.

Traditional software gets unit tests, integration tests and a CI pipeline. For most of the history of AI support tools, a prompt change got a few manual test chats and a hopeful deploy.

We think that's the biggest reliability problem in the category, so we built a simulator. Before a Procedure change is published, Pazau replays it against real historical conversations from that workspace: up to 500 in the preview that opens today, and up to 2,000 for our early-access customers. It grades the results and shows the author exactly what would change. This post explains how the simulator works.

If you haven't read Procedures, not prompts, start there. It explains why we structure agent behavior as Procedures in the first place, and that structure is what makes simulation possible.

The core loop

Simulation runs in four stages:

1. Sample historical conversations relevant to the change.
2. Replay each one with a simulated customer against the new Procedure version.
3. Mock every Action call with recorded or synthesized responses.
4. Grade the outcome against a rubric, and compare it with the same replay against the current published version.

The output isn't a single score. It's a diff: how many conversations changed outcome, in which direction, and why.

Stage 1: sampling

Replaying every historical conversation would be slow and expensive, and most of them would be irrelevant. A change to the gift-returns step doesn't need 40,000 WISMO conversations.

We sample in three layers:

• Relevance filter. Using the Procedure's trigger conditions and topic classifier, we find conversations from the last 90 days that the current or new version would have handled. For a narrow change, this is often a few hundred conversations. For a broad one, it's tens of thousands.
• Stratification. We split that set by outcome (AI-resolved, handed off, repeat contact, negative CSAT), channel, language and the branch of the Procedure taken. Then we sample proportionally, with minimums per stratum. Rare paths are the ones most likely to break, so each branch gets at least 20 conversations, or every conversation it has if there are fewer than 20.
• Pinned regressions. Any conversation a human QA reviewer has flagged as "must not regress" for this Procedure is always included. Teams build this set up over time, and it works like a golden test suite.

The cap is 2,000 conversations for early-access workspaces (500 in the preview). In practice, the median early-access simulation uses 640.

# simulation plan generated for a Procedure change
procedure: returns.gift_receipt
change: v14 -> v15
window: 90d
candidates: 3,812
strata:
  - { outcome: ai_resolved,  channel: chat,  n: 180 }
  - { outcome: ai_resolved,  channel: email, n: 110 }
  - { outcome: handed_off,   channel: chat,  n: 140 }
  - { outcome: repeat_7d,    channel: any,   n: 60  }
  - { branch:  store_credit_over_limit,      n: 24  }  # rare branch, min 20
  - { lang:    fr-CA,                         n: 20  }
pinned_regressions: 31
total: 565

Stage 2: replay with a simulated customer

You can't just feed the old customer messages into the new Procedure. If the new version asks a different clarifying question, the recorded customer reply no longer fits. Replaying a fixed transcript tests the old conversation, not the new one.

Instead, we build a customer persona from each historical conversation. That's a structured description of who the customer was and what they wanted:

{
  "goal": "Return a gift (a men's shell jacket, size L) for store credit",
  "known_facts": {
    "gift_receipt_code": "GR-88213-K",
    "order_number": null,
    "buyer_email": null
  },
  "disposition": "polite, brief, mildly impatient",
  "constraints": ["does not know who bought it", "on mobile"],
  "would_accept": ["store credit", "exchange for size M"],
  "would_escalate_if": ["asked for buyer's email twice"]
}

A separate model plays the customer. It pursues the goal, reveals only facts the real customer had, and reacts to the agent with the recorded disposition. When the new Procedure asks something the historical customer was never asked, the simulated customer answers in character from its known facts or says it doesn't know.

The persona model runs with a different system prompt, and often a different model family, from the agent. We don't want the agent and the customer to share blind spots.

Stage 3: mocking Actions

Simulations must never touch production systems. A replayed refund can't issue a real refund.

Every Action call in simulation goes to a mock layer that resolves in this order:

1. Recorded response. If the historical conversation made the same Action call with equivalent arguments, we return the recorded response. This covers about 70% of calls.
2. State-derived response. If the call is new but reads state we captured at conversation time, such as order details or inventory levels, we synthesize the response from that snapshot.
3. Schema-derived response. Otherwise, we generate a plausible response from the Action's schema and mark the conversation low-confidence so it's weighted accordingly in the results.

Write Actions such as issue_refund or create_exchange_order always return a synthesized success or failure. Limits and approval rules still apply in simulation. If the new Procedure would try a $400 refund against a $250 limit, the simulation records that it would have gone to the approval queue.

Stage 4: grading

Every simulated conversation is graded twice: once against the current published Procedure, and once against the proposed one. The grader is a judge model with a rubric, and the rubric is specific to the Procedure, not generic.

Criterion	What it checks	Type
Goal completion	Did the customer's goal get met, or correctly declined per policy?	Pass / fail
Policy adherence	Did every step match the Procedure, including eligibility checks?	Pass / fail per step
Action correctness	Right Action, right arguments, within limits	Pass / fail per call
Escalation correctness	Handed off when it should have, and only then	Pass / fail
Data exposure	Revealed nothing the customer wasn't entitled to see	Pass / fail, blocking
Tone	Matched the workspace's voice guidelines	1–5
Efficiency	Turns to resolution vs. the historical baseline	Numeric

Data exposure is blocking. If a single simulated conversation reveals, for example, a gift buyer's name to the recipient, the change can't be published until it's fixed. No aggregate score overrides it.

Is the judge right?

A simulator is only useful if its grades agree with a careful human. We calibrate continuously.

Every week, our QA partners and customers' own QA teams grade a random sample of simulated conversations blind, without seeing the judge's scores. Across the last quarter, 11,400 conversations were double-graded this way:

Criterion	Judge–human agreement (Cohen's κ)
Goal completion	0.93
Policy adherence	0.91
Action correctness	0.95
Escalation correctness	0.88
Data exposure	0.97
Tone	0.74

Overall agreement on pass/fail criteria is 0.91. For reference, two trained human QA reviewers grading the same conversations agree at 0.86. Tone is the weakest criterion, which isn't surprising, so it never gates a publish on its own.

When agreement for a criterion drops below its threshold in a given workspace, we review that workspace's rubric. Usually the fix is a clearer rubric line. Occasionally it's a judge prompt change, and that change goes through its own simulation against the calibration set.

Regression gates

The author sees a result like this:

returns.gift_receipt  v14 → v15     565 conversations simulated

  Improved      83   (handed off → resolved: 71, repeat → resolved: 12)
  Regressed      4   (resolved → handed off: 3, policy step skipped: 1)
  Unchanged    478
  Blocking       0

  Projected AI resolution for this Procedure:  41.2% → 55.9%  (±3.1)
  Median turns to resolution:                    7 → 5

Each regressed conversation is one click away, with a side-by-side view of the old and new transcripts and Traces.

Default gates, which admins can make stricter but not looser:

• Any blocking failure means the change can't be published.
• Net regression (more regressed than improved) requires a second approver.
• More than 2% of conversations regressed requires the author to acknowledge each regression.
• Low-confidence share above 30% shows a warning that the sample may not reflect the change well.

Cost and speed

Simulation only helps if people actually run it, and people only run it if it's fast. The median simulation finishes in 4 minutes 10 seconds. The p95 is 9 minutes, for a full 2,000-conversation run. We parallelize across conversations, cache recorded Action responses, and run the persona and judge on smaller, faster models than the agent itself.

Pazau absorbs the compute cost, which averages about $1.80 per simulation. It isn't billed and doesn't count as resolutions. We think of it the way we think of CI minutes: it's cheaper than the incident.

What we've seen

Since March, 40 early-access customers have required a simulation before every publish. Across those workspaces:

• Procedure-related incidents (a published change causing a spike in repeat contacts or escalations) fell 78%.
• Median time from edit to publish went down, from 2.6 days to 4 hours. Teams used to hold changes for a weekly review meeting. Now the simulation result is the review.
• 31% of simulations are abandoned without publishing. Each of those is a change that would have gone live in the old world.

Simulation is in preview for every workspace starting today, and it will be a core part of the next major release of Pazau Agent. See Pazau Agent for how it fits together.

We're hiring in evaluation. If building the test infrastructure for AI that talks to 140 million people a year sounds interesting, we'd like to hear from you.