TL;DR
Ben Hylak, CTO and co-founder of Raindrop, walks Hamel Husain through his method for testing agent changes before they ship: replay real past production traces in a rebuilt copy of the world as it was, then diff the results. The 9-step method covers capturing traces, rebuilding the world, proving the simulation is faithful, diffing first with cheap deterministic checks, then small yes/no judges — with starter prompts and a diff script included.
The idea in one paragraph
Old-style evals were ‘fixed prompt in, expected answer out.’ Agents broke that model: adding or removing a tool, renaming a tool field, swapping the model, changing memory or subagents, or switching harnesses can all change behavior, and most real failures are ones you never wrote an eval for. Ben Hylak’s answer is simulation: take real past traces, rerun your changed agent against them in a rebuilt copy of the world as it was, and diff the results. The question you are answering is ‘what does my change change?’
The method is Hylak’s, written up in our own words. The starter judge prompts and the trace_diff.py script are our templates for putting his rules into practice, not from the talk — watch the original for his demos. Before you start: your agent must already log full traces (every model call, tool call, input, output, and error). If it doesn’t, Step 1 fixes that. And you need a sandbox or local copy of the agent you can run against fake or read-only data — never replay against live production systems.
Step 1. Read your own traces first
Hylak calls looking at your own traces ‘by far probably the most important thing you can do,’ and says it is where you should start before any simulation. Read raw traces, not AI-written summaries of them. One free, local, open-source option is Raindrop Workshop: install it with `curl -fsSL https://raindrop.sh/install | bash` (read the install script before running it), then run `/instrument-agent` inside your coding agent in your repo. Useful commands from the same README: `raindrop workshop` starts and opens the UI (defaults to http://localhost:5899), `raindrop workshop status` checks health, `raindrop workshop reset` deletes the local trace database after asking. Any tracing tool works — what matters is that you can export a trace and read every tool call in it.
Starter prompt for a first read (paste a trace into your coding agent): ‘Here is one full trace from my agent. Walk through it tool call by tool call. Mark the exact step where it first went wrong, quote the input and output at that step, and suggest the smallest code or prompt change that would fix it. Do not summarize the whole trace; point me at the moments that matter.’
Steps 2 and 3. Name the one change, pick traces that exercise it
Write the change down in one line. Examples from the talk: a new model, adding a tool, removing a tool, changing a tool’s input type or field name, editing a prompt, switching harness or SDK. Then scenario choice matters more than volume: if you delete a tool, pull real past traces where that tool was called — a thousand replays that never touch the deleted tool tell you nothing. Hylak’s rough guidance: removing one tool, roughly 5 to 10 traces that used it; rewriting the whole harness, 50 or more. Always add a random sample of ordinary traffic so you don’t overfit to the one bug you’re chasing, and if you’re verifying a bug fix, include the bad traces that showed the bug.
Steps 4 and 5. Rebuild the world, then prove it’s faithful
This is the hard part. The world has moved since the trace was recorded — his example: a delivery app’s ‘how far away is my order?’ trace said 15 minutes last week; replay it against today’s database and the order is gone. What does not work: pointing the replay at live production (state has changed, and newer data tips the agent off), caching the old tool results and feeding them back (Hylak says the agent will ‘instantly call your bluff’ and say it’s in a simulation), or empty sandboxes (blank search, history, and issues tell the agent it’s fake). His ‘Swiss cheese’ approach: treat each recorded tool result as a window into the world at that time and seed the simulation with it; use a model to fill the holes so a slightly different call still gets a plausible answer; run as much of your real code as possible (parsers, internal query dialects, sandboxes) right up to the network boundary, only mocking at the edge; and for a brand-new data source the agent has never seen, give the world-builder real data — drop in JSON files or connect a read-only MCP to that source (his example was a read-only Supabase MCP).
Before testing any change, replay recent traces from your unchanged main branch — they should come out matching production. If they don’t, fix the simulated world first; he calls this the world-building phase. Then check for simulation awareness: after a batch of runs (he mentions around 20), have a model look at each pair blind and guess which one is the simulation. If it guesses right more often than chance, your world has tells. Starter blind-judge prompt: ‘Below are two traces of the same AI agent handling the same user request, labeled A and B. One ran in production and one ran in a simulated environment. Which one is the simulation? Answer with exactly A or B, then one sentence naming the clue you used.’ Shuffle which slot gets the real trace, and score at the population level, not trace by trace.
Steps 6 and 7. Replay, then diff first with cheap deterministic checks
Run your changed agent on the selected traces in the rebuilt world. Agents are nondeterministic, so replay the important ones a few times. Then Hylak separates diffs (what changed) from evals (scores you push up over time), and says to start as deterministic as you can before reaching for model judges. His checks: is the final response different? Any new tool errors? Different tool usage? Step count within about 20% of before? Cost and latency within about 20%? A starter template that runs these checks ships with this guide at /briefing/files/trace_diff.py — adapt its field names to your trace export, then run `python3 trace_diff.py baseline.json replay.json`. It exits non-zero when a trace needs review.
Step 8. Small yes/no judges for what code can’t check
His cheapest judge is ‘is the answer the same?’ Run it over every replay and only read the ones that say no. He prefers yes/no (pass/fail) judges over 1-to-5 scales. Starter ‘same answer’ judge: paste the original and new answers, then ask ‘Do these two answers give the user the same information and the same outcome? Ignore wording and formatting. Answer exactly SAME or DIFFERENT, then one sentence.’ Starter ‘did the fix work’ judge: describe the bug in one line, paste the new output for a request that used to trigger it, then ask ‘Does this output still show the bug? Answer exactly YES or NO, then one sentence.’ Any YES means the fix didn’t work. Fuzzy qualities like verbosity or ‘good writing’ come last, only after the humans on your team agree on what good means.
Step 9. Read the traces that moved, then ship
Read at least a few of the flagged replays by hand — he says that tells you more than any eval sweep. Then ship, watch production, and repeat: log, find issues, simulate the fix, ship. The common mistakes to avoid: replaying against live systems; only feeding back cached tool results; empty sandboxes; only replaying failures (you overfit to one bug — mix in random traffic); running each case once (nondeterminism hides regressions); never reading the simulated traces yourself; piling up thousands of input/output pairs and running them all on every change instead of picking scenarios that match the change; and starting with fuzzy LLM scores before the cheap deterministic diffs. Start by reading your traces locally — if your agent has a few tools and simple data, you may be able to build this against a limited read-only copy of production. Running a full shadow copy of the agent on all live traffic works but doubles cost and still leaves you reading traces; production stays the final source of truth. And the honest cutoff: claims we couldn’t verify from the talk or official docs were left out, including details of the OpenAI Codex simulation paper he mentions (he doesn’t name it) and the internals of Raindrop’s paid product.