ARTICLE · CONTROLS FOR AI WORK · PART 6 OF 6 · Commissioning · capability

Run it off like a process

You don’t hand a new line over to production because it ran once. You run it on real parts, at rate, including the odd ones, and you keep the numbers. AI deserves the same treatment, and most people skip it: they try an AI agent a few times, it looks good, and they trust it. One good run is a demo. Commissioning is how you find out what it does on the run that matters.

The same questions apply to a sales pipeline, a purchase order or a month-end close: what stays fixed, what needs judgment, and who decides?

What the AI field has found

Researchers at Sierra built a test called τ-bench (tau-bench) to measure how AI agents hold up with real customers. A second model plays the customer and talks to the agent under test, which has real tools and real company policies to follow. At the end, the test checks the result itself, not how good the conversation sounded.

The important part is how they score it. They don’t ask whether the agent can succeed. They ask whether it succeeds every time: the same task, run eight times, passing all eight. Even the best agents they tested at the time succeeded on fewer than half the tasks, and passed all eight tries on fewer than a quarter of the retail tasks.

That’s the gap between “it worked when I tried it” and “it works.”

From the shop floor: commissioning, and the test you do without thinking

On a new line, commissioning means proving it on representative work before production owns it. Capability studies. Measurement studies. Unusual parts as well as easy ones. Results checked against what a good operator would have produced. And it doesn’t end at handover; you keep collecting data, because the process drifts.

The best test I ever ran, I didn’t mean to run.

We were qualifying a new static-dissipative material for parts bins, the kind that keep electronic parts from being damaged by static. It was careful work: gloves on, meters on the parts, measuring how well the surface carried charge away. The supplier’s scientist was in the room watching.

Out of habit, I picked up one of the totes and threw it at the floor. I’d done that to every tote that came into the building for years. If a tote split at the gate, it told me there was stress in the molding. Ours could all take it.

This one shattered into hundreds of pieces.

Every electrical test on that material had passed. The test that found the problem was the one nobody had written into the qualification, the one I did on autopilot. A year later the supplier came back with a reformulated material, much tougher, and told me how many hours they’d put into what they called “Mike’s test.”

How it carries over

Test with a script, like a test fixture. A test script plays the user: it says each line, and checks each reply against what should happen. It plays the person an AI agent serves, from a plain-text script, and records pass or fail for each check.

A pass needs evidence. This is the rule I care most about. A good rule for any test script:

After every reply, look for evidence on each open check. A pass needs a quote: the turn number and the AI agent’s words. A fail needs the same. No evidence by the end is “not reached”, never a pass.

“It seemed fine” is not a pass. If the check never came up, it’s “not reached,” and you go write the test that reaches it. Never soft-pass a check, and never change what the check expects so the result comes out green.

Run it more than once. AI isn’t a PLC; the same input can get different answers. Run the same script several times. An AI agent that passes once and fails twice has a problem you need to see now, not in front of a customer.

Include the odd cases. Write scripts for the user who changes their mind halfway through, the one who gives the wrong unit, the one who asks the AI agent to do something it shouldn’t. That’s where AI agents break, the same way the odd parts are where a line breaks.

Keep your habit tests. Every plant has a few tests that aren’t in any procedure but catch real problems. Find yours for AI. Mine include taking a long session’s conclusion to a fresh session (Part 1) and asking for the source of any number that matters.

Test in a safe place. The tester never takes real actions. If the AI agent under test offers to send an email or place an order, the tester says “not now” and logs the offer. You commission on a test stand, not on the customer’s line.

Keep testing after handover. Models get updated, and an AI agent that passed last month can fail this month. Re-run the scripts after any change to the AI agent or the model behind it, the same as you’d re-qualify after a mold change.

Try this

Write a test script for one AI agent you use: five turns a typical user would say, and for each turn one thing the AI agent must do and one thing it must not do. Run it three times. Count the passes, and for every pass, write down the words that prove it.

Sources

  • Shunyu Yao and others (Sierra), “τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains,” ICLR 2025: arxiv.org/abs/2406.12045

These pages sit with the rest of the Advance Now family. The hotbotspot tool library writes the methods up for AI. A.N. Creation Office is where the public AI agents run as open experiments.