How to move a live process onto an agent without breaking it

Run the agent and the existing process side by side on the same real traffic, for one to two weeks, with a person still making the final call. Compare the two outputs every day and write down each disagreement. When the disagreements stop being surprises, hand it over. The parallel run is the step that catches what a demo structurally cannot.

Two weeks after Solar Vista's WhatsApp ordering agent went live, it started answering half-questions.

People do not send one message on WhatsApp. They send four: the product, then the size, then a photo, then "is this available". The agent replied to the first one before the rest had arrived, which made it look confidently wrong to a customer who had not finished talking. The fix was to stop treating a message as a question and start waiting for a pause in sending.

Nothing in a demo would have found that, because in a demo the person typing is the person who built it, and they send one complete message.

Why does a demo miss this?

Because a demo runs on inputs the builder chose, and real traffic arrives in a shape nobody designed for.

Every test case you write is a sentence you already imagined. The failures that matter are the ones you could not imagine, and they are almost never about the model being stupid. They are about the world being shaped differently from the test: four messages instead of one, an address typed into the wrong field, a customer who answers a question you did not ask, a photo with no text at all.

You cannot write those cases in advance. You can only put the system where they happen and watch.

What does a parallel run actually look like?

Both systems see every real message. Only the human output reaches the customer. You keep a list of every case where the two disagreed.

That is the whole mechanism, and the discipline is in the list rather than the setup. Each disagreement gets one line: what came in, what the person did, what the agent would have done, and which one was right. Some days there are none. The day there are nine, you have learned something a year of planning would not have given you.

One to two weeks is usually enough. The signal you are waiting for is not zero disagreements, which you will not get. It is that the disagreements stop surprising you, because by then you know the shape of what the system gets wrong and you have decided whether each one matters.

NIST's AI Risk Management Framework (AI 100-1, January 2023) organises this kind of work as four functions: govern, map, measure, manage. Measure is the one that goes missing when a deadline moves, and a parallel run is the cheapest version of it available to a small team. It also forces the thing most builds skip, which is writing down what a correct output looks like before you are under pressure to accept whatever you got.

What is the difference between deployed and ready?

Deployed means the code runs. Ready means somebody has agreed to be responsible for what it does.

These land on different days and treating them as the same day is how a cutover goes wrong. A system can be deployed on Tuesday, correct in testing, and still have nobody who checks it on Thursday, nobody who notices when it stops, and nobody whose job it is to decide whether today's odd case was acceptable.

So the handover is a conversation, not a deploy. One named person owns each automation. They know what it does without asking, they know what it is allowed to decide alone, and they know how to switch it off. If you cannot name that person, the system is deployed and is not ready, and the honest thing is to keep running in parallel until you can.

What do you watch in the first fortnight?

The cases the system handled quietly, more than the ones it escalated.

An escalation is a system working. Somebody will look at it. The expensive failures are the ones that looked fine: a confident answer nobody questioned, a message marked resolved that was not, an order confirmed against an address the customer had already corrected in the next message. Those are invisible by design, which is why the first two weeks need somebody actually reading the log rather than watching a count.

Two questions are worth answering in writing before the switch. What does the system do when it does not know, and who is told. And how long from noticing a problem to stopping the thing, measured in minutes, not in intentions. If the answer to the second is that somebody would have to message a developer, the number is hours and you should know that before you need it.

We keep the first two weeks after handover under live monitoring for this reason, which is how we work, and the same fortnight is where an ordering agent earns or loses the trust of whoever has to defend it internally. If you have not got as far as building the thing yet, the step before this one is testing it before it goes live.

What are you giving up for two weeks?

Two weeks of somebody doing a job twice, and the patience to not switch it on fully when it looks like it is working.

That is a real cost and it is the reason the step gets skipped. The parallel run has no visible output: nothing ships, nothing speeds up. The system you already have keeps running, and all you get out of the fortnight is a list of disagreements, which feels like homework next to a launch.

Everything after that list is cheaper because of it. The alternative is meeting the four-message problem in week two, with real customers on the other end of it. The list is the product.