AI7 min read
Build the eval before the feature
ARQON EngineeringARQON Singapore
The fastest way to lose three months on an AI project is to ship the demo first and work out whether it's right later.
When we're called into a stalled AI project, we usually find the same scene. There is a prototype, and in a demo it works beautifully. Someone in leadership has seen it and is asking when it can launch. And nobody in the room can answer a very simple question: is it right?
Right, here, doesn't mean right in general. It means right on the thousand inputs that matter to this business, checked in a way that two people measuring separately would reach the same conclusion. Without that, every decision after it is made on a hunch. Hunches don't survive production traffic.
What an eval is
An eval is not a benchmark score. It is a dataset gathered from real work, a scoring function that reflects what users actually care about, and the ability to run the two on every change. That is the whole idea. The hard part isn't the tooling. It is deciding what "good" means and writing it down.
- A dataset drawn from real inputs, including the awkward ones people currently handle by hand.
- Ground-truth labels from the people who make that call today: underwriters, clinicians, support leads.
- A scoring function that can tell "wrong" from "unhelpful", because the fixes are completely different.
- A held-out set nobody tunes against, so you can tell real improvement from overfitting.
"I don't know" is a feature
A system that can't hold back an answer ends up guessing. On an underwriting project for a US insurer, the biggest jump in user trust didn't come from better retrieval. It came when we let the assistant say "I don't know", and scored whether it said so when it should have. Once underwriters believed a confident answer had a reason to be confident, they stopped re-checking every single one.
For the first three weeks, what they built was a way to tell us whether we were wrong. Looking back, that was the project.
What changes
With an eval in place, arguments get shorter. A model upgrade stops being a leap of faith and becomes something you measure and decide. Cost cutting stops being frightening, because you can prove nothing degraded when you route the routine 80% of traffic to a cheaper model. And when leadership asks whether it's ready, you can answer with a number.
It takes a week or two up front. We have never regretted spending that time. We have been called in many times to clean up projects that didn't.
Filed under AI · 2026.06.18
Back to the journal