Evaluating an AI vendor when you're not technical.

You don't need to understand model architecture to evaluate an AI vendor well. You need to ask questions whose answers are hard to fake, and then listen to how they're answered rather than what's claimed.

These six have served us reliably. None require technical knowledge. All of them make an unprepared vendor visibly uncomfortable, which is itself the signal.

1. "What does this do badly?"

A vendor who has genuinely deployed their product knows its limits precisely, and good ones answer this immediately and specifically: it struggles with X, it needs clean Y, it's wrong about Z roughly this often.

A vendor who deflects — "it's really about how you use it" — either hasn't deployed at scale or isn't going to tell you the truth when something breaks. Either way you've learned what you needed.

2. "Show me a customer whose situation resembles ours, and let me speak to them unaccompanied."

Reference calls with a sales rep on the line are theater. Ask for the contact directly. Then ask that customer one question: what surprised you after signing?

3. "What do you need from us for this to work?"

Every real deployment has customer-side requirements: data in a particular shape, a named owner, a process change, integration effort. A vendor who claims to need almost nothing is describing a demo, not an implementation.

The honest answer to this question is usually longer and less comfortable than you'd like, and that's the answer you want.

4. "How do we measure whether this worked, and when?"

Push for a specific metric, a baseline method, and a timeframe. If a vendor can't propose how their own product should be judged, they've either never been held to it or don't expect to be.

Get the answer in writing before you sign. It becomes the standard both sides are working toward, and it prevents the conversation in month six where success is retroactively redefined.

5. "What happens to our data, and what happens if we leave?"

Where does it live, who can access it, is it used to train anything, and can you export everything in a usable format on the way out? Exit terms are the clearest signal of a vendor's confidence — firms who expect to earn renewal make leaving easy.

6. "Is this your technology, or are you reselling someone else's?"

Not a disqualifier. Plenty of excellent products are thoughtful layers over a foundation model, and the packaging can be the actual value. But you need to know, because it determines what you're paying for and whether you could buy the underlying capability directly for materially less.

The answer also tells you what happens when the underlying provider changes pricing or terms, which is a risk you're taking whether or not anyone names it.

The meta-question

Ask whoever is advising you on the selection whether they receive any compensation from the vendors under consideration. Referral fees and reseller margins are common and often undisclosed, and they shape recommendations in ways that are difficult to detect from the outside.

For what it's worth: we hold no vendor partnerships that pay us and we resell nothing. If the best answer is a tool you can buy directly for a fraction of a custom build, that's what we'll tell you — and it's a recommendation we've made more than once.