Feature comparison spreadsheets lie.

Every AI vendor has an impressive feature list. Every demo runs flawlessly on prepared data. Every sales presentation promises transformation that never quite arrives the way it was pitched, and you discover this only after signing a contract that locks you in for eighteen months.

The AI vendor landscape punishes traditional evaluation approaches because traditional approaches were designed for software that works the same way every time you run it, which is precisely what AI tools do not do. A model that excels at your test prompt might hallucinate on the real data you feed it three weeks after implementation. The vendor who seems responsive during sales might take days to reply after the contract closes.

Something has to change in how we evaluate.

What Feature Lists Actually Hide

Vendors compete on feature counts. More features suggest more value. This logic collapses when applied to AI.

A feature that exists is not a feature that works for your use case. The gap between “our product can do X” and “our product reliably does X for customers like you” is often enormous, and vendors have financial incentive to blur that distinction at every opportunity.

Consider model capabilities. Most vendors now offer access to frontier models from OpenAI, Anthropic, and Google. The model itself becomes commoditized. What matters is everything around that model: the prompting infrastructure, the integration quality, the error handling when things go wrong. These implementation details rarely appear on feature comparison pages.

simonw, creator of Datasette and a respected voice on AI tooling, captured this reality in a Hacker News discussion on AI evaluation:

“If you try to fix problems by switching from eg Gemini 2.5 Flash to OpenAI o3 but you don’t have any evals in place how will you tell if the model switch actually helped?”

The model matters less than your ability to measure what any model does for you. Vendors who push model names as their primary selling point are often hiding weak infrastructure behind borrowed credibility.

Red Flags That Vendor Presentations Create

Watch how vendors respond to specific questions about limitations, and you learn everything you need to know about the relationship you would be entering.

The pivot to prepared demos. You describe your specific use case. They show a different demo. This happens constantly. The prepared demo works because it was engineered to work. Your use case was not engineered. The pivot tells you they either cannot handle your scenario or choose not to show you their tool struggling.

Vagueness about training data. Where did the data come from that trained their custom models? Many vendors cannot or will not answer. This matters for both quality and legal risk. Models trained on scraped data of uncertain provenance carry copyright exposure that could land on your desk later.

The missing failure stories. Every tool fails sometimes. Vendors who claim otherwise are lying or have not been tested at scale. Honest vendors describe where their tools struggle. They know their limits because they have watched real customers hit those limits. This honesty signals partnership rather than salesmanship.

Future features as current value. “That capability is on our roadmap” translates to “we do not have that capability.” Evaluate what exists, not what might exist. Roadmaps change. Funding dries up. Priorities shift. Features promised for Q3 sometimes never arrive at all.

Running Evaluations That Reveal Truth

Demos show best cases. Real evaluation requires building tests that your chosen tool might fail, then watching closely to see how it fails.

Start with edge cases from your actual work. Not representative samples. Edge cases. The weird requests that confuse your human team. The messy data formats you actually receive. The unusual questions customers sometimes ask. AI tools that handle typical cases well but collapse on edge cases will generate escalations and frustration once deployed.

Nathan Lambert, a researcher who writes extensively about AI model capabilities, described his own switching experience:

“Claude 3.5 just does what I need a few percentage points more reliably than ChatGPT”

A few percentage points. This is how real differences manifest. Not dramatic capability gaps that anyone could spot in a demo, but small reliability differentials that compound over thousands of uses into major workflow impacts. You cannot see these differentials without sustained testing on your actual tasks.

Structure your evaluation to reveal these differentials:

Run identical prompts across vendors. Same input, different tools, measured outputs. Do this at scale. Not five tests. Fifty tests minimum. One hundred if the decision matters enough.

Test over time. A tool that works perfectly on Monday might struggle on Thursday if the vendor is managing capacity issues or rolling out updates. A one-day evaluation tells you about one day. A two-week evaluation begins to reveal patterns.

Involve the people who will actually use the tool. Technical evaluators test different things than daily users. Both perspectives matter. Someone who will use this tool eight hours per day notices friction that someone testing for an afternoon will miss.

Document failures precisely. When something goes wrong, capture exactly what went wrong. Vendor support quality shows itself in how they respond to documented failures. Some vendors troubleshoot. Some vendors deflect.

The Lock-In Consideration Nobody Mentions Early Enough

Switching costs in AI compound faster than people expect.

You build prompts. You train teams on interfaces. You integrate tools into workflows. You build internal documentation. You develop tribal knowledge about what works and what to avoid. All of this becomes sunk cost that makes switching painful even when switching would be smart.

A 2025 survey of IT leaders found that 45% report vendor lock-in has already hindered their ability to adopt better tools. Nearly half of organizations feel trapped with vendors they chose before understanding the full implications of that choice.

Consider lock-in during initial evaluation, not after. Ask vendors uncomfortable questions:

Can you export all prompt templates and configurations in a portable format? What happens to your data if you leave? Are there exit fees? How long does data deletion take? Do they use your data to train models that competitors could benefit from?

The vendors who answer these questions clearly and favorably are vendors who believe their product quality, not your switching costs, will keep you as a customer. That confidence itself is a signal worth noting.

Architectural decisions made during implementation also affect lock-in. Building abstractions between your systems and the vendor’s API creates future flexibility. Hard-coding vendor-specific logic throughout your codebase creates dependency that grows harder to escape as time passes.

Some lock-in is acceptable. You cannot achieve deep integration without some commitment. But knowing your lock-in level and choosing it deliberately differs from discovering it accidentally when you try to leave.

What Demonstrations Cannot Show You

Support quality.

During sales, every question gets answered quickly. After contract close, response times sometimes expand dramatically. The support team selling you is not the support team helping you, and incentives shift once the deal completes.

Ask for references specifically about support experiences. Not reference customers who implemented successfully and never needed help. References who hit problems. How were those problems handled? How long did resolution take? Did they feel like partners or like tickets in a queue?

Organizational change capacity matters too. A tool your team will not use fails regardless of capability. Understanding your organization’s readiness for new technology, training requirements, and change tolerance should influence vendor selection as much as feature comparison.

And perhaps most importantly: the evaluation process itself matters. How vendors behave during evaluation predicts how they will behave as partners. Pressure tactics during sales suggest pressure tactics during renewals. Transparency about limitations suggests transparency about issues. The relationship you experience while evaluating is often the best version of the relationship you will ever have with that vendor.

The Question That Replaces All Checklists

Evaluation frameworks provide structure. Structure helps. But every framework eventually yields a weighted score that obscures the judgment call no scoring system can make for you.

When practitioners describe their best AI vendor decisions, they rarely talk about evaluation frameworks. They talk about fit. The tool that worked was the tool that matched how their team actually works, that addressed their specific problems, that felt right in daily use after the demo shine wore off.

The question that matters: “Based on everything we learned during evaluation, do we believe this vendor will help us succeed, and do we trust them enough to build dependency on their infrastructure?”

Trust is hard to score on a spreadsheet. It emerges from watching how people behave when things get difficult. The best evaluations create small difficulties intentionally, then observe carefully.

Some vendors will not like this approach. Those vendors are telling you something important.