Five months ago, your enterprise software vendors sold you "cloud-first digital transformation." Today they're selling you "AI-powered intelligent automation." The product hasn't changed. The pitch deck has.
The Problem
ChatGPT created a gold rush. Every vendor in the enterprise software market is scrambling to add "AI" to their product descriptions, their roadmaps, and their pricing. Some of them are doing genuinely interesting work. Most of them are wrapping an OpenAI API call around their existing product and calling it innovation.
For enterprise leaders trying to make informed decisions, the signal-to-noise ratio has never been worse.
Here's how to filter.
The Five-Question Hype Filter
1. What Data Does Your AI Use?
The answer to this question tells you almost everything you need to know.
Good answers: "Our AI uses your organisation's data, processed through our proprietary pipeline, combined with a foundation model for inference. Here's how we handle data governance."
Bad answers: "We use the latest AI technology" or "Our AI learns from millions of data points" (whose data points?) or any answer that avoids specifics about data provenance, storage, and governance.
If a vendor can't clearly explain what data their AI uses and how they handle it, they either don't understand their own product or they're hoping you won't ask.
2. When Did Your AI Capability Ship?
Timing matters. AI companies that have been building AI capabilities for years have had time to solve real problems, accumulate domain knowledge, and iterate on accuracy. Companies that "pivoted to AI" in January 2023 are selling aspiration, not capability.
This doesn't mean new entrants can't be good. It means you should verify claims more carefully when the AI strategy is newer than ChatGPT.
3. How Do You Measure Accuracy?
This is the question that separates real AI products from marketing. A real AI product has:
- Defined accuracy metrics for specific tasks
- Benchmarks measured on representative data
- Clear documentation of failure modes
- Honest communication about limitations
If a vendor says their AI is "highly accurate" without specific metrics, or claims it "doesn't hallucinate," they're either not measuring or not being honest. Both are disqualifying.
4. What Happens When the AI Is Wrong?
Every AI system makes mistakes. The important question is what the product does about it.
Good systems: Provide confidence scores, source attribution, human-in-the-loop review workflows, clear escalation paths, and audit trails.
Bad systems: Present AI outputs as definitive answers with no mechanism for verification, correction, or override.
How a vendor handles AI errors tells you more about the maturity of their product than any feature demo.
5. Can I Talk to a Customer Who's Using It in Production?
Not a beta tester. Not a pilot participant. A production customer who has been using the AI capability in their actual operations for at least three months.
If the vendor can't produce one, their AI capability is a roadmap item, not a product. That's fine - just make sure you're pricing it accordingly.
>1,000
enterprise vendors added 'AI' to their product descriptions in Q1 2023
Source: CB Insights, AI Hype Tracker, Q1 2023
Red Flags
- "AI-powered" appears on every page of the marketing site but nowhere in the technical documentation. Marketing-led, not product-led.
- The demo is impressive but you can't trial it with your own data. Demo data is cherry-picked. Your data is messy. Those produce very different results.
- The AI is described as a black box. If they can't explain how it works at a conceptual level, either they don't understand it or they don't want you to.
- Pricing increased substantially "because of AI." Sometimes justified. Often opportunistic.
- The AI roadmap is all futures, no present. "Coming in Q3" means "not built yet."
What Genuine AI Looks Like
You'll know it when you see it. The vendor explains clearly what their AI does and doesn't do. They have specific accuracy metrics. They show you failure modes alongside successes. They can connect you with production customers. They talk about data governance without being prompted.
And, tellingly, they're often the ones who talk about AI the least in their marketing. Because when you've built something that works, you don't need to oversell it.
Be Sceptical, Not Cynical
I don't want to be the "AI is overhyped" voice. It's not. The technology is genuinely transformative. But the gap between what's possible and what's being sold is wide, and it's your job as an enterprise leader to navigate that gap.
Ask hard questions. Demand specifics. Verify claims. And remember: the vendors doing the most shouting are usually the ones with the least to show.