"Our AI system improved processing speed by 47%." This is the kind of impact claim that fills enterprise AI case studies. What it rarely includes: how "improved" was measured, over what period, against what baseline, with what confidence interval, and whether 47% is the point estimate or the upper bound of a range. Tania and I have been working on measurement frameworks that hold up to the same standard we'd expect from any evidence-based claim.
Key Takeaways
- Impact measurement without a proper baseline is comparison to nothing. Establish baselines before deployment, not after.
- A/B testing is the gold standard for AI impact measurement but is rarely feasible in enterprise contexts. Interrupted time series and matched comparison designs are practical alternatives.
- Statistical significance is necessary but not sufficient. Practical significance, whether the measured effect is large enough to matter operationally, is equally important.
- Measurement should be continuous, not one-off. AI system performance changes over time as inputs, users, and processes evolve.
73%
of enterprise AI teams report impact using single point estimates without confidence intervals
Source: Gartner, AI in the Enterprise Survey 2024
2-4x
typical variance between best-case and worst-case AI impact when properly measured with confidence intervals
Source: RIVER advisory assessments, 2025
The Baseline Problem
The most fundamental measurement error: deploying AI and then measuring how well things are going, without a rigorous record of how things were going before. This is comparison without a comparator. Any observed improvement could be due to the AI system, or to seasonal variation, staffing changes, process improvements that happened simultaneously, or simply regression to the mean.
A proper baseline requires:
- Measuring the same metrics for a sufficient period before deployment (minimum 3 months for most enterprise processes)
- Documenting all confounding factors that could change between the baseline and measurement periods
- Using the same measurement methodology for both periods
Practical Measurement Designs
Interrupted Time Series
When A/B testing isn't feasible (which is most of the time in enterprise), interrupted time series is the strongest practical alternative. You measure the outcome metric over a long period before AI deployment and a long period after, then use statistical methods to determine whether the deployment caused a change in the level or trend.
If processing speed has been stable at 12 minutes per case for 6 months and drops to 8 minutes after deployment, the statistical test tells you whether that drop is larger than the normal variation in the series. It is not proof of causation, but it is substantially stronger evidence than a simple before-after comparison.
Dr Vincent Russell
Machine Learning (AI) Engineer
Matched Comparison
If you're deploying AI in one team or department first, use a comparable team without AI as a comparison group. The teams should be matched on key characteristics: volume, complexity, staffing, and historical performance.
This design is susceptible to selection bias (the team chosen for AI deployment may be systematically different from the comparison team), but it provides stronger evidence than no comparison at all.
Staggered Rollout
Deploy AI to different teams at different times. Each team serves as its own control (before their deployment) and as a comparison for other teams (before and after). This design controls for time-varying confounders and provides multiple independent estimates of the AI impact.
Beyond Statistical Significance
Practical Significance
A processing speed improvement that is statistically significant but amounts to 30 seconds per case may not be practically significant. The question is not just "is this effect real?" but "is this effect large enough to matter?"
Define minimum practically significant effects before measurement. "We need at least a 15% improvement to justify the investment." Then evaluate whether the measured effect exceeds this threshold, not just whether it exceeds zero.
Effect Stability
A large effect measured over two weeks may not persist. Novelty effects (people try harder with new tools), Hawthorne effects (people perform better when observed), and learning curves all influence early measurements. Measure over at least three months before drawing conclusions about sustained impact.
Heterogeneous Effects
Average impact hides variation. AI may dramatically improve performance on routine cases while providing no benefit (or even degrading performance) on complex cases. Break impact measurement down by case type, user experience level, and workflow context.
Measuring AI impact properly is harder than measuring it poorly, but the investment in methodology pays for itself. Rigorous measurement builds the evidence base for continued AI investment, identifies where AI is underperforming, and gives leadership defensible numbers for board conversations. The alternative, vague claims and cherry-picked metrics, eventually undermines credibility.