Skip to main content

From Experimentation to Execution: The AI Transition Most Enterprises Get Wrong

Your AI experiments worked. Now comes the organisational shift that determines whether any of it matters.
5 December 2023·9 min read
Tim Hatherley-Greene
Tim Hatherley-Greene
Chief Operating Officer
Isaac Rolfe
Isaac Rolfe
Managing Director
enterprise-aitransformationstrategy
2023 was the year of AI experimentation. Every enterprise ran proofs of concept. Most of them worked. And now comes the harder question: how do you turn experiments into operations?

What You Need to Know

  • The gap between AI experimentation and execution isn't technical. It's organisational. The skills that make a pilot succeed (small team, narrow scope, fast iteration) are different from the skills that make production work (integration, governance, change management).
  • Most enterprises are stuck in "experiment mode," running more pilots instead of operationalising the ones that worked. This creates pilot fatigue and organisational cynicism.
  • The transition requires three shifts: from innovation team to operations team, from demo metrics to business metrics, and from standalone tools to integrated workflows.
  • Change management is not an afterthought. It's the main event. Technical deployment is 30% of the work; adoption and integration are 70%.
  • The enterprises that succeed in 2024 will be the ones that stop experimenting and start building foundations.
Enterprise AI Delivery Effort Split
Source: Gartner, Emerging Technology Roadmap for Large Enterprises, 2023
80%+
of organisations have piloted generative AI, but less than 5% have integrated it into core workflows
Source: Gartner, Emerging Technology Roadmap for Large Enterprises, 2023

The Experimentation Trap

Here's the pattern we've seen play out across dozens of enterprises in 2023:
January: Board asks about AI. CTO gets tasked with exploration. March: Innovation team starts experimenting with GPT-4. May: First proof-of-concept demo impresses leadership. July: Second POC started. First one still hasn't moved to production. September: Third POC. Some people are using ChatGPT informally. No governance. November: Board asks for AI ROI numbers. Nobody has them.
The organisation has done a lot of experimenting and very little executing. Each POC proved that AI can work. None of them proved that AI does work: in production, at scale, integrated into real operations, delivering measurable business outcomes.
This is the experimentation trap. It feels productive because things are happening. But nothing is compounding.

The Three Shifts

Shift 1: From Innovation Team to Operations Team

Experiments are run by small, skilled teams with freedom to move fast and break things. Operations require different people with different skills:
  • Integration engineers who can connect AI to legacy systems
  • Domain experts who validate outputs and define edge cases
  • Operations managers who design workflows around AI outputs
  • Support teams who handle escalations when AI gets it wrong
This doesn't mean the innovation team disbands. It means they hand off to an operations-ready team, or better, they embed within the operations team for the transition.
The best transitions I've seen involve the innovation team and the operations team working side by side for 4-6 weeks. Both sides learn things that make the deployment better.
Tim Hatherley-Greene
Chief Operating Officer

Shift 2: From Demo Metrics to Business Metrics

Experiments are measured by technical performance: accuracy, speed, capability. Operations are measured by business performance: cost reduction, throughput improvement, error rate reduction, customer satisfaction.
The translation isn't automatic. A model that's 95% accurate in a demo might deliver a 30% throughput improvement in production, or it might deliver 5%, depending on integration quality, edge case handling, and adoption rates.
Define business metrics before the production deployment:
  • What number do we expect to change?
  • By how much?
  • Over what timeframe?
  • How will we measure it?

Shift 3: From Standalone Tools to Integrated Workflows

The biggest shift, and the hardest. Most AI experiments exist as standalone tools: upload a document, get an analysis, copy the result into another system. This is fine for a demo. It's terrible for operations.
Production AI needs to be embedded in existing workflows:
  • The claims document gets processed automatically when it arrives
  • The AI analysis appears in the claims management system, not a separate tool
  • The adjudicator reviews the AI's work in their existing interface
  • Exceptions are routed automatically, not manually escalated
This integration work is where most of the production investment goes. It's also where most of the production value comes from, because it removes the manual steps that make AI usage optional.

The Change Management Reality

Here's what nobody talks about in AI vendor pitches: the technology is the easy part.
The hard part is getting 200 claims handlers to trust and use an AI tool that changes how they've done their job for a decade. The hard part is getting the compliance team comfortable with AI-assisted decisions. The hard part is getting the IT team to prioritise integration work alongside their existing backlog.
Change management for AI involves:
Building trust incrementally. Start with AI as a suggestion engine, not a decision engine. Let users see the AI's work, correct it, and build confidence over time. Force-deploying AI into high-stakes workflows creates resistance that can kill the entire initiative.
Involving users in design. The claims handlers who'll use the system should be in the room when it's being designed. They know the edge cases, the workarounds, and the real workflow, not the documented workflow.
Communicating honestly. "AI will help you focus on the interesting, complex cases instead of routine processing" is true and motivating. "AI will make you more efficient" sounds like a headcount reduction. Words matter.
Measuring adoption, not just accuracy. A technically perfect AI system that nobody uses delivers zero value. Track usage rates, user satisfaction, and workflow integration alongside model performance.

A Practical Transition Plan

Weeks 1-2: Production readiness assessment Which POC is closest to production-ready? What's the gap? Who needs to be involved?
Weeks 3-6: Integration and governance Connect to real systems. Implement security and governance. Build monitoring and logging.
Weeks 7-8: Pilot production (limited deployment) Deploy to a small group of real users doing real work. Measure business outcomes, not just technical performance.
Weeks 9-12: Rollout and optimisation Expand to full deployment. Optimise based on real-world performance. Establish feedback loops for continuous improvement.
Ongoing: Foundation building As you operationalise the first capability, invest in making the infrastructure reusable for capabilities #2, #3, and #4. This is where compound value starts.
How long does the transition from experiment to production take?
For a well-scoped AI capability with reasonable data readiness: 8-12 weeks from the end of the experiment to production. Most of this time is integration, governance, and change management, not model development.
Should we keep experimenting while transitioning?
Yes, but limit new experiments to 1-2 at a time and ring-fence them from the production transition. The worst outcome is spreading attention so thin that nothing reaches production. Prioritise finishing over starting.
What if the team that built the experiment isn't available for the production transition?
This is common and manageable. Invest 2-3 weeks in knowledge transfer before the production team takes over. Document the experiment's architecture, known limitations, and design decisions. And keep a communication channel open for questions during the transition.