Ask five AI vendors to explain their "stack" and you'll get five different diagrams, each suspiciously placing their product at the centre. Here's a vendor-neutral view of what enterprise AI architecture actually looks like, and which layers matter most.
What You Need to Know
- The enterprise AI stack has five layers. Most vendor conversations focus on one (the model). The other four determine whether the model delivers value.
- The model is the least differentiated layer. GPT-4, Claude, and open-source alternatives are increasingly commoditised. The value lives in the layers around the model.
- The layers that matter most for enterprise (data infrastructure, integration, and governance) are the least discussed and least funded.
- You don't build the stack bottom-up. You build it capability-first, and the stack emerges from real delivery.
- Understanding the stack helps you evaluate vendors, plan architecture, and avoid building throwaway infrastructure.
60-70%
of enterprise AI cost is infrastructure and integration, not model development
Source: McKinsey & Company, The State of AI in 2022, December 2022
The Five Layers
Layer 1: Foundation Infrastructure
The bottom layer: compute, storage, networking. Where your AI systems physically run.
Options:
- Cloud AI services (Azure OpenAI, AWS Bedrock, Google Vertex AI) - managed, scalable, enterprise-grade
- Private cloud - your infrastructure, your control, higher setup cost
- Hybrid - sensitive workloads on-premises, others in cloud
What matters: Data sovereignty (where is data processed?), latency (how fast do responses need to be?), and cost predictability (inference costs at scale). For most NZ enterprises, cloud AI services with appropriate data handling agreements are the pragmatic choice.
Layer 2: Data & Knowledge Layer
The layer that makes AI your AI: document processing, knowledge bases, vector databases, data pipelines.
Components:
- Document processing - extracting structured data from unstructured sources (PDFs, emails, scanned forms)
- Knowledge base - your organisation's processed, indexed knowledge in a searchable format
- Vector database - stores embeddings for semantic search (the core of RAG)
- Data pipelines - automated flows that keep your knowledge base current as source documents change
What matters: This is where most enterprise AI value is created and where most time is spent. The quality of your data layer determines the quality of your AI outputs. A mediocre model with an excellent data layer outperforms an excellent model with a mediocre data layer.
The Foundation Layer
This is also the layer with the highest foundation potential. A document processing pipeline built for claims intelligence serves fraud detection, compliance, and customer communication. Build it once, use it everywhere.
Layer 3: AI Models & Orchestration
The layer everyone talks about: the models themselves and the logic that orchestrates them.
Components:
- Language models - GPT-4, Claude, Llama, or domain-specific models
- Orchestration - the logic that decides which model to use, what context to provide, how to chain multiple steps, and how to handle errors
- Prompt management - versioned, tested prompts that produce consistent results
- Evaluation - automated testing of model outputs against known-correct answers
What matters: Model choice matters less than orchestration quality. A well-orchestrated pipeline using GPT-3.5 for simple tasks and GPT-4 for complex ones will outperform (and cost less than) GPT-4 for everything. Orchestration is the engineering layer. It's where AI becomes reliable.
Layer 4: Application & Integration Layer
The layer that connects AI to your existing systems and presents it to users.
Components:
- API layer - how other systems consume AI capabilities
- Integration connectors - connections to CRM, ERP, document management, legacy systems
- User interfaces - the AI-native interfaces your team actually uses
- Workflow integration - embedding AI into existing business processes, not alongside them
What matters: The best AI in the world is useless if it's not integrated into the workflows where decisions happen. This layer is often underestimated in planning and underinvested in delivery. Budget 30-40% of your AI initiative for integration work.
Layer 5: Governance & Monitoring
The layer that ensures AI stays safe, accurate, and compliant over time.
Components:
- Access control - who can use which AI capabilities, with which data
- Audit logging - every AI interaction recorded for compliance and review
- Performance monitoring - accuracy tracking, drift detection, latency monitoring
- Feedback loops - capturing corrections and improvements from users
- Policy enforcement - security guardrails, content filters, compliance rules
What matters: AI systems degrade over time as data changes, models update, and business requirements shift. Monitoring and governance aren't a launch requirement. They're an ongoing operational requirement. Plan for them from day one.
Which Layers to Invest In
| Layer | Investment priority | Why |
|---|---|---|
| Data & Knowledge | Highest | This is your competitive advantage. Models are commoditised; your data isn't. |
| Application & Integration | High | This determines whether AI delivers value in real workflows. |
| Governance & Monitoring | High | This determines whether AI stays safe and effective over time. |
| AI Models & Orchestration | Medium | Important but increasingly commoditised. Don't over-invest here. |
| Foundation Infrastructure | Lower | Use managed services. Don't build what you can buy. |
The most common mistake is inverting this priority: spending most of the budget on model selection and infrastructure while underinvesting in data, integration, and governance.
- Should we build our own models or use commercial ones?
- Use commercial models for 90%+ of enterprise use cases. Building custom models is expensive, requires rare talent, and the commercial models are improving faster than you can build. The exceptions: highly specialised domains with proprietary data where commercial models consistently underperform, and cases where data sovereignty requires fully on-premises model hosting.
- How does the AI stack relate to our existing IT architecture?
- It extends it, it doesn't replace it. Your existing identity management, networking, security, and data systems are still the foundation. The AI stack layers on top, using your existing infrastructure where possible and adding AI-specific components where needed.
- What should we build first?
- Start with Layer 2 (Data & Knowledge) for a specific use case, using Layer 1 (managed cloud services) and Layer 3 (commercial models). Build Layers 4 and 5 as you move to production. The layers emerge from delivery, not from architecture diagrams.