RAG (retrieval-augmented generation) solved the first problem: getting AI to use your organisation's data instead of making things up. But it has a limitation that becomes obvious at scale. RAG retrieves documents. It doesn't understand relationships between them. Knowledge graphs are the next layer.
Where RAG Hits Its Ceiling
RAG works well for straightforward questions against a document collection. "What's our policy on remote work?" - the system finds the relevant policy document, extracts the answer, and presents it with source attribution. Solid.
But enterprise knowledge isn't just documents. It's relationships between documents, between concepts, between people, between decisions. And this is where RAG struggles.
Example: a compliance question. "Does our proposed product comply with the latest FMA guidance?" Answering this requires:
- Understanding the proposed product (from multiple documents)
- Identifying the relevant FMA guidance (which may span multiple publications)
- Mapping the product's characteristics to specific regulatory requirements
- Identifying any gaps or conflicts
- Considering how previous similar products were assessed
RAG can retrieve relevant documents for each of these steps. But it can't reason across the relationships. It doesn't know that Guidance Note A supersedes Guidance Note B, that Product X has the same characteristics as Product Y (which was previously assessed), or that Regulatory Requirement Z was amended last month.
That reasoning requires structure. Knowledge graphs provide it.
What Knowledge Graphs Add
A knowledge graph represents information as entities and relationships. Not as flat documents, but as a network of connected facts.
| Concept | RAG Approach | Knowledge Graph Approach |
|---|---|---|
| A policy document | Stored as text chunks with embeddings | Stored as entity with relationships to products, regulations, teams, and dates |
| A regulatory change | Retrieved as a document if query terms match | Connected to affected policies, products, and compliance requirements |
| A product feature | Found in product documentation via semantic search | Linked to regulatory requirements, customer segments, risk assessments, and team ownership |
| Historical decisions | Found if the right documents are retrieved | Traversable as a decision chain - what was decided, why, what it affected |
The knowledge graph doesn't replace RAG. It augments it. RAG handles the "find relevant information" problem. The knowledge graph handles the "how does this information relate to everything else?" problem.
The Architecture
In practice, a knowledge graph-enhanced AI system has three retrieval layers:
Layer 1: Semantic Search (RAG)
The familiar pattern. User query is embedded, compared against document embeddings, and the most similar documents are retrieved. This handles direct factual questions and document-level retrieval.
Layer 2: Graph Traversal
The query is also parsed for entities and relationships. The knowledge graph is traversed to find connected information. "What regulations apply to Product X?" triggers a graph traversal from the Product X node, through its feature nodes, to the regulatory requirement nodes they're connected to.
Layer 3: Hybrid Reasoning
The results from semantic search and graph traversal are combined. The language model synthesises an answer that draws on both retrieved documents (for detail and nuance) and graph relationships (for structure and completeness).
This hybrid approach produces answers that are both contextually rich (from RAG) and structurally complete (from the graph).
35-45%
improvement in answer completeness when combining RAG with knowledge graph retrieval on enterprise compliance queries
Source: RIVER Group, internal benchmark on enterprise knowledge system, Q2-Q3 2024
Building the Graph
This is where it gets real. Knowledge graphs don't build themselves.
Entity Extraction
The first step is identifying entities in your documents: products, people, regulations, policies, decisions, dates, amounts. Modern NLP models (including GPT-4 and Claude 3) are good at this - you can use LLMs to extract entities from documents at reasonable accuracy.
Practical approach: Don't try to extract everything. Start with the entity types that matter for your specific use cases. For compliance: regulations, products, requirements, dates, decisions. For customer service: products, features, issues, solutions, customers.
Relationship Mapping
Entities alone aren't a graph. You need the relationships: "Product X is-governed-by Regulation Y," "Policy A supersedes Policy B," "Decision D was-made-by Person P on Date Z."
This is harder than entity extraction. Relationships are often implicit in documents rather than stated explicitly. You'll need a combination of:
- Automated extraction using LLMs (decent for explicit relationships)
- Rule-based extraction for structured metadata (document dates, authors, categories)
- Human curation for complex or critical relationships (regulatory mappings, decision chains)
Schema Design
The graph needs a schema - a defined set of entity types and relationship types. This is a modelling exercise, not a coding exercise. Get domain experts in the room.
Key principle: Start simple. A graph with 5 entity types and 10 relationship types that's complete and accurate is far more useful than a graph with 50 entity types that's sparse and unreliable.
The mistake everyone makes with knowledge graphs is trying to model everything from day one. Start with one use case, get it working, then expand - the graph grows with your understanding of what's useful, not with your ambition for what's possible.
Mak Khan
Chief AI Officer
When You Need a Knowledge Graph (and When You Don't)
You probably need one when:
- Questions require reasoning across multiple documents or concepts
- Relationships between entities are as important as the entities themselves
- Temporal reasoning matters (what changed, when, what it affects)
- Completeness matters (not just a relevant answer, but a complete one)
- The same information is asked about in different relationship contexts
You probably don't when:
- Questions are mostly direct factual lookups
- Your document collection is small (< 1,000 documents)
- Relationships between documents are simple or don't matter for your use cases
- RAG is delivering satisfactory results without structural enhancements
The Technology Landscape
As of mid-2024, the knowledge graph tooling for AI is maturing:
Graph databases: Neo4j remains the leader for enterprise knowledge graphs. Amazon Neptune is solid for AWS-native organisations. Both integrate with modern AI stacks.
Graph + vector hybrid: Some platforms now combine graph traversal with vector search natively. This eliminates the need to maintain separate systems for RAG and graph-based retrieval.
LLM-powered graph construction: Using LLMs to build and maintain knowledge graphs is an active area of development. It's promising but not yet reliable enough for production without human oversight.
Graph RAG frameworks: LangChain and LlamaIndex both have graph-enhanced retrieval capabilities. Not production-grade for complex enterprises, but useful for prototyping and simpler use cases.
Actionable Takeaways
- Don't abandon RAG. Knowledge graphs augment RAG; they don't replace it. Start with solid RAG, then add graph structure where relationships matter.
- Start with one use case. Build the graph for a specific, high-value problem. Compliance mapping, product-regulation relationships, or decision audit trails are good candidates.
- Invest in schema design. Get domain experts to define entity types and relationships before you start building. The schema is the foundation.
- Use LLMs for graph construction, cautiously. Automated entity and relationship extraction accelerates graph building, but requires human validation for accuracy.
- Measure the delta. A/B test answers with and without graph enhancement. Measure completeness, accuracy, and user satisfaction. If the graph doesn't improve outcomes, it's not worth the maintenance cost.