Most enterprise AI programmes measure deployment, not transfer. They know how many people have access to the AI tool. They know how many have logged in. They may even know how many queries have been submitted. What they rarely know is whether anyone has actually learned to work with AI effectively. Tania and I have been developing measurement approaches that distinguish between tool access and genuine knowledge transfer.
What You Need to Know
- Tool usage metrics (login rates, query counts) are poor proxies for knowledge transfer. High query counts can indicate effective use or confused experimentation.
- Knowledge transfer in AI adoption has three measurable dimensions: comprehension (do they understand what the tool does?), application (can they use it in their workflow?), and adaptation (can they modify their approach when context changes?)
- Pre-post assessment with delayed follow-up is the most reliable measurement approach for enterprise AI knowledge transfer
- Organisations that measure knowledge transfer detect adoption problems 2-3 months earlier than those that only measure usage metrics
Why Usage Metrics Mislead
The Login Fallacy
"85% of staff have logged into the AI tool" tells you that 85% of people clicked a link. It does not tell you whether they found the tool useful, learned anything, or intend to use it again. Login rates are a measure of compliance, not transfer.
The Query Count Illusion
High query counts can mean effective use. They can also mean users are repeatedly trying to get useful results and failing. Without quality data alongside quantity data, query counts are ambiguous.
Course enrolment is not learning. Attendance is not learning. Even assignment completion is not learning. Learning is demonstrated when someone can apply knowledge in a new context.
Dr Josiah Koh
Education & AI Innovation
The Satisfaction Trap
Post-training satisfaction surveys measure how people felt about the training, not what they learned from it. A highly rated AI workshop that doesn't translate into changed behaviour is an entertainment event, not a learning intervention.
The Three Dimensions of Knowledge Transfer
Dimension 1: Comprehension
Can the person explain what the AI tool does, what it needs to work well, and what its limitations are?
How to measure: Short assessment (5-10 questions) administered before training, immediately after, and at 30 and 90 days. The questions should test conceptual understanding, not recall of specific facts.
What to look for: Scores that remain stable or improve between the immediate post-training assessment and the 30-day follow-up. Significant decline indicates the training created temporary understanding that didn't convert to lasting knowledge.
Dimension 2: Application
Can the person use the AI tool effectively in their actual work context?
How to measure: Task-based assessment using scenarios from the person's real workflow. Ask them to complete a representative task using the AI tool, then evaluate the process (not just the output).
What to look for: Efficient tool use, appropriate prompt construction, critical evaluation of outputs, and correct handling of tool limitations. Compare performance to a pre-training baseline.
Dimension 3: Adaptation
Can the person modify their AI use when the context changes: new task types, updated tools, different requirements?
How to measure: Present a novel scenario that requires adapting AI skills to a new context. Measure whether the person can transfer their learning rather than only applying it to the exact tasks covered in training.
What to look for: Flexible problem-solving, willingness to experiment, and ability to evaluate unfamiliar AI outputs. This is the highest-value dimension because it predicts ongoing capability development.
Practical Measurement Approaches
Pre-Post With Delayed Follow-Up
Assess before training (baseline), immediately after (learning), and at 30 and 90 days (retention and transfer). This design separates learning effects from retention effects and reveals whether knowledge is durable.
Workflow Observation
Observe actual AI tool use in context, with the user's permission and in a non-evaluative frame. Look for: effective prompt construction, critical evaluation of outputs, appropriate workflow integration, and recovery from tool errors.
Tania's research methodology brings rigour here: structured observation protocols, consistent evaluation criteria, and inter-rater reliability checks ensure the observations produce reliable data.
Peer Assessment
Ask team members to evaluate each other's AI capability. Peer assessment captures information that self-assessment and formal testing miss, because peers observe each other's daily practice.
Measuring knowledge transfer is harder than measuring tool deployment. But it answers the question that actually matters: are people learning to work with AI, or are they just logging in? The organisations that answer this question accurately are the ones that can invest in the right interventions, at the right time, for the right people.