I've sat through enough steering committee meetings to know the pattern. Someone presents a velocity chart. Someone else shows a burndown. A third person shares a RAG status. Everyone nods. Nobody knows whether the project is actually succeeding. We're measuring activity, not outcomes, and the difference matters more than most teams realise.
What You Need to Know
- Velocity and story points measure team activity, not project value. High velocity with wrong priorities delivers nothing useful
- Outcome-based measurement starts with defining what success looks like before the project begins
- Three outcome layers matter: delivery outcomes (did we build it?), adoption outcomes (do people use it?), and impact outcomes (did it change anything?)
- The best measurement frameworks are simple, visible, and updated frequently enough to influence decisions
The Velocity Trap
Velocity is the most commonly tracked metric in agile delivery, and it's almost entirely useless as a measure of project success.
A team can have increasing velocity, completing more story points every sprint, while building the wrong features, ignoring technical debt, and heading towards a delivery that nobody wants. Velocity tells you how fast the engine is running. It says nothing about whether you're driving in the right direction.
70%
of enterprise projects track velocity as a primary metric, but only 30% track user adoption post-launch
Source: VersionOne State of Agile Report, 2021
Story points have a related problem. They're meant to be relative estimates of effort, but in practice they become a currency. Teams negotiate them, inflate them, and game them. Managers track them as if they're comparable across teams, which they aren't. The metric that was supposed to help with planning becomes a performance measurement tool it was never designed to be.
When you measure activity instead of outcomes, you get more activity. You don't necessarily get better results.
Dr Tania Wolfgramm
Chief Research Officer
Three Outcome Layers
Useful measurement operates at three levels. Each one answers a different question.
Delivery Outcomes: Did We Build It?
This is the layer most teams already measure, though often poorly. Did the features ship? Were they on time? Did they meet the acceptance criteria? Quality metrics sit here too: defect rates, test coverage, performance benchmarks.
Delivery outcomes are necessary but not sufficient. A project can deliver every feature on time and still fail if nobody uses them.
Adoption Outcomes: Do People Use It?
This is where most measurement frameworks fall over. The project ships, the team celebrates, and nobody checks whether the target users actually changed their behaviour. Adoption metrics include: active user counts, feature usage rates, time-to-task-completion, and support ticket volumes.
45%
of enterprise software features are rarely or never used after launch
Source: Standish Group, 2020
If you're not measuring adoption, you're measuring delivery in a vacuum. We've seen projects that shipped every feature on time and within budget, only to discover six months later that the users went back to their spreadsheets because the new system was harder to use.
Impact Outcomes: Did It Change Anything?
This is the layer that connects the project to the business case. Did the new system reduce processing time? Did the customer portal decrease call centre volume? Did the workflow automation save the hours it promised?
Impact measurement takes longer and is harder to attribute. Other factors change alongside the new system. But without even attempting it, you can't close the loop between investment and return.
Building an Outcome Framework
A practical outcome measurement framework doesn't need to be complicated. It needs four things.
Clear success criteria defined before the project starts. Not "improve the customer experience" but "reduce average onboarding time from fourteen days to five days within six months of launch." Specific, measurable, time-bound.
Baseline measurements taken before the project begins. If you don't know where you started, you can't measure how far you've moved. This step gets skipped constantly because it requires effort before the build begins, when everyone is impatient to start building.
Regular check-ins against outcomes, not just delivery milestones. Monthly reviews that ask "are we still solving the right problem?" not just "are we on schedule?"
Post-launch measurement baked into the project scope. The project isn't done when it ships. Budget for three to six months of post-launch measurement. If you can't afford it, you can't afford to know whether the project worked.
What to Measure (Practically)
For most enterprise projects, start with these five metrics:
- Feature adoption rate - what percentage of target users are actively using the new system within 30/60/90 days?
- Task completion time - for the core workflows the system was built to support, how long do they take now vs before?
- Error/rework rate - are people getting it right the first time, or are mistakes increasing?
- Support volume - are support requests going up (bad sign) or down (good sign) post-launch?
- User satisfaction - a simple quarterly survey. Not NPS, just "does this system help you do your job?"
None of these are revolutionary. All of them are more useful than a velocity chart.