Every enterprise platform rebuild includes a data migration. It's always in the plan. It's always underscoped. And it's the single most common reason enterprise rebuilds run late. Not because the new system is hard to build, but because getting the old data into the new system is harder than anyone expected.
What You Need to Know
- Data migration is consistently the most underestimated phase of enterprise rebuilds
- The technical challenge is rarely the transfer itself. It's understanding, cleaning, and transforming data that was never documented
- Budget 20-30% of total project time for data migration on any rebuild project
- Start the migration analysis in discovery, not after the new system is built
Why It's Always Underscoped
The Optimism Problem
Isaac: The migration conversation in sales and scoping meetings goes like this: "We'll export the data from the old system, transform it, and import it into the new system." Everyone nods. It sounds straightforward. Three steps. How hard can it be?
The answer, consistently, is much harder than anyone expects. And the reason is that the scoping conversation happens before anyone has actually looked at the data.
The Data Quality Problem
John: Legacy systems accumulate data quality issues over years. Fields that were mandatory in 2012 but made optional in 2015. Records that were entered by different teams with different conventions. Phone numbers stored as text with inconsistent formatting. Dates in three different formats because three different integrations wrote to the same table.
None of this is visible from a schema diagram. You only discover it when you start reading the actual data. And what you discover is almost always worse than what you expected.
30%
of enterprise data typically fails quality checks during migration
Source: Gartner Data Quality Market Survey, 2018
The Undocumented Logic Problem
Legacy systems contain business logic that nobody documented and the original developers have left the organisation. Status codes that mean different things depending on which module created the record. Foreign keys that point to records that no longer exist. Archived data in a format that the current system doesn't support.
The hardest part of data migration isn't moving data. And in legacy systems, the only documentation is often the code itself.
John Li
Chief Technology Officer
What Actually Goes Wrong
1. The Schema Mismatch
The old system and the new system model the world differently. The old system stored addresses as a single text field. The new system has structured address fields. The old system used integer status codes. The new system uses descriptive enums. Every mismatch requires a transformation rule, and every transformation rule has edge cases.
2. The Volume Surprise
The old system has 4 million records. The migration script that worked perfectly on 10,000 test records takes 36 hours to run against the full dataset. Or it runs fine until record 2.3 million, where a malformed entry crashes the transform.
3. The Relationship Tangle
Enterprise data is relational. Customers have orders. Orders have line items. Line items reference products. Products belong to categories. Migrating any one of these tables requires the others to exist first. Circular dependencies create chicken-and-egg problems that require careful staging.
4. The Parallel Running Nightmare
The old system and the new system need to run in parallel during transition. Data entered in the old system during transition needs to be synchronised to the new system. But the two systems model data differently. Bidirectional synchronisation between different schemas during a transition period is one of the most complex integration challenges in enterprise IT.
How to Budget for It
Start in Discovery
Isaac: We now include data migration analysis in every discovery phase for rebuild projects. Before we've designed a single screen, we're looking at the source data. How much? In what format? What quality? What undocumented logic?
This analysis consistently reveals scope that would otherwise surface three months into the build, when it's too late to adjust the timeline.
Budget 20-30% of Project Time
John: For any enterprise rebuild that includes data migration, we budget 20-30% of the total project time for migration work. This includes analysis, transformation development, testing, and the migration execution itself.
On a six-month project, that's six to eight weeks dedicated to migration. This sounds like a lot. On every project where we've allocated this time, we've used it.
Test with Real Data Early
Don't wait until the new system is complete to test migration. Run trial migrations as early as possible, against real data extracts from the source system. Each trial migration will reveal issues: data quality problems, transformation edge cases, performance bottlenecks.
We typically run three to four trial migrations before the real one. Each trial gets cleaner and faster. By the final migration, the process is proven and the team is confident.
Plan the Cutover
The cutover from old system to new system needs a detailed plan. What time does it start? How long can the old system be offline? What's the rollback procedure if the migration fails? Who validates the data after migration? What's the acceptance criteria?
This plan should be documented and rehearsed. A data migration is a one-time, high-stakes operation. You don't get to practice in production.
The Conversation with the Client
Isaac: The most important thing we do regarding data migration is have the conversation early. "Your data is messy. Not because anyone did anything wrong, but because all enterprise data accumulates quality issues over time. We need to plan for that, and it needs real time in the project plan."
Clients appreciate this honesty. They've usually suspected their data quality was an issue but haven't had someone quantify it. Showing them the analysis from discovery, with specific examples of the challenges their data presents, builds trust and sets realistic expectations.
The alternative is not having the conversation. And then having a much harder conversation three months later when the project is running late because the migration is taking twice as long as the two weeks allocated in the original plan.