adrift in the sea of experience

Tuesday, May 20, 2008

Lessons from IT projects: migrating existing data to a new system

I'm currently working on a redesign/reimplementation/integration of an administrative system that currently exists as a mix of 3 databases (with semantically vague and manually maintained links between them), two applications and lord knows how many excel spreadsheets. The goal is to have 1 database and 1 application, less room for mistakes, and drastically reduced administrative overhead.

As I approach the deadline for this project, I often find myself thinking "If only I had known that ..." or "I should remember X next time".

A first observation: Migrating old data to a new system is harder than you think. It forces you to follow stupid constraints for the new system, because otherwise the old data can't be expressed in the new system. Imagine trying to build a house in the middle of the forest without being allowed to cut down certain trees. You'll end up with alot of extra walls to build around the forbidden trees, and your shiny new house will start to look like a maze.

If at all possible, go for a green-field scenario where the data can be left behind or is migrated gradually/manually by the customer. After all, it's the customer who knows about the old data and business rules. If you absolutely need to migrate the existing data, don't underestimate the effort of having to map out the existing organically evolved "solution". That may be a project in it's own right.

No comments: