Last week I attended the Data Migration Matters conference http://datamigrationmatters.co.uk/ in London and what I learned was that there are differences in approach between integration and migration, however there are common factors, two of which I will cover in this blog.
The first is customer involvement. The data in an organisation is utilised by the business, defined by the business and ultimately informs business decisions, so any project, IT or otherwise that is required to make changes to data ultimately needs business buy in and involvement.
The second is understanding the data. Many projects have failed because of incorrect assumptions and gaps in knowledge that only manifest themselves when a project is in full flight. It is imperative that the source data is mapped out and understood prior to coding.
In some ways these two requirements go hand in hand. To understand and make sense of the data, you need the business to add their experience of using it to the mix. To involve the business, you have to be able to deliver technical information in such a way that it becomes possible to interpret the data in a non-technical way.
This is where data profiling and quality tools come into their own. These tools analyse the source data and present the user with a high level view of the data, enabling the user to view patterns and statistics and relationships at both file and field level.
Profiling information is often a revelation for business users. Operationally the data works, so is deemed fit for purpose, however when profiled it is not uncommon to see genuine problems with the data, such as duplicate records, missing fields and records and often just plain incorrect values. The ability also for drill down to the actual data is imperative in order to show that the statistics marry up to real data on real systems within the organisation.
It is often at this point, when the illusion of “perfect data” evaporates, that the business buys into the project and begins to understand why the ownership of the data and the associated business rules fall squarely within their domain. It is surprising how showing people their own data “warts and all” can have a profound effect on their involvement in a project.
How often have we heard the term “if it isn’t broken, don’t fix it” and for many users their opinion is that their data isn’t broken, so it is perhaps hard to understand why IT make such a fuss during a data migration.
The truth is that for many organisations their data is, to varying degrees, somewhere between broken and fixed and it is only when it is utilised en masse, say for reporting or migration, that problems suddenly begin to appear.
