There are multiple reasons why an organisation would undertake a data migration project. For example, it could be moving data to a third-party cloud provider, or retiring legacy systems due to discontinued support from its system provider.
In fact, legacy systems are one of the key drivers of data migrations due to:
- Challenges finding staff with the right skills to maintain the system
- Outdated functionality creating negative user experiences and limiting innovation
- Inability to keep up with growing data storage requirements
- Inefficiencies that increase the risk of data loss and security threats
Despite being a well-established discipline, most data migration projects overrun on time and budget, thereby impacting business transformations. Too often companies ignore some subtle but important data-specific considerations when planning data migrations.
There’s also a lack of understanding of what data migration actually is. A common definition is, “the process of moving or transferring data from one storage system to another”. However, in my view it is actually the process of copying data from one storage system to another (or multiple). This difference is very important to understand, as it will impact the scope and activities that a project team undertakes as part of a migration exercise.
Of course, the reality is often much more complex. Data generated in one system is typically shared with multiple systems to meet various business needs. Similarly, we often don’t migrate all data from the legacy system during a migration project as some data will no longer be required.
It is also worth noting that the data is generated and used by the business processes, managed and applied by the business users, and transported by the interfaces between the systems. These processes, users and interfaces are what’s actually transferred to new systems during a migration, as shown below:
Ultimately, data migrations aren’t always fully understood and are often more complex than they appear. That’s why it is important to always start with the ‘why?’ What are the business reasons, needs and implications for migrating all or some of your data? Answering this question will stand you in good stead for the rest of the project.
But that’s not all you have to think about. Here are ten considerations that can help make your next data migration project a success.
1) Discover and analyse legacy estate
One of the reasons data migration projects get delayed is the scope creep that can occur due to a lack of visibility of the legacy estate – especially data, interfaces and systems. Project teams often uncover new items during the process that could have considerable impact on the delivery. Therefore, it is imperative to perform discovery at the outset to minimise the chance of unexpected scope and budget creep further down the line.
2) Understand the differences between legacy and target
Since migration is the transfer of data from one system to another, it is important to understand the similarities and differences between the legacy and target systems. Particularly important is how they store and manage data, and what native functionality they offer to process data. This will help to map the source to target, identify gaps in data and functionality, and define approaches to address those gaps.
3) Separate data migration from business change
One common mistake businesses make is combining these two activities. For example, banks often combine interest calculation changes (e.g moving to daily calculation instead of monthly or yearly) with data migrations. This type of business change requires communication with users, which creates a headache as any problems will necessitate additional communication to satisfy regulatory requirements. This can have a negative impact on business reputation. That’s why it is advisable to separate business change from data migration. That way, if the project fails or needs to be re-planned, you don’t have to tell users as it can be classed as system maintenance.
4) Think about “all” data
There’s a general tendency to focus too much on operational and active data and either completely forget about archived data or leave it for a later stage. We must review and make decisions about all data impacted by the migration activity, as any forgotten data could cause problems later on – especially if it is stored in an old system that doesn’t integrate well with newer systems and isn’t as easy to access.
5) It is not just data being migrated
Remember, what actually gets migrated are the legacy system users, processes, and its upstream and downstream interfaces (data flows). These must be identified, analysed and properly migrated along with the data, as any issues with their migration will impact the migrated data. In most cases, changes to these are inevitable and must be considered and planned for accordingly. For example some interfaces may no longer be needed, certain users won’t need access to the new system etc.
6) Evaluate any direct and indirect impacts
In the modern world, systems do not work in isolation and are connected through interfaces with other systems. Therefore, when performing migrations, ensure that any potential impact on connected upstream and downstream systems is analysed, understood and taken care of.
7) An opportunity to improve data quality
Remember the old adage “garbage in garbage out”! Use the migration process as a data quality improvement opportunity to avoid transferring quality issues to the target system(s). Where possible, remediate data quality issues prior to actual migration.
8) Think about compliance
Processing data of any sort comes under the compliance radar, especially PII (personally identifiable information). Non-compliance could result in heavy fines and damage to business reputation. Therefore, ensure that compliance requirements are understood and addressed. Involve data governance teams from the outset.
9) Govern changes to data
Data migrations involve many data decisions, ranging from what to migrate and what to leave behind to how to remediate data quality issues. To make such decisions, create a data board for your project that has the required expertise and authority. Establish a clear escalation route for decisions that require input from business stakeholders who understand the key data requirements. Decisions made without this oversight could prove costly.
10) Decommission Legacy
Data migration is complex and time consuming. As a result, we often forget about decommissioning legacy systems. As well as leaving the migration unfinished, this also presents the risk of the legacy system(s) becoming unsupported shadow IT. Over time, dependencies will be created and it will become difficult to decommission systems, so plan this in advance and execute once all required data has been successfully migrated to the new system(s).
Take all these considerations into account, and you’ll be much better placed to reduce the challenges and complications that can arise with data migrations. If you’re interested in learning about cloud migrations, take a look at this blogfrom my colleague Paul Swinfield.
Explore our full range of Digital and Data Services
Stay up to date with our experts' perspectives on the latest trends,
issues and technologies sent directly to your inbox.