By Preetham Michael, Client Partner
In last week’s post, we discussed the four “R’s” of application migration. In this piece, let’s dig in to some of the strategies and best practices for data warehouse migrations to the cloud.
Did you know? According to Gartner, 83% of data migrations fail or exceed their budgets and schedules.
Migrating to the cloud is not a matter of if, but when. The days of managing physical servers in data centers is soon to be over. What’s causing this shift? A few of the key reasons that organizations decide to migrate to the cloud include:
- New Data Sources: There are new data sources that have come into play and organizations need these sources to make swift business decisions.
- Legacy Platforms: Legacy platforms don’t support business needs for scalability, unlimited on-demand computer, nor some of the newer data types that have come into existence.
- Cost: The pay-as-you-go model, available on the cloud, enables organizations flexibility to monitor costs.
Now that your organization has decided to make the investment to migrate your on-premises data warehouse to the cloud, let’s explore the two common approaches used – lift and shift or build a new cloud data warehouse (a.k.a. transformation approach).
Lift and Shift
Depending on the maturity, nature, and associated costs of a current data platform, the Lift and Shift approach may be a viable option. This is a low cost approach and can be achieved relatively quickly. As part of the migration strategy, some other considerations include source and destination database architectures. Fundamentally, there are two architectures that databases are divided into: Shared Disk and Shared Nothing. The data distribution of these two architectures are vastly different and therefore impact the overall migration strategy.
Fortunately, some of the popular cloud data warehouses like Snowflake have addressed some of the limitations of these architectures.
Best practices for a lift and shift approach include:
- Identify Current Landscape: Many organizations don’t have their current systems documented nor do they know what parts of their data are actually being used. Query usage is one of the easiest ways to identify how often an organization uses data. This can also be a great time to clean up your code base and eliminate unused code.
- Migrate What is Needed: Once data usage is identified, simply migrate the necessary data!
- Identify Data Types: Identify data type mismatches up front and have a plan to address them. When mismatches occur, it can cause several inaccuracies in reporting.
- Data Quality: One of the key pillars of successful data migration is having a solid data quality framework in place. Over 44% of data migration projects have been delayed due to data quality issues (!).
Transformational Approach
A complete transformational approach is where you leave your current on-prem data warehouse as-is to “keep lights on” and build your cloud data warehouse from scratch. What was necessary and valuable to a business 10 years ago may not be so now; building a new cloud data warehouse is a journey that organizations should be willing to embrace. Start small and be flexible.
A few other considerations to keep in mind: make sure that your architecture is future-proof to meet exponential data growth demands and have a data governance framework that includes data cataloging with a solid metadata management framework.
At Paradigm, in addition to helping organizations build new cloud data warehouses, we have enabled them to migrate their data warehouses to the cloud successfully. We have several accelerators and solutions that enable these solutions. Reach us at info@pt-corp.com to learn more.