Challenges and opportunities
When managing a data lake or data warehouse
Manufacturing
The advantages of automation, AI, and machine learning have caused a shift toward smart factories and data-driven manufacturers. We help you to become competitive, innovative, and resilient.
Financial Services
How to navigate intense competition, stay in compliance with ever-changing regulations and leverage commercial opportunities. We help you create an IT landscape that enables efficient operations and connects processes and data.
Professional Services
People-centered, client-focused and project-driven: that’s professional services in a nutshell. We design and implement technology solutions that enable better decision-making and effective working.
Retail & Distribution
We modernize the retail workplace, bringing previously unseen insights to the business, implementing intelligent order management and get you closer to your customer.
Public Sector
HSO helps public and government organizations to meet and leverage increasing expectations for digital service, responsiveness and transparency.
Non-Profit
We support Non-Profits by modernizing applications, automating processes, and providing insight. Through technology, we nourish successful relationship management and enable greater impact.
What we do
HSO is one of the world’s top business solution and implementation partners, large enough to serve, small enough to care. Through our services, we help companies transform the way they work and improve business performance.
Business Applications
Microsoft Dynamics 365 Business Applications for ERP and CRM are leading in the industry. But their real strength is in the integration. Combined, they deliver even more value.
Cloud Services
HSO has a proven track record in supporting our customers in their cloud journey. With your business in mind, we offer a comprehensive suite of services to help you gain value from the cloud.
Data & AI
We help customers build an operational and future-proof data platform, ready to realize business value from AI and machine learning technologies.
Our Solutions
HSO develops industry-specific solutions that leverage the latest cloud and AI technologies from Microsoft, accelerating our customers' digital transformation.
Join HSO
Our workforce is a growing dynamic, unique, and diverse global community that is quality and performance-driven. For every stage in your career, we offer challenging opportunities. We look forward to meeting you!
Life at HSO
We are a global community of people with an entrepreneurial spirit, in an environment where you can really make a difference and have your voice heard. We constantly challenge ourselves to be better. Come join the journey with us!
Graduate Programs
Skyrocket your career with our graduate programs and accredited certification opportunities at HSO around the world!
HSO Academy
From technical skills, soft skills, mentorships, leadership coaching, masterclasses, informal knowledge sharing and globally recognised certifications - HSO Academy is your personal and professional knowledge accelerator.
Vacancies
Find opportunities in Consultancy & Technology, Project Management, Leadership, Sales and Business Development, Marketing, Legal and HR.
Join HSO
Our workforce is a growing dynamic, unique, and diverse global community that is quality and performance-driven. For every stage in your career, we offer challenging opportunities. We look forward to meeting you!
Life at HSO
We are a global community of people with an entrepreneurial spirit, in an environment where you can really make a difference and have your voice heard. We constantly challenge ourselves to be better. Come join the journey with us!
Graduate Programs
Skyrocket your career with our graduate programs and accredited certification opportunities at HSO around the world!
HSO Academy
From technical skills, soft skills, mentorships, leadership coaching, masterclasses, informal knowledge sharing and globally recognised certifications - HSO Academy is your personal and professional knowledge accelerator.
Vacancies
Find opportunities in Consultancy & Technology, Project Management, Leadership, Sales and Business Development, Marketing, Legal and HR.
Data Warehouses within organizations have for many years provided the insights that support important decisions. Although Massive Parallel Processing (MPP) architecture has enabled Data Warehouses to easily process large amounts of data, Data Warehouses are primarily focused on structured data.
As medium and large organizations increasingly deal with unstructured data as well as streaming data, they are running into the limitations of their Data Warehouses. The first blog in this series of two discusses what a Lakehouse architecture is and why it is the next step in data-driven processes within these organizations. In this second blog, we'll tell you how to go about migrating to a Lakehouse.
When an organization opts for a Lakehouse architecture to achieve a future-proof data platform, that is the moment to address shaping the migration. For the migration to be successful, proper preparation is required. This preparation consists of at least the following items:
After that, we cover the following topics:
1. Platform and security
After preparing steps 1 through 4, the first step in actually performing the migration is setting up the platform. This involves setting up the Landing Zones on which the data platform will be built. Microsoft recommends Data and Data Management Landing Zones for this purpose, part of the Cloud Adoption Framework. Part of the platform rollout also includes the security setup. Before the migration starts, it should be clear who has rights to do what. This is central to setting up the help of firewalls, Network Security Groups and Private Endpoints, among others.
Once the platform is in place, it's time for the data migration. This contains two sub-steps: migrating the history and setting up the loading processes. In each, we recommend a side-by-side migration over an in-place migration. This involves the new data platform being set up alongside the existing platform, making testing and validation easy. This way, you can easily compare the newly established data platform 1-to-1 with the already existing platform.
Migrating history
When migrating history, all relevant data is copied from the original Data Warehouse into the Data Lake. When data is copied from a Data Warehouse, the source is usually a database environment. To access these, we recommend metadata-driven extraction. This involves using an ETL tool to create one copy per table in the Data Lake, in a predefined structure.
Setting up the loading processes
When setting up the loading processes, it is necessary to keep the data from both environments the same by updating them. To do this, the loading processes from the Data Warehouse must be set up in the Data Lake. For this, too, we recommend a metadata-driven solution, which in this case connects directly to the source: the Data Warehouse. When migrating from an on premise solution to a cloud solution, a Gateway is required in many cases.
The next step in migrating from a Data Warehouse to a Lakehouse is to convert the data transformations. It too contains two sub-steps: migration of transformation processes and rebuilding of data processes. The decision for this should already be made at the stage and depends on the answers to the following questions:
If rebuilding is chosen, this is the time to also review data layering. Many Lakehouses use a medallion architecture, with Bronze, Silver and Gold layers.
Whether migration or rebuilding of data transformations is chosen, the location where the transformed data is stored must be changed in any case. This is different for a Lakehouse than for a Data Warehouse. The transformed data in the Silver and Gold layers is also stored back in the Data Lake.
4. Data products
Once the data and transformation processes are successfully migrated, the same datasets are available in the new environment. This is the time to convert the data products to the new environment. A distinction can be made in this between managed reporting and self-service analytics.
In managed reporting, the data products are managed by a central reporting team. This team can take care of converting the data itself, by referring the data products to the new environment. For self-service use, users should be informed of the change in three stages:
Challenges and opportunities
When managing a data lake or data warehouse
Want to learn more about the key benefits and challenges of Lakehouse architecture, or get started right away? Connect with one of our Data & Analytics experts.
Read more
These customers rely on our data & analytics expertise
We, and third parties, use cookies on our website. We use cookies to keep statistics, to save your preferences, but also for marketing purposes (for example, tailoring advertisements). By clicking on 'Settings' you can read more about our cookies and adjust your preferences. By clicking 'Accept all', you agree to the use of all cookies as described in our privacy and cookie policy.
Purpose
This cookie is used to store your preferences regarding cookies. The history is stored in your local storage.
Cookies
Location of Processing
European Union
Technologies Used
Cookies
Expiration date
1 year
Why required?
Required web technologies and cookies make our website technically accessible to and usable for you. This applies to essential base functionalities such as navigation on the website, correct display in your internet browser or requesting your consent. Without these web technologies and cookies our website does not work.
Purpose
These cookies are stored to keep you logged into the website.
Cookies
Location of Processing
European Union
Technologies Used
Cookies
Expiration date
1 year
Why required?
Required web technologies and cookies make our website technically accessible to and usable for you. This applies to essential base functionalities such as navigation on the website, correct display in your internet browser or requesting your consent. Without these web technologies and cookies our website does not work.
Purpose
This cookie is used to submit forms to us in a safe way.
Cookies
Location of Processing
European Union
Technologies Used
Cookies
Expiration date
1 year
Why required?
Required web technologies and cookies make our website technically accessible to and usable for you. This applies to essential base functionalities such as navigation on the website, correct display in your internet browser or requesting your consent. Without these web technologies and cookies our website does not work.
Purpose
This service provided by Google is used to load specific tags (or trackers) based on your preferences and location.
Why required?
This web technology enables us to insert tags based on your preferences. It is required but adheres to your settings and will not load any tags if you do not consent to them.
Purpose
This cookie is used to store your preferences regarding language.
Cookies
Why required?
We use your browser language to determine which language to show on our website. When you change the default language, this cookie makes sure your language preference is persistent.
Purpose
This service provided by uMarketingSuite is used to track anonymized analytics on the HSO.com application. We find it very important that your privacy is protected. Therefore, we collect and store this data anonymously on our own servers. This cookie helps us collect data from HSO.com so that we can improve the website. Examples of this are: it allows us to track engagement by page, measuring various events like scroll-depth, time on page and clicks.
Cookie
Purpose
With your consent, this website will load Google Analytics to track behavior across the site.
Cookies
Purpose
With your consent, this website will load the Google Advertising tag which enables HSO to report user activity from HSO.com to Google. This enables HSO to track conversions and create remarketing lists based on user activity on HSO.com.
Possible cookies
Please refer to the below page for an updated view of all possible cookies that the Google Ads tag may set.
Cookie information for Google's ad products (safety.google)
Technologies Used
Cookies
Purpose
With your consent, we use IPGeoLocation to retrieve a country code based on your IP address. We use this service to be able to trigger the right web technologies for the right people.
Purpose
With your consent, we use Leadfeeder to identify companies by their IP-addresses. Leadfeeder automatically filters out all users visiting from residential IP addresses and ISPs. All visit data is aggregated on the company level.
Cookies
Purpose
With your consent, this website will load the LinkedIn Insights tag which enables us to see analytical data on website performance, allows us to build audiences, and use retargeting as an advertising technique. Learn more about LinkedIn cookies here.
Cookies
Purpose
With your consent, this website will load the Microsoft Advertising Universal Event Tracking tag which enables HSO to report user activity from HSO.com to Microsoft Advertising. HSO can then create conversion goals to specify which subset of user actions on the website qualify to be counted as conversions. Similarly, HSO can create remarketing lists based on user activity on HSO.com and Microsoft Advertising matches the list definitions with UET logged user activity to put users into those lists.
Cookies
Technologies Used
Cookies
Purpose
With your consent, this website will load the Microsoft Dynamics 365 Marketing tag which enables HSO to score leads based on your level of interaction with the website. The cookie contains no personal information, but does uniquely identify a specific browser on a specific machine. Learn more about Microsoft Dynamics 365 Marketing cookies here.
Cookies
Technologies Used
Cookies
Purpose
With your consent, we use Spotler to measures more extensive recurring website visits based on IP address and draw up a profile of a visitor.
Cookies
Purpose
With your consent, this website will show videos embedded from Vimeo.
Technologies Used
Cookies
Purpose
With your consent, this website will show videos embedded from Youtube.
Cookies
Technologies Used
Cookies
Purpose
With your consent, this website will load the Meta-pixel tag which enables us to see analytical data on website performance, allows us to build audiences, and use retargeting as an advertising technique through platforms owned by Meta, like Facebook and Instagram. Learn more about Facebook cookies here. You can adjust how ads work for you on Facebook here.
Cookies
Purpose
With your consent, we use LeadInfo to identify companies by their IP-addresses. LeadInfo automatically filters out all users visiting from residential IP addresses and ISPs. These cookies are not shared with third parties under any circumstances.
Cookies
Purpose
With your consent, we use TechTarget to identify companies by their IP address(es).
Cookies
Purpose
With your consent, we use this service provided by uMarketingSuite to run A/B tests across the HSO.com application. A/B testing (also called split testing) is comparing two versions of a web page to learn how we can improve your experience.
Purpose
With your consent, we use this service provided by uMarketingSuite to personalize pages and content across the HSO.com application. Personalization helps us to tailor the website to your specific needs, aiming to improve your experience on HSO.com.