Challenges and opportunities
When managing a data lake or data warehouse
Manufacturing
De snelle ontwikkelingen in automatisering, AI en machine learning hebben een shift veroorzaakt naar smart en data-driven manufacturing. Wij helpen jou meer waarde uit nieuwe technologie te halen.
Financiële dienstverlening
Verstevig je concurrentiepositie in dynamisch speelveld en zorg tegelijkertijd da je voldoet aan de steeds veranderende regelgeving. HSO bouwt een compleet en geïntegreerd applicatieplatform dat efficiëntie en innovatie mogelijk maakt en processen en data verbindt.
Professionele dienstverlening
Mensgericht, klantgericht en projectgedreven: dat is professionele dienstverlening in een notendop. Onze technologische oplossingen maken betere besluitvorming, business development en effectief werken mogelijk.
Retail
Een optimale customer experience, het verstevigen van customer loyalty en slim supply chain management en fulfillment: onze retail- en distributieoplossingen brengen intelligentie, inzicht en efficiëntie in de complete keten.
Publieke sector
Non-Profit
Wij ondersteunen non-profitorganisaties door applicaties te moderniseren, processen te automatiseren en inzicht te verschaffen. Intelligente technologie maakt succesvol relatiebeheer en meer impact mogelijk.
Wat we doen
HSO heeft de kennis, ervaring en oplossingen die je nodig hebt om te groeien en voorop te lopen in jouw markt.
Business Applications
De producten en oplossingen uit de Microsoft Cloud, inclusief Dynamics 365 zijn leidend in de markt. Maar de werkelijke kracht van Dynamics 365 Business Applications zit 'm in de integratie.
Cloud Services
HSO heeft een succesvol track record in het begeleiden van organisaties naar de cloud. Op basis van de wensen en behoeften van jouw organisatie bieden we een uitgebreide set diensten die je helpen om meer waarde uit de cloud te halen.
Data & AI
Wij helpen organisaties de kracht van data te benutten en succesvol te zijn met AI dankzij een dataplatform dat flexibel, schaalbaar en betrouwbaar is.
Onze Producten
HSO ontwikkelt eigen producten om gaps in de processen en technologie van Microsoft Dynamics 365 op te vullen. We leveren nieuwe apps, verbeterde processen en industriegedreven best practices.
Join HSO
HSO biedt meer dan een carrière in IT. Het is een kans om je doel te ontdekken, ambities waar te maken en een wereldwijde impact te maken. Join the journey!
Life at HSO
Bij HSO werken betekent meer dan alleen een stap in je carrière; het gaat om het vinden van je doel en het maken van een betekenisvolle reis op wereldschaal.
Graduate Programma's
Geef je carrière een kickstart met onze programma's voor net-afgestudeerden en starters.
HSO Academy
Van technische vaardigheden, soft skills, leiderschapscoaching, masterclasses, kennisdeling en erkende certificeringen - HSO Academy geeft je een persoonlijke en professionele kennis boost.
Alle vacatures
Vind kansen in Consultancy & Technologie, Projectmanagement, Leiderschap, Sales en Business Development, Marketing, Legal en HR.
Join HSO
HSO biedt meer dan een carrière in IT. Het is een kans om je doel te ontdekken, ambities waar te maken en een wereldwijde impact te maken. Join the journey!
Life at HSO
Bij HSO werken betekent meer dan alleen een stap in je carrière; het gaat om het vinden van je doel en het maken van een betekenisvolle reis op wereldschaal.
Graduate Programma's
Geef je carrière een kickstart met onze programma's voor net-afgestudeerden en starters.
HSO Academy
Van technische vaardigheden, soft skills, leiderschapscoaching, masterclasses, kennisdeling en erkende certificeringen - HSO Academy geeft je een persoonlijke en professionele kennis boost.
Alle vacatures
Vind kansen in Consultancy & Technologie, Projectmanagement, Leiderschap, Sales en Business Development, Marketing, Legal en HR.
Data Warehouses within organizations have for many years provided the insights that support important decisions. Although Massive Parallel Processing (MPP) architecture has enabled Data Warehouses to easily process large amounts of data, Data Warehouses are primarily focused on structured data.
As medium and large organizations increasingly deal with unstructured data as well as streaming data, they are running into the limitations of their Data Warehouses. The first blog in this series of two discusses what a Lakehouse architecture is and why it is the next step in data-driven processes within these organizations. In this second blog, we'll tell you how to go about migrating to a Lakehouse.
When an organization opts for a Lakehouse architecture to achieve a future-proof data platform, that is the moment to address shaping the migration. For the migration to be successful, proper preparation is required. This preparation consists of at least the following items:
After that, we cover the following topics:
1. Platform and security
After preparing steps 1 through 4, the first step in actually performing the migration is setting up the platform. This involves setting up the Landing Zones on which the data platform will be built. Microsoft recommends Data and Data Management Landing Zones for this purpose, part of the Cloud Adoption Framework. Part of the platform rollout also includes the security setup. Before the migration starts, it should be clear who has rights to do what. This is central to setting up the help of firewalls, Network Security Groups and Private Endpoints, among others.
Once the platform is in place, it's time for the data migration. This contains two sub-steps: migrating the history and setting up the loading processes. In each, we recommend a side-by-side migration over an in-place migration. This involves the new data platform being set up alongside the existing platform, making testing and validation easy. This way, you can easily compare the newly established data platform 1-to-1 with the already existing platform.
Migrating history
When migrating history, all relevant data is copied from the original Data Warehouse into the Data Lake. When data is copied from a Data Warehouse, the source is usually a database environment. To access these, we recommend metadata-driven extraction. This involves using an ETL tool to create one copy per table in the Data Lake, in a predefined structure.
Setting up the loading processes
When setting up the loading processes, it is necessary to keep the data from both environments the same by updating them. To do this, the loading processes from the Data Warehouse must be set up in the Data Lake. For this, too, we recommend a metadata-driven solution, which in this case connects directly to the source: the Data Warehouse. When migrating from an on premise solution to a cloud solution, a Gateway is required in many cases.
The next step in migrating from a Data Warehouse to a Lakehouse is to convert the data transformations. It too contains two sub-steps: migration of transformation processes and rebuilding of data processes. The decision for this should already be made at the stage and depends on the answers to the following questions:
If rebuilding is chosen, this is the time to also review data layering. Many Lakehouses use a medallion architecture, with Bronze, Silver and Gold layers.
Whether migration or rebuilding of data transformations is chosen, the location where the transformed data is stored must be changed in any case. This is different for a Lakehouse than for a Data Warehouse. The transformed data in the Silver and Gold layers is also stored back in the Data Lake.
4. Data products
Once the data and transformation processes are successfully migrated, the same datasets are available in the new environment. This is the time to convert the data products to the new environment. A distinction can be made in this between managed reporting and self-service analytics.
In managed reporting, the data products are managed by a central reporting team. This team can take care of converting the data itself, by referring the data products to the new environment. For self-service use, users should be informed of the change in three stages:
Challenges and opportunities
When managing a data lake or data warehouse
Want to learn more about the key benefits and challenges of Lakehouse architecture, or get started right away? Connect with one of our Data & Analytics experts.
Read more
These customers rely on our data & analytics expertise
Wij, en derde partijen, gebruiken cookies op onze website. We gebruiken cookies om statistieken bij te houden, uw voorkeuren op te slaan, maar ook voor marketingdoeleinden (bijvoorbeeld het op maat aanbieden van advertenties). Door op 'Instellingen' te klikken kunt u meer lezen over onze cookies en uw voorkeuren aanpassen. Door op 'Alles accepteren' te klikken, ga je akkoord met het gebruik van alle cookies zoals beschreven in onze privacy- en cookie policy.
Purpose
This cookie is used to store your preferences regarding cookies. The history is stored in your local storage.
Cookies
Location of Processing
European Union
Technologies Used
Cookies
Expiration date
1 year
Why required?
Required web technologies and cookies make our website technically accessible to and usable for you. This applies to essential base functionalities such as navigation on the website, correct display in your internet browser or requesting your consent. Without these web technologies and cookies our website does not work.
Purpose
These cookies are stored to keep you logged into the website.
Cookies
Location of Processing
European Union
Technologies Used
Cookies
Expiration date
1 year
Why required?
Required web technologies and cookies make our website technically accessible to and usable for you. This applies to essential base functionalities such as navigation on the website, correct display in your internet browser or requesting your consent. Without these web technologies and cookies our website does not work.
Purpose
This cookie is used to submit forms to us in a safe way.
Cookies
Location of Processing
European Union
Technologies Used
Cookies
Expiration date
1 year
Why required?
Required web technologies and cookies make our website technically accessible to and usable for you. This applies to essential base functionalities such as navigation on the website, correct display in your internet browser or requesting your consent. Without these web technologies and cookies our website does not work.
Purpose
This service provided by Google is used to load specific tags (or trackers) based on your preferences and location.
Why required?
This web technology enables us to insert tags based on your preferences. It is required but adheres to your settings and will not load any tags if you do not consent to them.
Purpose
This cookie is used to store your preferences regarding language.
Cookies
Why required?
We use your browser language to determine which language to show on our website. When you change the default language, this cookie makes sure your language preference is persistent.
Purpose
This service provided by uMarketingSuite is used to track anonymized analytics on the HSO.com application. We find it very important that your privacy is protected. Therefore, we collect and store this data anonymously on our own servers. This cookie helps us collect data from HSO.com so that we can improve the website. Examples of this are: it allows us to track engagement by page, measuring various events like scroll-depth, time on page and clicks.
Cookie
Purpose
With your consent, this website will load Google Analytics to track behavior across the site.
Cookies
Purpose
With your consent, this website will load the Google Advertising tag which enables HSO to report user activity from HSO.com to Google. This enables HSO to track conversions and create remarketing lists based on user activity on HSO.com.
Possible cookies
Please refer to the below page for an updated view of all possible cookies that the Google Ads tag may set.
Cookie information for Google's ad products (safety.google)
Technologies Used
Cookies
Purpose
With your consent, we use IPGeoLocation to retrieve a country code based on your IP address. We use this service to be able to trigger the right web technologies for the right people.
Purpose
With your consent, we use Leadfeeder to identify companies by their IP-addresses. Leadfeeder automatically filters out all users visiting from residential IP addresses and ISPs. All visit data is aggregated on the company level.
Cookies
Purpose
With your consent, this website will load the LinkedIn Insights tag which enables us to see analytical data on website performance, allows us to build audiences, and use retargeting as an advertising technique. Learn more about LinkedIn cookies here.
Cookies
Purpose
With your consent, this website will load the Microsoft Advertising Universal Event Tracking tag which enables HSO to report user activity from HSO.com to Microsoft Advertising. HSO can then create conversion goals to specify which subset of user actions on the website qualify to be counted as conversions. Similarly, HSO can create remarketing lists based on user activity on HSO.com and Microsoft Advertising matches the list definitions with UET logged user activity to put users into those lists.
Cookies
Technologies Used
Cookies
Purpose
With your consent, this website will load the Microsoft Dynamics 365 Marketing tag which enables HSO to score leads based on your level of interaction with the website. The cookie contains no personal information, but does uniquely identify a specific browser on a specific machine. Learn more about Microsoft Dynamics 365 Marketing cookies here.
Cookies
Technologies Used
Cookies
Purpose
With your consent, we use Spotler to measures more extensive recurring website visits based on IP address and draw up a profile of a visitor.
Cookies
Technologies Used
Cookies
Purpose
With your consent, this website will show videos embedded from Vimeo.
Technologies Used
Cookies
Purpose
With your consent, this website will show videos embedded from Youtube.
Cookies
Technologies Used
Cookies
Purpose
With your consent, this website will load the Meta-pixel tag which enables us to see analytical data on website performance, allows us to build audiences, and use retargeting as an advertising technique through platforms owned by Meta, like Facebook and Instagram. Learn more about Facebook cookies here. You can adjust how ads work for you on Facebook here.
Cookies
Purpose
With your consent, we use LeadInfo to identify companies by their IP-addresses. LeadInfo automatically filters out all users visiting from residential IP addresses and ISPs. These cookies are not shared with third parties under any circumstances.
Cookies
Purpose
With your consent, we use TechTarget to identify companies by their IP address(es).
Cookies
Purpose
With your consent, we use this service provided by uMarketingSuite to run A/B tests across the HSO.com application. A/B testing (also called split testing) is comparing two versions of a web page to learn how we can improve your experience.
Purpose
With your consent, we use this service provided by uMarketingSuite to personalize pages and content across the HSO.com application. Personalization helps us to tailor the website to your specific needs, aiming to improve your experience on HSO.com.