Following on from my last post from my Data Platforms series, a deep dive on Azure & Self-Hosted Integration Runtimes (which you can find here) if you missed it. I promised to discuss where you should deploy your Self-Hosted Integration runtime.
Let's start by laying out a few common scenarios by looking at the different locations our data sources can sit:
Azure (IaaS Data Store): for example, a SQL Server VM.
Hybrid (IaaS Data in Azure & On-premise or multi-cloud) for example, a SQL Server VM in Azure & an Oracle DB on-premise.
Hybrid (IaaS Data in Azure & AWS) for example, a SQL Server in Azure and an SQL Server in AWS.
On-premise: On-premise only data, for example an Oracle DB sat in your data centre.
It's worth noting, if you require, you can host multiple SHIRs, this post assumes you're only deploying one.
Azure IaaS
This one is pretty easy, right? If your data source is in Azure, deploy your Self-Hosted Integration Runtime into Azure. Sorted.
Hybrid - Azure IaaS & On-premise
This is the single most common scenario I see day-to-day. Some Data on-premise & some in Azure. Usually on-premise is a large data warehouse which wouldn't make sense to migrate because of the original CapEx investment, and newer databases in Azure.
The first question to ask is, do you have hybrid connectivity configured? By means of an ExpressRoute or Site-to-Site VPN. If the answer is no, the second question is, are you willing to connect to your data source by means of Public IP - hopefully the answer to this is no - it's possible, but from a security perspective, I wouldn't recommend it (at all). At this stage it would be worth looking at your landing zone and considering getting some hybrid connectivity.
Generally in this scenario, providing you have the right infrastructure to support it Azure is the best place to deploy, just ensure you have bandwidth to support the application on your VPN or ExpressRoute.
Hybrid: Azure & AWS
A scenario to consider is IaaS Data Sources in an external cloud provider such as AWS. The potential Data-In / Data-Out VPN Costs could be quite high, especially if your taking full copies of data. A SHIR deployed to your external provider could reduce the costs with compression, the question you need to think about after is would the cost of an EC2 instance provide enough cost saving when it comes to VPN costs to justify itself?
Another thing to look at here is the maturity of your management approach to maintaining Virtual Machines, for example are you using Azure Auto Patch or Update Manager in Azure but don't have a strategy in AWS yet? Reducing time & maintenance of VMs is still a prominent challenge of IT Teams everywhere & day to day these machines don't need much attention, the SHIR agent can be configured to auto-update and the hosts, like all hosts, need to be kept patched. So a good question to ask is, where is the easiest place for my IT team to manage this?
On-premise
The biggest consideration to make here is your S2S / ExpressRoute bandwidth, how much of the bandwidth you're currently using, and how much data you expect to ingest, and how often you'll be doing that. If you're already using most of your bandwidth and you're reluctant to upgrade your VPN SKU - you may want to deploy your SHIR on-premise to be able to leverage the compression of data. You can also look to set a QoS Policy on the machine to throttle the upload, but this obviously comes with speed implications. Some notes on that here: Quality of Service (QoS) Policy | Microsoft Learn & Throttling Network File Transfer Speed on Windows | Windows OS Hub (woshub.com). This isn't something Microsoft officially recommend when it comes to ADF or Synapse documentation, but it is possible and something to take into consideration.
TL;DR
Here's the short version, of course, like mentioned in all the points above, it's not always as simple as this, and there are various scenarios which are supported & it really is dependant on your current infrastructure landscape.
Kommentare