Securing outbound and inbound activity using Azure Firewall for a Private AKS Cluster.
Securing outbound and inbound activity using Azure Firewall for a Private AKS Cluster?
Hello welcome back! Today i want to talk more about securing your AKS cluster and in particular securing incoming and outgoing traffic.
Why securing Azure Kubernetes Cluster for Outbound and Inbound traffic? Let me explain and answer this question.
Securing outbound and inbound traffic for your Azure Kubernetes Cluster is important if you want to stay in control of what your workloads can reach and what can reach them. By default, AKS has open outbound access, which means your cluster can talk to anything on the internet. That’s not ideal if you care about data protection or want to meet security requirements. On the inbound side, you don’t want just anything being able to try and access your apps. Using a service like Azure Firewall helps you lock things down. You can allow only trusted traffic, block the rest, and stay more secure overall without breaking what your apps need to do.
It is good to mention that for a production web application we do not want to use port 80 but want to secure it with port 443 in combination with a certificate, but in this blog post I only want to show you how to secure the traffic flows for Azure Kubernetes with an Azure Firewall. In my next blog I will tell you how to put the web application behind port 443 in combination with cert manager.
What are we going to do in this blog?
In my previous blog I already talked about setting up a WAF aligned private AKS cluster. I will set up this cluster again using the same infrastructure as code, but we will also deploy an Azure Firewall with a User Defined Route that can be set on the virtual network so that all AKS traffic comes from and to the Azure Firewall.
So what do we need?
- Resource Group for the Azure Kubernetes Cluster and other resources.
- Resource Group for the system and work nodes and VMSS.
- Log Analytics Workspace for logging and monitoring.
- Managed Identity with rights on the AKS cluster.
- Azure Kubernetes Cluster (Private)
- Virtual Network with different subnets.
- Virtual Network Gateway for creation of the Point 2 Site connection.
- Azure Firewall
- Azure Policy
- User Defined Route
After the deployment of the resources we will set-up an application and we need to do some extra tweaks to help traffic to the application.
Let's deepdive in the Bicep.
In this set-up we use a bicep, bicepparam file and a deployment script. Let's first look at the bicepparam.
As in my previous article, we use the same resources for creating the private aks cluster, but because we now want to redirect outbound and inbound traffic via the Azure Firewall, we use a User Defined Route and of course an Azure Firewall. This Azure Firewall is built using an Azure Firewall Policy that allows us to use "enable proxy", this option is mandatory when you want to use FQDN in combination with network rules. See also the Microsoft documentation for this. Of course, we ensure that the created User Defined Route is linked to the subnet of the AKS cluster so that traffic is sent to the Appliance.
In this demo we have also already accepted a few components through the network and application rules that are required, such as allowing data in the same region where your cluster is located or the service tag "AzureKubernetesService", which are required for the cluster to function properly and to get the AKS cluster deployed. I also have set some extra rules towards Github because I want to use some sources from Github when building the test application.
Keep in mind that you can use below files but you can certainly use your own settings and namingconventions if you want.
1targetScope = 'subscription'
2
3param updatedBy string
4
5@allowed([
6 'test'
7 'dev'
8 'prod'
9 'acc'
10])
11param environmentType string
12
13param subscriptionId string
14
15// param tenantId string
16
17@description('Unique identifier for the deployment')
18param deploymentGuid string = newGuid()
19
20@description('Product Type: example aks.')
21param productType string
22
23@description('Azure Region to deploy the resources in.')
24@allowed([
25 'westeurope'
26 'northeurope'
27
28])
29param location string
30@description('Location shortcode. Used for end of resource names.')
31param locationShortCode string
32
33@description('Add tags as required as Name:Value')
34param tags object = {
35 Environment: environmentType
36 LastUpdatedOn: utcNow('d')
37 LastDeployedBy: updatedBy
38}
39
40// resource group parameters
41param resourceGroupName string = 'rg-${productType}-${environmentType}-${locationShortCode}'
42param nodeResourceGroup string = '${resourceGroupName}-aks-nodes'
43
44//parameters for AKS Cluster
45param aksClusterName string
46
47//parameters for Log Analytics Workspace
48param logAnalyticsWorkspaceName string
49
50//parameters for Managed Identity
51param managedIdentityName string
52
53
54
55
56//virtual network parameters
57param virtualNetworkName string
58param defaultSubnetName string
59param defaultPrefix string
60param addressPrefix string
61param privateEndpointPrefix string
62param privateEndpointSubnetName string
63param aksSubnetName string
64param aksPrefix string
65param gatewaySubnetName string
66param gatewayPrefix string
67param FWSubnetName string
68param FWAddressPrefix string
69
70//Firewall parameters
71param fwName string
72param firewallPublicIpAddressPrefixName string
73
74//Route table parameters
75param routeTableName string
76param routeTableRoute array
77
78
79var VNetConfiguration = {
80 Subnets: [
81 {
82 name: gatewaySubnetName
83 addressPrefix: gatewayPrefix
84 }
85 {
86 name: FWSubnetName
87 addressPrefix: FWAddressPrefix
88
89 }
90 {
91 name: aksSubnetName
92 addressPrefix: aksPrefix
93 routeTableResourceId: createRouteTable.outputs.resourceId
94 }
95 {
96 name: defaultSubnetName
97 addressPrefix: defaultPrefix
98 }
99 {
100 name: privateEndpointSubnetName
101 addressPrefix: privateEndpointPrefix
102 }
103 ]
104
105 }
106
107//paramamters for the Azure VPN Gateway
108param vpnGatewayName string
109
110// Deploy required Resource Groups - New Resources
111module createResourceGroup 'br/public:avm/res/resources/resource-group:0.4.0' = {
112 scope: subscription(subscriptionId)
113 name: 'rg-${deploymentGuid}'
114 params: {
115 name: resourceGroupName
116 location: location
117 tags: tags
118
119
120 }
121
122}
123
124//deploy route table for the shared meraki
125module createRouteTable 'br/public:avm/res/network/route-table:0.4.0' = {
126 scope: resourceGroup(resourceGroupName)
127 name: 'rt-shared-${deploymentGuid}'
128 params: {
129 name: routeTableName
130 location: location
131 routes: routeTableRoute
132 tags: tags
133 }
134 dependsOn: [
135 createResourceGroup
136 ]
137}
138
139// Deploy required Public IP prefix for Azure Firewall
140module createFirewallPublicIPaddress 'br/public:avm/res/network/public-ip-address:0.6.0' = {
141 scope: resourceGroup(resourceGroupName)
142 name: 'pip-fw-${deploymentGuid}'
143 params: {
144 name: firewallPublicIpAddressPrefixName
145 location: location
146 tags: tags
147 }
148 dependsOn: [createResourceGroup]
149}
150
151module createFirewallPolicy 'br/public:avm/res/network/firewall-policy:0.3.1' = {
152 scope: resourceGroup(resourceGroupName)
153 name: 'fwpolicy-${deploymentGuid}'
154 params: {
155 name: 'fwpolicy-${deploymentGuid}'
156 location: location
157 tags: tags
158 enableProxy: true
159 ruleCollectionGroups: [
160 {
161 name: 'aks-rule-001'
162 priority: 5000
163
164 ruleCollections: [
165 {
166 ruleCollectionType: 'FirewallPolicyFilterRuleCollection'
167 action: {
168 type: 'Allow'
169 }
170 rules: [
171 {
172 ruleType: 'NetworkRule'
173 name: 'AKS-Rule-Network-Collection'
174 ipProtocols: [
175 'TCP'
176 'UDP'
177 ]
178 sourceAddresses: [
179 '*'
180 ]
181 sourceIpGroups: []
182 destinationAddresses: []
183 destinationIpGroups: []
184 destinationFqdns: [
185 'AzureCloud.westeurope'
186 'ntp.ubuntu.com'
187 'ghcr.io'
188 'pkg-containers.githubusercontent.com'
189 'docker.io'
190 'registry-1.docker.io'
191 'production.cloudflare.docker.com'
192 ]
193 destinationPorts: [
194 '1194'
195 '9000'
196 '123'
197 '443'
198 ]
199 }
200 ]
201 name: 'network-rules'
202 priority: 101
203 }
204 {
205 ruleCollectionType: 'FirewallPolicyFilterRuleCollection'
206 action: {
207 type: 'Allow'
208 }
209 rules: [
210 {
211 ruleType: 'ApplicationRule'
212 name: 'aks-application-rule'
213 protocols: [
214 {
215 protocolType: 'Http'
216 port: 80
217 }
218 {
219 protocolType: 'Https'
220 port: 443
221 }
222 ]
223 fqdnTags: [
224 'AzureKubernetesService'
225 ]
226 webCategories: []
227 targetFqdns: []
228 targetUrls: []
229 terminateTLS: false
230 sourceAddresses: [
231 '*'
232 ]
233 destinationAddresses: []
234 sourceIpGroups: []
235 httpHeadersToInsert: []
236 }
237 ]
238 name: 'application-rules'
239 priority: 102
240 }
241 ]
242 }
243 ]
244
245 tier: 'Standard'
246
247 }
248 dependsOn: [createResourceGroup]
249}
250
251
252module createAzureFirewall 'br/public:avm/res/network/azure-firewall:0.6.0' = {
253 scope: resourceGroup(resourceGroupName)
254 name: 'fw-${deploymentGuid}'
255 params: {
256 name: fwName
257 virtualNetworkResourceId: createVnet.outputs.resourceId
258 azureSkuTier: 'Standard'
259 firewallPolicyId: createFirewallPolicy.outputs.resourceId
260
261 publicIPResourceID: createFirewallPublicIPaddress.outputs.resourceId
262 location: location
263 diagnosticSettings: [
264 {
265 name: 'customSetting'
266 metricCategories: [
267 {
268 category: 'AllMetrics'
269 }
270 ]
271 workspaceResourceId: createLogAnalyticsWorkspace.outputs.resourceId
272 }
273 ]
274 tags: tags
275 zones: [
276 '1'
277 '2'
278 '3'
279 ]
280 }
281
282 dependsOn: [
283 createResourceGroup
284 ]
285}
286
287module createLogAnalyticsWorkspace 'br/public:avm/res/operational-insights/workspace:0.11.1' = {
288 name: 'law-${deploymentGuid}'
289 scope: resourceGroup(resourceGroupName)
290 params: {
291 name: logAnalyticsWorkspaceName
292 location: location
293 tags: tags
294 }
295 dependsOn: [createResourceGroup]
296}
297
298module createManagedIdentity 'br/public:avm/res/managed-identity/user-assigned-identity:0.4.0' = {
299 name: 'mi-${deploymentGuid}'
300 scope: resourceGroup(resourceGroupName)
301 params: {
302 name: managedIdentityName
303 location: location
304
305
306 tags: tags
307 }
308 dependsOn: [createResourceGroup]
309}
310
311module createPrivateDNSZone 'br/public:avm/res/network/private-dns-zone:0.2.0' = {
312 scope: resourceGroup(resourceGroupName)
313 name: 'dns-${deploymentGuid}'
314 params: {
315 name: 'privatelink.westeurope.azmk8s.io'
316 virtualNetworkLinks: [
317 {
318 name: 'vnet-${deploymentGuid}'
319 registrationEnabled: true
320 virtualNetworkResourceId: createVnet.outputs.resourceId
321 }
322 ]
323 location: 'global'
324 roleAssignments:[ {
325 roleDefinitionIdOrName: 'Contributor'
326 principalId: createManagedIdentity.outputs.principalId
327 principalType: 'ServicePrincipal'
328 }
329]
330 tags: tags
331 }
332 dependsOn: [createResourceGroup]
333}
334
335module createVnet 'br/public:avm/res/network/virtual-network:0.5.4' = {
336 name: 'vnet-${deploymentGuid}'
337 scope: resourceGroup(resourceGroupName)
338 params: {
339 name: virtualNetworkName
340 addressPrefixes: [
341 addressPrefix
342 ]
343 subnets: VNetConfiguration.Subnets
344 dnsServers: [
345 '10.0.5.4'
346 ]
347 roleAssignments:[ {
348 roleDefinitionIdOrName: 'Contributor'
349 principalId: createManagedIdentity.outputs.principalId
350 principalType: 'ServicePrincipal'
351 }
352]
353 location: location
354 tags: tags
355
356 }
357 dependsOn: [createResourceGroup]
358}
359
360module createAksCluster 'br/public:avm/res/container-service/managed-cluster:0.8.3' = {
361 name: 'aks-${deploymentGuid}'
362 scope: resourceGroup(resourceGroupName)
363 params: {
364 name: aksClusterName
365 enablePrivateCluster: true
366 primaryAgentPoolProfiles: [
367 {
368 availabilityZones: [
369 3
370 ]
371 count: 1
372 enableAutoScaling: true
373 maxCount: 1
374 maxPods: 50
375 minCount: 1
376 mode: 'System'
377 name: 'systempool'
378 nodeTaints: [
379 'CriticalAddonsOnly=true:NoSchedule'
380 ]
381 osDiskSizeGB: 0
382 osType: 'Linux'
383 type: 'VirtualMachineScaleSets'
384 vmSize: 'Standard_DS2_v2'
385 vnetSubnetResourceId: createVnet.outputs.subnetResourceIds[2]
386 }
387 ]
388 agentPools: [
389 {
390 availabilityZones: [
391 3
392 ]
393 count: 1
394 enableAutoScaling: true
395 maxCount: 1
396 maxPods: 50
397 minCount: 1
398 minPods: 2
399 mode: 'User'
400 name: 'userpool1'
401 nodeLabels: {}
402 osDiskType: 'Ephemeral'
403 osDiskSizeGB: 60
404 osType: 'Linux'
405 scaleSetEvictionPolicy: 'Delete'
406 scaleSetPriority: 'Regular'
407 type: 'VirtualMachineScaleSets'
408 vmSize: 'Standard_DS2_v2'
409 vnetSubnetResourceId: createVnet.outputs.subnetResourceIds[2]
410 }
411 {
412 availabilityZones: [
413 3
414 ]
415 count: 1
416 enableAutoScaling: true
417 maxCount: 1
418 maxPods: 50
419 minCount: 1
420 minPods: 2
421 mode: 'User'
422 name: 'userpool2'
423 nodeLabels: {}
424 osDiskType: 'Ephemeral'
425 osDiskSizeGB: 60
426 osType: 'Linux'
427 scaleSetEvictionPolicy: 'Delete'
428 scaleSetPriority: 'Regular'
429 type: 'VirtualMachineScaleSets'
430 vmSize: 'Standard_DS2_v2'
431 }
432 ]
433 autoUpgradeProfileUpgradeChannel: 'stable'
434 autoNodeOsUpgradeProfileUpgradeChannel: 'Unmanaged'
435 outboundType: 'userDefinedRouting'
436 maintenanceConfigurations: [
437 {
438 name: 'aksManagedAutoUpgradeSchedule'
439 maintenanceWindow: {
440 schedule: {
441 weekly: {
442 intervalWeeks: 1
443 dayOfWeek: 'Sunday'
444 }
445 }
446 durationHours: 4
447 utcOffset: '+00:00'
448 startDate: '2024-07-15'
449 startTime: '00:00'
450 }
451 }
452 {
453 name: 'aksManagedNodeOSUpgradeSchedule'
454 maintenanceWindow: {
455 schedule: {
456 weekly: {
457 intervalWeeks: 1
458 dayOfWeek: 'Sunday'
459 }
460 }
461 durationHours: 4
462 utcOffset: '+00:00'
463 startDate: '2024-07-15'
464 startTime: '00:00'
465 }
466 }
467 ]
468 networkPlugin: 'azure'
469 networkPolicy: 'azure'
470 skuTier: 'Standard'
471 dnsServiceIP: '10.10.200.10'
472 serviceCidr: '10.10.200.0/24'
473 omsAgentEnabled: true
474 monitoringWorkspaceResourceId: createLogAnalyticsWorkspace.outputs.resourceId
475 disableLocalAccounts: true
476 enableAzureDefender: true
477 diagnosticSettings: [
478 {
479 name: 'customSetting'
480 logCategoriesAndGroups: [
481 {
482 category: 'kube-apiserver'
483 }
484 {
485 category: 'kube-controller-manager'
486 }
487 {
488 category: 'kube-scheduler'
489 }
490 {
491 category: 'cluster-autoscaler'
492 }
493 ]
494 metricCategories: [
495 {
496 category: 'AllMetrics'
497 }
498 ]
499 workspaceResourceId: createLogAnalyticsWorkspace.outputs.resourceId
500 }
501 ]
502 privateDNSZone: createPrivateDNSZone.outputs.resourceId
503 enablePrivateClusterPublicFQDN: true
504 nodeResourceGroup: nodeResourceGroup
505 managedIdentities: {
506 userAssignedResourceIds: [
507 createManagedIdentity.outputs.resourceId
508 ]
509 }
510 tags: tags
511 aadProfile: {
512 aadProfileEnableAzureRBAC: true
513 aadProfileManaged: true
514 }
515 }
516 dependsOn: [
517 createResourceGroup
518
519 ]
520 }
521
522module createAzureVpnGateway 'br/public:avm/res/network/virtual-network-gateway:0.6.0' = {
523 scope: resourceGroup(resourceGroupName)
524 name: 'vpn-${deploymentGuid}'
525 params: {
526 name: vpnGatewayName
527 gatewayType: 'Vpn'
528 vpnType: 'RouteBased'
529 skuName: 'VpnGw1AZ'
530 clusterSettings:{
531 clusterMode:'activePassiveNoBgp'
532 }
533 virtualNetworkResourceId: createVnet.outputs.resourceId
534 location: location
535 tags: tags
536 }
537 dependsOn: [createResourceGroup]
538}
In the bicepparam we use some fixed values that we use in the bicep file.
1using 'aks-udr.bicep'
2
3//parameters for the deployment.
4param updatedBy = ''
5param subscriptionId = ''
6param environmentType = 'prod'
7param location = 'westeurope'
8param locationShortCode = 'weu'
9param productType = 'aks'
10
11param virtualNetworkName = 'vnet-${productType}-${environmentType}-${locationShortCode}'
12param defaultSubnetName = 'snet-default'
13param defaultPrefix = '10.0.2.0/24'
14param addressPrefix = '10.0.0.0/16'
15
16param aksSubnetName = 'snet-aks'
17param aksPrefix = '10.0.3.0/24'
18
19param privateEndpointPrefix = '10.0.4.0/24'
20param privateEndpointSubnetName = 'snet-pe'
21
22param gatewayPrefix = '10.0.1.0/27'
23param gatewaySubnetName = 'GatewaySubnet'
24
25param FWSubnetName = 'AzureFirewallSubnet'
26param FWAddressPrefix = '10.0.5.0/24'
27
28param aksClusterName = '${productType}-pof-${environmentType}-${locationShortCode}'
29
30param logAnalyticsWorkspaceName = 'law-${productType}-${environmentType}-${locationShortCode}'
31
32param managedIdentityName = 'mi-${productType}-${environmentType}-${locationShortCode}'
33
34param vpnGatewayName = 'vgw-${productType}-${environmentType}-${locationShortCode}'
35
36param fwName = 'fw-${productType}-${environmentType}-${locationShortCode}'
37
38param firewallPublicIpAddressPrefixName = 'pip-fw-${productType}-${environmentType}-${locationShortCode}'
39
40param routeTableName = 'rt-aks-to-fw'
41param routeTableRoute = [
42 {
43 name: 'aks-to-firewall'
44 properties: {
45 addressPrefix: '0.0.0.0/0'
46 nextHopIpAddress: '10.0.5.4'
47 nextHopType: 'VirtualAppliance'
48
49 }
50 }
51
52]
This deployment script is similar to the script from my previous blog but uses some different naming conventions for the bicep files. The script deploys the bicep with a whatif statement, then after checking and accepting it creates groups to ensure that the user running the script is part of the administrator group to be able to access the AKS cluster.
1param(
2 [Parameter(Mandatory = $true)][string] $subscriptionID = "",
3 [Parameter(Mandatory = $true)][ValidateSet("northeurope", "westeurope")][string] $location = "",
4 [Parameter(Mandatory = $true, ParameterSetName = 'Default')] $productType = "",
5 [Parameter(Mandatory = $true, Position = 3)] [ValidateSet("prod", "acc", "dev", "test")] [string] $environmentType = "",
6 [switch] $deploy
7)
8
9# Ensure parameters are captured
10Write-Host "Subscription ID: $subscriptionID"
11Write-Host "Location: $location"
12Write-Host "Product Type: $productType"
13Write-Host "Environment Type: $environmentType"
14
15$deploymentID = (New-Guid).Guid
16
17<# Set Variables #>
18az account set --subscription $subscriptionID --output none
19if (!$?) {
20 Write-Host "Something went wrong while setting the correct subscription. Please check and try again." -ForegroundColor Red
21 exit 1
22}
23
24$updatedBy = (az account show | ConvertFrom-Json).user.name
25$userId = (az ad signed-in-user show --query id --output tsv)
26$location = $location.ToLower() -replace " ", ""
27
28$LocationShortCodeMap = @{
29 "westeurope" = "weu";
30 "northeurope" = "neu";
31}
32
33$locationShortCode = $LocationShortCodeMap.$location
34$resourceGroup = "rg-$productType-$environmentType-$locationShortCode"
35
36Write-Host "Using Resource Group Name: '$resourceGroup'" -ForegroundColor Cyan
37
38if ($deploy) {
39 Write-Host "Running a Bicep deployment with ID: '$deploymentID' for Environment: '$environmentType'..." -ForegroundColor Green
40 az deployment sub create `
41 --name $deploymentID `
42 --location $location `
43 --template-file ./aks-udr.bicep `
44 --parameters ./aks-udr.bicepparam `
45 --parameters updatedBy=$updatedBy location=$location locationShortCode=$LocationShortCode productType=$productType environmentType=$environmentType `
46 --confirm-with-what-if `
47
48 if (!$?) {
49 Write-Host "Bicep deployment failed. Stopping execution." -ForegroundColor Red
50 exit 1
51 }
52 Write-Host "Bicep deployment completed successfully." -ForegroundColor Green
53}
54
55Write-Host "Proceeding with Entra ID group creation and role assignment for Resource Group '$resourceGroup'." -ForegroundColor Cyan
56
57# List of required Entra ID groups and their corresponding Azure roles
58$groupRoleMap = @{
59 "aks-gg-$environmentType-administrators" = "Azure Kubernetes Service RBAC Cluster Admin";
60 "aks-gg-$environmentType-reader" = "Azure Kubernetes Service RBAC Reader";
61}
62
63# Ensure Entra ID groups exist and assign roles
64foreach ($groupName in $groupRoleMap.Keys) {
65 $existingGroup = az ad group list --query "[?displayName=='$groupName']" --output json | ConvertFrom-Json
66
67 if (-not $existingGroup) {
68 Write-Host "Creating Entra ID group: $groupName" -ForegroundColor Yellow
69 $groupId = az ad group create --display-name $groupName --mail-nickname $groupName.Replace("-", "") --query id --output tsv
70
71 if (!$groupId) {
72 Write-Host "Failed to create Entra ID group: $groupName. Please check your permissions." -ForegroundColor Red
73 exit 1
74 }
75 Write-Host "Entra ID group '$groupName' created successfully." -ForegroundColor Green
76 } else {
77 Write-Host "Entra ID group '$groupName' already exists. Skipping creation." -ForegroundColor Cyan
78 $groupId = $existingGroup[0].id
79 }
80
81 # Wait for group replication before assigning roles
82 Write-Host "Waiting for Entra ID replication..." -ForegroundColor Cyan
83 Start-Sleep -Seconds 30
84
85 # Assign role at the Resource Group level (retry logic)
86 $roleName = $groupRoleMap[$groupName]
87 $maxRetries = 3
88 $retryCount = 0
89 $roleAssignmentSuccess = $false
90
91 while ($retryCount -lt $maxRetries -and -not $roleAssignmentSuccess) {
92 Write-Host "Assigning '$roleName' role to group '$groupName' on Resource Group '$resourceGroup'" -ForegroundColor Yellow
93 az role assignment create --assignee $groupId --role "$roleName" --scope /subscriptions/$subscriptionID/resourceGroups/$resourceGroup --output none
94
95 if ($?) {
96 Write-Host "Successfully assigned '$roleName' role to '$groupName'." -ForegroundColor Green
97 $roleAssignmentSuccess = $true
98 } else {
99 Write-Host "Retrying role assignment in 20 seconds..." -ForegroundColor Yellow
100 Start-Sleep -Seconds 20
101 $retryCount++
102 }
103 }
104
105 if (-not $roleAssignmentSuccess) {
106 Write-Host "Failed to assign '$roleName' role to '$groupName' after multiple attempts." -ForegroundColor Red
107 }
108
109 # Add the signed-in user to the administrator group
110 if ($groupName -match "administrators") {
111 Write-Host "Adding current user ($updatedBy) to group '$groupName'." -ForegroundColor Yellow
112 az ad group member add --group $groupName --member-id $userId
113
114 if (!$?) {
115 Write-Host "Failed to add user '$updatedBy' to '$groupName'. Please check permissions." -ForegroundColor Red
116 } else {
117 Write-Host "Successfully added user '$updatedBy' to '$groupName'." -ForegroundColor Green
118 }
119 }
120}
Save the files and you can execute these files by running the deployment script within VSCode.
.\deploy-aks-udr.ps1 -subscriptionID "" -location "westeurope" -productType "aks" -environmentType "prod" -deploy
Let's build the application.
The deployment in total will take up to 40 minutes so you have some spare time ;-)
Since we already explained in my previous blog how you can connect to the cluster using the p2s, I would like to refer you to that article.
When the AKS cluster is ready we will begin setting up the application
Let's first see if we can set up a test web application to do the rest of the configuration and testing. This information can of course be found in the Microsoft documentation.
- Log in to the AKS cluster you have build.
- Create a namespace with name 'test-application' using
kubectl create namespace test-application - We are building the test application in the namespace 'test-application' using
kubectl apply -f https://raw.githubusercontent.com/Azure-Samples/aks-store-demo/main/aks-store-quickstart.yaml -n test-application - Check the created pods using
kubectl get pods -n test-application
So we have created the application, lets go to the Azure Firewall!
Let's configure the Azure Firewall
When using Azure Firewall with a user-defined route (UDR) to direct egress traffic, you must configure a DNAT rule to support ingress. Without it, return traffic from public load balancers may route asymmetrically coming in through the public IP but returning via the firewall’s private IP. Azure Firewall, being stateful, drops such unmatched responses. This breaks ingress for AKS clusters using a LoadBalancer service. To ensure connectivity, create a DNAT rule mapping the firewall’s public IP to the internal IP of your service. So lets create a DNAT rule.
The destination address in your DNAT rule needs to refer the port on the Azure Firewall that external users will access. The translated address must be the internal IP of the Kubernetes service’s load balancer. Likewise, the translated port should match the port exposed by your Kubernetes service.
To correctly configure this rule, retrieve the internal IP assigned to the Kubernetes service's loadbalancer. You can find this IP by using the kubectl get services -n test-application
Write down the external IP-Address of the store-front, if you want you can browse copy paste this in your browser but you won't get an answer we will fix that ;-).
Write down the external IP-Address of the firewall.
Create a DNAT rule in the firewall to translate the external IP-Address from the Firewall to the store front we are using Azure CLI.
This policy will add an nat rule collection and the specific rule to translate the Azure Firewall to the loadbalancer. Of course you need to check you own naming and ip details and modify it in the command, also make sure you are logged in with a correct account using az login.
az network firewall policy rule-collection-group collection add-nat-collection -n nat_collection --collection-priority 10003 --policy-name fwpolicy-9c1063a1-5d13-4454-8805-5e75733df783 -g rg-aks-prod-weu --rule-collection-group-name aks-rule-001 --action DNAT --rule-name network_rule --description "dnat-aks" --destination-addresses 9.163.171.212 --source-addresses * --translated-address 74.178.216.186 --translated-port 80 --destination-ports 80 --ip-protocols TCP UDP
- If the command is finished with the deployment you can copy the external ip-address from the Azure Firewall and paste this in the browser and check if you can access the dummy application.
As you can see the page cannot be loaded, the reason is Asymmetric routing, Asymmetric routing means traffic takes different paths to and from a destination. Firewalls lose track of the session and drop returning packets because they expect the traffic on the same path.
So let's fix this:
- Create a extra route in the existing Route Table:
az network route-table route create -g rg-aks-prod-weu --route-table-name rt-aks-to-fw -n fw-host-route --next-hop-type Internet --address-prefix 9.163.171.212/32
This will create an route to the Internet from the Azure Firewall, use your own details when running this command.
When the rule is applied you can test the same external ip-address from the Azure Firewall in the browser and you will see the application.
Conclusion
Setting up an Azure Kubernetes Cluster seems easy and yes Azure and especially Bicep makes it look easy, but of course you have to do this safely. An Azure Firewall is extremely suitable to contribute to this process to safely restrict inbound and outbound traffic to and from the Azure Kubernetes Cluster.
When deploying an Azure Firewall you will need to do some additional tweaks which I explained in this blog, you could also integrate these commands when using a Github Action or an Azure DevOps pipeline for example.
I hope you found this blog interesting again, until next time.