Custom Image Templates part 4
Create Custom Image Templates using DevOps Pipelines
Hello there, welcome back! As i mentioned in the previous posts we created Image Templates using different methods. But how cool would it be to do it all automated using Azure DevOps in combination with DevOps Pipelines I want to give you a little overview what is possible using this technique.
Azure DevOps Pipelines is a cloud-based Continuous Integration (CI) and Continuous Deployment (CD) service that automates the process of building, testing, and deploying code. It enables development teams to streamline their workflows, ensure code quality, and deploy applications efficiently across multiple platforms.
Some explanation about the service connections
We will first create an connection to the DevOps environment and yes, I assume you are already having an DevOps organization. If you don't have one you can easily create this, you can find this information on the Microsoft website and the follow up step, how to create a new project. When you have set-up these resources you can configure your service connection to your Azure Subscription. You can do this multiple ways with some different pro's and cons for each.
- Workload Identity
- Service Principal
- Managed Identity
I will not name them all, but here is some explanation:
- Workload Identity:
Workload Identity is a secure and seamless way to enable applications, services, or workloads to authenticate and access Azure resources using EntraID without requiring credentials like passwords or secrets.
Instead of managing credentials manually, Workload Identity leverages Managed Identities or other federated identities assigned to the underlying infrastructure. This eliminates the need to store sensitive credentials, enhancing security and reducing operational overhead.
- Service Principal
A Service Principal is an identity created in EntraID that is used by applications, automation tools, or services to access Azure resources. It acts as a "service account" for your application or service, enabling secure and programmatic access to Azure.
- Managed Identity
Managed Identity is a feature of Azure that provides an automatically managed identity for applications and services running on Azure. This identity can be used to securely authenticate and authorize access to Azure resources without the need for managing credentials manually. Using a managed identity allows Azure pipelines to authenticate and access Azure resources on behalf of the pipeline without using a service principal or secrets.
Create the Service Principal trough the portal
In this case we are gonna use an Service Principal because it's for demo purposes.
When you want to create a service connection, you can create the service principal automatic trough the portal, but i will also show you how to do this using the command line.
First i will give you some directions to do it trough the portal:
- Go to your created project and click on "project settings".
- On the left click on "service connections" and click on "new service connections"
- Select Azure Resource Manager en click on "next":
- Choose the automatic Service Principal and click on next, fill in the information and the connection is created.
- When the Service Principal is created, you need to give the Service Principal access to the Resource Group or Subscription the minimum is Contributor, but i prefer in this demo to set Owner rights, create custom RBAC roles when you want more control.
Create the Service Principal trough the Command Line
These are the steps how you can create this, using the command line:
- First login using
az login - Set the subscription use
az account set --subscription "" - Run the command
az ad sp create-for-rbac --name imagePipeline --role owner --scopes /subscriptions/yoursubscription - Save the output but delete this afterwards because of sensitive data!!
- Go to the "new service connection" in the DevOps environment and choose "service principal manual" and use the information you have got from the output in the commandline.
Because we have given Service Principal Owner access to the subscription using the --scopes in the commandline you don't have to do anything in the portal anymore.
What's next?
We have now set the base for running some pipelines let's make a begin and start to create a yaml file, for some examples you can check my repository.
My goal is to deploy an Virtual Machine based on the Image Templates infrastructure with Bicep executed by a pipeline, let's start.
You can find the whole yaml file in my repository.
- First we need to have some validation inputs, i will begin with the variables and parameters explanation below this code section.
1name: Deploy infrastructure install image deploy machine
2trigger:
3 branches:
4 include:
5 - main
6
7variables:
8 - group: "Credentials"
9
10parameters:
11 - name: deploymentType
12 displayName: "Select the deployment type"
13 type: string
14 default: "sub"
15 values:
16 - tenant
17 - mg
18 - sub
19 - group
20
21 - name: subscriptionId
22 displayName: "Azure Subscription"
23 type: string
24 default:
25
26 - name: productType
27 displayName: "Select the product type"
28 type: string
29 default: "avd"
30 values:
31 - avd
32
33 - name: environmentType
34 displayName: "Select the environment"
35 type: string
36 values:
37 - prod
38 - acc
39 - dev
40 - test
41
42 - name: location
43 displayName: "Select the location"
44 type: string
45 default: "westeurope"
46 values:
47 - westeurope
48 - northeurope
49
50 - name: vnetAddressPrefix
51 displayName: "Fill in the vnet address prefix"
52 type: string
53
54 - name: avdSubnetPrefix
55 displayName: "Fill in the subnet address prefix"
56 type: string
57
58 - name: imageVersion
59 displayName: "Select base image"
60 type: string
61 values:
62 - win11-24h2-avd-m365
63 - 2022-datacenter-azure-edition
64
65
66 - name: offerName
67 displayName: "Select base image"
68 type: string
69 values:
70 - office-365
71 - WindowsServer
72
73 - name: publisherName
74 displayName: "Select base image"
75 type: string
76 values:
77 - MicrosoftWindowsDesktop
78 - MicrosoftWindowsServer
79
80 - name: vmCount
81 displayName: "Number of VMs to Create"
82 type: string
83 default: 1
84
85 - name: avdSize
86 displayName: "Size of the AVD VM"
87 type: string
88 default: "Standard_D2s_v3"
89
- The variable group
Credentialsis an group with information such as an username, password tenantID as shown in the example below, we can use this information for parsing in sensitive information in the pipeline.
As you can see you can store you information from the Service Principal which you can use to login and use the pipeline.
The input paremeters can be used to make the code more reusable so it can be deployed in multiple environments.
The first stage will install some necessary system packages and do a first validation check.
1stages:
2 - stage: ValidateInputs
3 displayName: "Validate Inputs"
4 jobs:
5 - job: validate_inputs
6 displayName: "Validate Deployment Inputs"
7 pool:
8 vmImage: "ubuntu-latest"
9 steps:
10 - task: Bash@3
11 displayName: Install System Packages
12 inputs:
13 targetType: "inline"
14 script: |
15 sudo apt update
16 sudo apt install -y uuid-runtime
17
18 - task: Bash@3
19 displayName: Configure Globalization Setting
20 inputs:
21 targetType: "inline"
22 script: echo "##vso[task.setvariable variable=DOTNET_SYSTEM_GLOBALIZATION_INVARIANT]true"
- The second stage is to install, update modules and login in to the environment, using the Credentials information, at the end it will deploy the infrastructure.
In this bit of code you will see the command we are using to connect to the environment az login --service-principal --username $(AVDPipelines) --password $(Secret) --tenant $(tenantId) you can see you that we are calling the credentials group as shown in a early stage, if you have named them in the libary differently, you will need to use the same names to connect.
Because we don not have a bicepparam file we are generating the file in the yaml file itself. At the end of this file you will see the az deployment command which deploys the bicep code.
1- stage: Deploy_Bicep
2 displayName: "Deploy Bicep"
3 jobs:
4 - job: Deploy_Bicep
5 displayName: "Deploy Bicep"
6 pool:
7 vmImage: "ubuntu-latest"
8 steps:
9 - task: Bash@3
10 displayName: Azure CLI Login
11 inputs:
12 targetType: "inline"
13 script: |
14 # Install Azure CLI
15 sudo apt-get update
16 sudo apt-get install -y libicu-dev curl
17 curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
18
19 # Login to Azure
20 az login --service-principal --username $(AVDPipelines) --password $(Secret) --tenant $(tenantId)
21
22 if [ $? -ne 0 ]; then
23 echo "Azure login failed. Please check the service principal credentials."
24 exit 1
25 fi
26 echo "Azure login successful."
27
28 # Confirm subscription
29 az account show
30
31 - task: PowerShell@2
32 displayName: "Generate and Deploy Bicep Parameters"
33 inputs:
34 targetType: "inline"
35 script: |
36 # Login to Azure
37 az login --service-principal --username "$(AVDPipelines)" --password "$(Secret)" --tenant "$(tenantId)"
38
39 # Validate location parameter
40 $location = '${{ parameters.location }}'
41 if (-not $location) {
42 Write-Error "Error: 'location' parameter not provided."
43 exit 1
44 }
45
46 # Set location short code
47 switch ($location) {
48 "westeurope" { $locationShortCode = "weu" }
49 "northeurope" { $locationShortCode = "neu" }
50 default {
51 Write-Error "Unknown location: $location"
52 exit 1
53 }
54 }
55
56 # Define parameter file path
57 $paramPath = "$(System.DefaultWorkingDirectory)/params.bicepparam"
58 Write-Output "Resolved parameter file path: $paramPath"
59 Write-Output "System.DefaultWorkingDirectory: $(System.DefaultWorkingDirectory)"
60 Write-Output "Location: $location"
61 Write-Output "Location Short Code: $locationShortCode"
62
63 # Ensure directory exists
64 $paramDir = [System.IO.Path]::GetDirectoryName($paramPath)
65 if (-not (Test-Path $paramDir)) {
66 New-Item -ItemType Directory -Path $paramDir | Out-Null
67 }
68
69 # Extract parameters
70 $subscriptionId = '${{ parameters.subscriptionId }}'
71 $productType = '${{ parameters.productType }}'
72 $environmentType = '${{ parameters.environmentType }}'
73 $updatedBy = 'demopurposes'
74 $vnetAddressPrefix = '${{ parameters.vnetAddressPrefix }}'
75 $avdSubnetPrefix = '${{ parameters.avdSubnetPrefix }}'
76 $skuVersion = '${{ parameters.imageVersion }}'
77 $offerName = '${{ parameters.offerName }}'
78 $publisherName = '${{ parameters.publisherName }}'
79
80 # Derived values
81 $storageAccountName = "st$productType$environmentType$locationShortCode"
82 $azureSharedImageGalleryName = "gal$productType$environmentType$locationShortCode"
83 $imagesSharedGalleryName = "img-$productType-$environmentType-$locationShortCode"
84 $imageTemplateName = "it-$productType-$environmentType-$locationShortCode"
85 $userAssignedManagedIdentityName = "mi-$productType-$environmentType-$locationShortCode"
86 $vnetName = "vnet-$productType-$environmentType-$locationShortCode"
87 $avdHostpoolName = "vdpool-$productType-$environmentType-$locationShortCode"
88 $applicationGroupName = "vdag-$productType-$environmentType-$locationShortCode"
89 $workspaceName = "vdws-$productType-$environmentType-$locationShortCode"
90 $availabilitySetName = "avail-$productType-$environmentType-$locationShortCode"
91 $subnetName = "snet-$productType"
92 $networksecurityGroupName = "nsg-$productType"
93
94 # Validate required parameters
95 $requiredParams = @($subscriptionId, $productType, $environmentType, $location, $locationShortCode, $vnetAddressPrefix, $avdSubnetPrefix)
96 foreach ($param in $requiredParams) {
97 if (-not $param) {
98 Write-Error "Missing required parameter: $($param)"
99 exit 1
100 }
101 }
102
103 # Generate parameter file content
104 $params = @"
105 using './infra/avd.bicep'
106
107 param subscriptionId = '$subscriptionId'
108 param productType = '$productType'
109 param environmentType = '$environmentType'
110 param location = '$location'
111 param updatedBy = '$updatedBy'
112 param locationShortCode = '$locationShortCode'
113 param storageAccountName = '$storageAccountName'
114 param azureSharedImageGalleryName = '$azureSharedImageGalleryName'
115 param imagesSharedGalleryName = '$imagesSharedGalleryName'
116 param imageTemplateName = '$imageTemplateName'
117 param userAssignedManagedIdentityName = '$userAssignedManagedIdentityName'
118 param vnetName = '$vnetName'
119 param avdHostpoolName = '$avdHostpoolName'
120 param applicationGroupName = '$applicationGroupName'
121 param workspaceName = '$workspaceName'
122 param availabilitySetName = '$availabilitySetName'
123 param subnetName = '$subnetName'
124 param vnetAddressPrefix = '$vnetAddressPrefix'
125 param avdSubnetPrefix = '$avdSubnetPrefix'
126 param skuVersion = '$skuVersion'
127 param offerName = '$offerName'
128 param publisherName = '$publisherName'
129 param networksecurityGroupName = '$networksecurityGroupName'
130 "@
131
132 # Write to file
133 Write-Output "Writing parameter file to: $paramPath"
134 $params | Out-File -FilePath $paramPath -Encoding UTF8
135
136 # Verify file content
137 Write-Output "Generated parameter file content:"
138 Get-Content $paramPath
139
140 # Generate a UUID using PowerShell
141 $uuid = [guid]::NewGuid()
142
143 # Deploy Bicep template using the correct file path and UUID
144 az deployment sub create `
145 --name "action-deploy-$uuid" `
146 --location $location `
147 --template-file "$(System.DefaultWorkingDirectory)/infra/avd.bicep" `
148 --parameters $paramPath
- The 3th step is to build the image.
First we are checking for the latest image template and we declared the variable. We continue with building the image, using the az image builder command and use the declared variable. It's good to mention we are also using the wait parameter in combination with the az image builder command, because you want to wait for this process to finish, as you are installing a Virtual Machine based on this image.
1- stage: Create_Image
2 jobs:
3 - job: CreateImage
4 displayName: "Create Image"
5 pool:
6 vmImage: "ubuntu-latest"
7 steps:
8 - task: Bash@3
9 displayName: List and Run the Latest Image
10 inputs:
11 targetType: "inline"
12 script: |
13 # Set location shortcode
14 case "${{ parameters.location }}" in
15 "westeurope") locationShortCode="weu" ;;
16 "northeurope") locationShortCode="neu" ;;
17 *) echo "Unknown location: ${{ parameters.location }}"; exit 1 ;;
18 esac
19
20 resourceGroupName="rg-${{ parameters.productType }}-${{ parameters.environmentType }}-${locationShortCode}"
21
22 # Login to Azure
23 az login --service-principal --username $(AVDPipelines) --password $(Secret) --tenant $(tenantId)
24
25 # List image builders in the resource group
26 imageList=$(az image builder list -g "$resourceGroupName" --query "[].{Name:name, CreationTime:timeCreated}" -o json)
27
28 if [[ -z "$imageList" || "$imageList" == "[]" ]]; then
29 echo "Error: No image builders found in resource group '$resourceGroupName'"
30 exit 1
31 fi
32
33 # Get the latest image
34 latestImageName=$(echo "$imageList" | jq -r 'sort_by(.CreationTime) | last | .Name')
35
36 if [[ -z "$latestImageName" ]]; then
37 echo "Error: Could not find the latest image builder in resource group '$resourceGroupName'"
38 exit 1
39 fi
40
41 # Run the image builder
42 az image builder run -n "$latestImageName" -g "$resourceGroupName" --no-wait
43
44 # Wait for completion
45 az image builder wait -n "$latestImageName" -g "$resourceGroupName" --custom "lastRunStatus.runState!='Running'"
46
47 echo "Image creation completed successfully."
- And the final step deploy a Virtual Machine based on the latest image.
In this step we are building the Virtual Machine using the latest image that we have created in the previous steps. We are looking for the latest stored image in the Azure Compute Gallery. Because we have an input variable in the first step based on the vm count, it will deploy the amount of Virtual Machines based on this number. We also declare a username, you can store this in the Credentials group if you want, and call this using the $(userName). It will create a NIC and the Virtual Machine for the password i use --admin-password $(ADMIN_PASSWORD) so you need to set this in the Credentials as well.
1
2- stage: DeployVMs
3 displayName: "Deploy VMs using the latest image"
4 jobs:
5 - job: BuildVMs
6 displayName: "Build Virtual Machines"
7 pool:
8 vmImage: "ubuntu-latest"
9 steps:
10 - task: AzureCLI@2
11 displayName: "Create VMs"
12 inputs:
13 azureSubscription: "AVDPipelines"
14 scriptType: bash
15 scriptLocation: inlineScript
16 inlineScript: |
17
18 # Login to Azure
19 az login --service-principal --username $(AVDPipelines) --password $(Secret) --tenant $(tenantId)
20
21 if [ $? -ne 0 ]; then
22 echo "Azure login failed. Please check the service principal credentials."
23 exit 1
24 fi
25 echo "Azure login successful."
26
27 # Confirm subscription
28 az account show
29 # Define parameters
30 locationShortCode=""
31 case "${{ parameters.location }}" in
32 "westeurope") locationShortCode="weu" ;;
33 "northeurope") locationShortCode="neu" ;;
34 *) echo "Unknown location: ${{ parameters.location }}"; exit 1 ;;
35 esac
36
37 resourceGroupName="rg-${{ parameters.productType }}-${{ parameters.environmentType }}-${locationShortCode}"
38
39
40 # Get the latest image version from the shared image gallery
41 sigImageVersion=$(az sig image-version list \
42 --resource-group $resourceGroupName \
43 --gallery-name "gal${{ parameters.productType }}${{ parameters.environmentType }}${locationShortCode}" \
44 --gallery-image-definition "img-${{ parameters.productType }}-${{ parameters.environmentType }}-${locationShortCode}" \
45 --query "[].{Version:name}" -o tsv | sort -V | tail -n 1)
46
47 if [[ -z "$sigImageVersion" ]]; then
48 echo "Error: Could not find the latest image version."
49 exit 1
50 fi
51
52 vmCount=${{ parameters.vmCount }}
53 username="adm_installuser"
54 new_vm_names=""
55
56 for i in $(seq 1 $vmCount); do
57 vmIndex=1
58 while true; do
59 vmname="ins-${{ parameters.environmentType }}-$(printf "%02d" $vmIndex)"
60 existingVM=$(az vm list --resource-group $resourceGroupName --query "[?name=='$vmname']" -o tsv)
61
62 if [[ -z "$existingVM" ]]; then
63 break
64 else
65 ((vmIndex++))
66 fi
67 done
68
69 nicName="nic-$vmname"
70
71 # Get the Subnet Resource ID
72 subnetId=$(az network vnet subnet show \
73 --resource-group $resourceGroupName \
74 --vnet-name "vnet-${{ parameters.productType }}-${{ parameters.environmentType }}-${locationShortCode}" \
75 --name "snet-${{ parameters.productType }}" \
76 --query id -o tsv)
77
78 echo $subnetId
79
80 az network nic create \
81 --resource-group $resourceGroupName \
82 --name $nicName \
83 --subnet $subnetId \
84 --accelerated-networking true \
85 --vnet-name "vnet-${{ parameters.productType }}-${{ parameters.environmentType }}-${locationShortCode}" \
86
87 # Create the VM
88 az vm create \
89 --resource-group $resourceGroupName \
90 --name $vmname \
91 --image "/subscriptions/${{ parameters.subscriptionId }}/resourceGroups/${resourceGroupName}/providers/Microsoft.Compute/galleries/gal${{ parameters.productType }}${{ parameters.environmentType }}${locationShortCode}/images/img-${{ parameters.productType }}-${{ parameters.environmentType }}-${locationShortCode}/versions/$sigImageVersion" \
92 --admin-username $username \
93 --size ${{ parameters.avdSize }} \
94 --authentication-type password \
95 --admin-password $(ADMIN_PASSWORD) \
96 --nics $nicName \
97 --security-type TrustedLaunch \
98 --public-ip-address "" \
99 --license-type Windows_Server \
100 --nsg-rule None \
101
102 echo "VM '$vmname' created successfully."
103 new_vm_names="$new_vm_names $vmname"
104 done
105
106 echo "VMs created: $new_vm_names"
107 echo "##vso[task.setvariable variable=vmNames]$new_vm_names"
- Save the files under your repository in your project and push the content.
If you have finished cloning/copying or modified this yaml file you can store/save/push this to your project.
Let's start the pipeline
We have done the deepdive trough our yaml file, but how can you trigger it? Good question, let me tell you.
- In your DevOps environment and in your project go to the pipeline section
- Click on "new pipeline"
- Choose "Azure Repos Git"
- Select your Repo
- Click on the "existing Azure pipelines yaml file"
- Select your yaml file
- Click on the dropdown button next to run and choose for save
- Run the pipeline
- Fill in you desired values
- Autorise the pipeline this is needed for every first start!!!!
- Wait till the deployment is finished
You have deployed and run your first pipeline!
Wrapping up!
In this blog part 4, I have explained, how you can start a infrastructure deployement, start the image build process and deploy a Virtual Machine all in one go, by using a DevOps pipeline. I have explained how to get started and connect an Azure environment with a DevOps project, when using a Service Principal and explain the difference between an Workload Identity, Service Principals or Managed Identity. You can use those resources to give the Pipeline the necessary rights, to deploy and run specific commands in your environment automated.