Development of your linear pipeline

After a few days of development, you have successfully put together a comprehensive Azure DevOps pipeline to deploy a whole heap of infrastructure and components for a new service. It’s all working well, but there is just one annoying problem, and that’s how long it’s taking to deploy!

Because of the size of each deployment step which may include some very slow Azure virtual machine deployments that can take anywhere from 30 to 40 minutes, it’s taking way too long! Sometimes, even more than 60 minutes, which means you’re now hitting the timeout limit on a Microsoft-hosted agent with a private project/repository. Oh no, what now?!

Introducing parallel pipelines

If you’re at this point, you’re not alone. Over the past week, I’ve been tackling ways in which I could optimise and improve an Azure DevOps pipeline that is deploying quite a few components. This pipeline was very linear, but it didn’t need to be as there were a few Azure Powershell tasks that didn’t have dependencies on each other, so why not run them in parallel? Here was my thinking to get the pipeline, knowing each steps timings, under 60 minutes again.

Layout and understand what you can run in parallel

First, to start optimising your pipeline, I recommend laying out and understanding your dependencies between each deployment step. This understanding can be quite time-consuming, especially if you’re not entirely familiar with each step in a pipeline. However, do take the time to get this right, as it will save you considerable time in the long run as I didn’t do this work upfront myself. Mapping out the pipeline could end up looking something like it did below for me.

As you can see, there were quite a few dependencies on each PowerShell script (step), but there were some things that could run at the same time. As a matter of fact, in my mapping, I actually found where I could have three jobs running at the same time, which helped reduce the pipeline from 60 minutes to about 35 minutes! 

Making the switch to parallel pipelines

To start, I needed to make sure of a couple of things. First and foremost, I made sure I had purchased some additional Microsoft-hosted agents parallel jobs as I was running this in a private project/repository. Secondly, I made sure that all of my published variables were output variables by setting isOutput=true on the task.setvariable logging command. Without doing this, variables cannot be passed between jobs.

At this point, I then needed to switch my deployment steps pipeline template (a pipeline YAML file sitting under azure-pipelines.yml) to the higher level of a deployment jobs template. Jobs are the lowest point where you can have them running in parallel, and with the added support of re-running failed jobs without needing to re-run the entire pipeline, it made a lot of sense to go to this level. 

 

Old Pipeline New Pipeline

azure-pipelines.yml

-- deployment-steps.yml

azure-pipelines.yml

-- deployment-jobs.yml

 

What was done previously in the deployment-steps.yml. 

What is done with the new deployment-jobs.yml

 

Variable syntax

As you can tell with these YAML extracts, the main change between old and new is how variables are passed between jobs and tasks in the pipeline. In the old deployment-steps.yml, the default macro syntax can be used, but with the new deployment-jobs.yml pipelines, the expression needs to include the dependency on the previous step/job. For example, the Deploy-Identities.ps1 produces the variable “azureADGroupId_resourceGroupContributor”. To use this variable elsewhere, you must;

  • For a step/task in the same job, you need to specify the name of the previous step/task. i.e:

    $(Identities.azureADGroupId_resourceGroupContributor)

  • For another jobs step/task, you need to:
    • First, define the dependsOn to the previous job(s) that must be completed first before starting this job:

      dependsOn: 
        - deployADPFoundation

    • Second, define the variable in the receiving job: 

      azureADGroupId_resourceGroupContributor: $[ dependencies.deployADPFoundation.outputs['deployADPFoundation.Core. Identities.azureADGroupId_resourceGroupContributor’] ]

    • Third, use the variable as you would normally with a macro syntax variable: 

      $(azureADGroupId_resourceGroupContributor)
       

It’s crucial to note that for deployment jobs in Azure DevOps pipelines, the matrices syntax for runOnce, canary and rolling strategies does vary for variable expression syntax. See the exact syntax you need by reviewing this Microsoft Docs page (Support for output variables). I hope you don’t make the same mistake I did where I was not defining the job name! I was mistakenly referring to the build job documentation, which does not need this defined.

With the variables passing through correctly, we now have a pipeline that runs in almost half the time as it did before, well below any timeout threshold!

Here is a screenshot from the Azure DevOps pipeline in action, running three jobs in parallel! 

Conclusion

As you can see, saving almost half the time in a pipeline deployment provides a massive advantage, not only technically but business-impacting as well. Parallel pipelines can help organisations reduce downtime while deploying releases, while also giving peace of mind to developers and operations teams that everything is being deployed in the right order.  

I encourage you to utilise parallel pipelines jobs in the next major update of your Azure DevOps pipeline(s)!