top of page
Search
  • Writer's pictureNathan

Azure DevOps YAML Pipelines - Part 5

Updated: Jan 21

Pools and Agents


There are a lot of topics to cover for YAML-based Pipelines in Azure DevOps. So, I will be breaking this up into a multi-part series. This section will be updated over time with links to the whole series of articles once they are written.

 

Agent Pools


Agent Pools are just a collection of Agent machines that run your Pipeline Jobs. There are Microsoft-hosted Pools that you can use free of charge (with some limits). Or, you can create your own Self-Hosted Agents that are attached to your own Self-Hosted Pool.


By default, your Azure DevOps Organization comes with 2 Agent Pools:

  1. Azure Pipelines: this is the default Microsoft-hosted Pool, which can use various Ubuntu, MacOS, and Windows Server Agents

  2. Default: if desired, you can use this one to attach your Self-hosted Agents

In your Pipeline, you can define which Agent Pool to use at 3 different places. You can do it at the Pipeline-level, the Stage-level, or the Job-level:

Pipeline-level is the least specific, while Job-level is the most specific. More specific Pools take priority or less specific ones.


Defining a Pool is completely optional. By default, a YAML Pipeline will use the Microsoft-hosted Pool, and it will use the 'ubuntu-latest' Agent.


There are 4 different ways you can define an Agent Pool:

 

Agents


Microsoft-hosted Agents

Microsoft-hosted Agents are fully managed by Microsoft. You can pick from a handful of supported Agent operating systems, see the full list here. Each Microsoft-hosted Agent comes with a large amount of pre-installed software (which you can also find via the previous link). Microsoft takes care of maintaining, upgrading, and patching these Agents. You have no control over the hardware specs (CPU, RAM, etc.) of the Microsoft-hosted Agents. Each Microsoft-hosted Agent is only used for one Job and then destroyed immediately afterwards. Since these are hosted by Microsoft, traffic between Microsoft-hosted Agents and your private resources will be over the public internet. Lastly, there are limits on the amount of Parallel Jobs and the total amount of time that Jobs can run on Microsoft-hosted Agents.


Self-hosted Agents

Self-hosted Agents are machines that you own and manage. Self-hosted Agents are supported on a wider range of Operating Systems, most notably more flavors of Linux and even Windows Client machines. If you wanted, you could even run a Self-Hosted Agent inside of a Docker container. You must install and configure the Agent software on these machines. You are responsible for installing all of the necessary software required by your Pipelines on these machines. You must upgrade and patch these machines yourself. You have full control over the hardware specs of these machines. Self-hosted Agents are NOT destroyed after running a Job, so they are good for running incremental builds.

Microsoft-hosted Agents

Self-hosted Agents

Supported OS

MacOS, Ubuntu, Windows Server (full list)

Updates & maintenance

Handled by Microsoft

Your responsibility

Pre-installed software

Handled by Microsoft

Your responsibility

Hardware Specs

Fixed:

- 2 core, 7 GB RAM for Ubuntu & Windows

- 3 core, 14 GB RAM for MacOS

Your choice

Networking

Public only

Your choice

Destroyed after each Job

Yes

No (but can be configured on VMSS Agents)

Parallel Jobs

10 for Public Project

1 for Private Project

Unlimited for Public Project

1 for Private Project

Individual Job Time Limit

6 hours for Public Project

1 hour for Private Project

Unlimited

Monthly Time Limit

Unlimited for Public Project

30 hours for Private Project

Unlimited

Auto-scaled VMSS Agents

Not supported

Supported

Container Agents

Not supported

Supported

Container Jobs

Supported for Ubuntu & Windows Server

Supported for Linux (except RHEL 6) and Windows


 

Container Jobs


By default, Jobs run directly on the Agent machine. However, you also have the option of running your Jobs inside of a Container which is running on the Agent machine. This is supported by Linux and Windows Agents, both Microsoft-hosted and Self-hosted. This is not supported by MacOS Agents, or by Self-hosted RedHat Enterprise Linux 6 Agents.


Linux Agents can run Linux-based Containers that meet certain requirements. Windows Agents can run Windows-based Containers, with a few caveats:

  • The Windows Server container image must match the kernel version of the Windows Agent machine where it is running. For example, you must run a Windows 2019 Container on a Windows 2019 Agent

  • The Windows Server Nano container image is missing some required dependencies

Container Jobs will be covered in more detail in a future post.

 

Container Agents


What if you didn't want to install the Agent and all the supporting software directly on your machines? What if you wanted to run all of that stuff inside of a container, instead? Well that is possible with Container Agents. And, with all of your Agents running on Containers, you could even use technologies such as Azure Kubernetes Services and Azure Container Instances to manage and run them.


Container Agents are only supported for Self-hosted Pools. Linux host machines can run Container Agents that are based on an Ubuntu container image. Windows host machines can run Container Agents that are based on a Windows Server Core container image.


Nested containers cannot be used. In other words, you cannot run a Container Job on a Container Agent.


More details on Container Agents can be found here.

 

Virtual Machine Scale Set (VMSS) Agents


VMSS Agents are a form of Self-hosted Agents, and therefore enjoy all of the same benefits as Self-hosted Agents. VMSS Agents are fully controlled and automatically scaled for you by Azure Pipelines.


At a high level, you simply create the VMSS yourself using a supported OS (Ubuntu, Windows Server or Client), turn off Azure-based autoscaling (as Azure Pipelines will handle that), attach it to the Self-hosted Pool of your choice, and finally configure it with your desired settings.


Some of the interesting settings that you can configure are:

  • Configure the max number of VMs in the scale set

  • Configure the minimum number of VMs to always keep on standby, even if no Jobs are running. You can even configure a TTL to control how long an idle VM will be kept running before it is decommissioned

  • Make each VM instance single use. Meaning, after it runs a Job the VM instance will go offline and reimage itself automatically

Azure Pipelines automatically takes care of installing the Agent software for every new VM instance that it provisions. In other words, you do not need to pre-install it in the image that you use.


More details on VMSS Agents can be found here.

 

To summarize, we talked about Agent Pools. We also talked about Microsoft-hosted Agents and Self-hosted Agents. We discussed Container Agents and running Jobs inside of Containers. Lastly, we briefly discuss VMSS Agents.


Next up we'll be discussing Parameters and Variables.

Part 6 - Parameters and Variables



1,267 views
bottom of page