Patching and Updating Azure Kubernetes Service (AKS)
Updated: Jul 8, 2022
When it comes to running AKS, it's a good idea to familiarize yourself with the various update processes. Updates to Kubernetes are released frequently. If you don't stay on top of things, you'll quickly find yourself on an unsupported version.
Updates to AKS and the underlying Nodes come in 3 different forms:
Daily patches to the Nodes
Weekly updates to the Node Images
Kubernetes version updates
This post is going to discuss the 3 update options and the differences between them. This post will NOT go into detail on how to actually perform the updates.
Daily Node Patches
Daily patches apply only to Linux Nodes, they do not apply to Windows Nodes.
By default, Linux Nodes automatically receive security patches every evening. These patches come from the distro update channel and include OS security updates or kernel updates.
Some of these patches will require a reboot of the Node. However, the Nodes will not reboot themselves automatically, that responsibility falls onto you. You will know when a Node requires a reboot because it will create a new file on its disk named /var/run/reboot-required.
There is a commonly used, open-source tool called “kured” that is helpful in this situation. Kured stands for KUbernetes REboot Daemon. As the name suggests, it is deployed into your cluster as a DaemonSet, which means it will run one Pod on each Node.
Kured will automatically scan each Node for the existence of the file /var/run/reboot-required. When kured finds this file, it will initiate the process to reboot the Node. Kured does the reboot process in a safe manner. It ensures that only one Node reboots at a time. It also cordons and drains each Node before the reboot. Finally, after the reboot is complete it will uncordon the Node.
Node Images
The disk images used to build your AKS Nodes are updated regularly and include OS patches, kernel updates, Kubernetes security updates, newer versions of binaries (like kubelet), and new versions of components.
Node Images are not tied to Kubernetes versions. You can update your Nodes to the latest Node Image without changing the version of Kubernetes running on those Nodes.
Node Images are updated on a weekly basis. Linux Node Images include new OS patches every week, whereas Windows Node Images include new OS patches every month. Remember, Windows nodes do not receive daily the security patches like Linux Nodes do. So, Node Image upgrades are the primary way to patch your Windows Nodes and make sure they receive the latest security updates.
Some helpful links:
You can find out what’s been updated in each Node Image by reading the release notes here: https://github.com/Azure/AKS/releases
It may take up to a week for a new Node Image to be rolled out to all Azure Regions. You can use the AKS Release Tracker to find out which Node Image is currently available in each Region: https://releases.aks.azure.com/webpage/index.html
Upgrading AKS Versions
Quick recap on semantic versioning
In the following example version number: 1.22.6
1 is the Major version
22 is the Minor version
6 is the Patch version
AKS Minor Version Support
Microsoft supports AKS minor versions using an N-2 standard. That means they will support the current Minor version, as well as the previous 2 Minor versions. So, if the latest supported Minor version is 1.23, then Microsoft will also support 1.22 and 1.21. But, anything older, like 1.20, is unsupported.
It is important to note that you cannot skip Minor versions when doing upgrades. You must go from 1.21 to 1.22 to 1.23. You cannot upgrade directly from 1.21 to 1.23. However, there is one exception to this rule. If you are on a really old version of Kubernetes that is not supported, then you may skip versions and upgrade directly to a supported version of Kubernetes.
AKS Patch Version Support
Microsoft supports AKS patch versions using an N-1 standard. That means they will support the current Patch version, as well as the previous Patch version. For example, say that 1.17 was a currently supported Minor version and the current Patch version was 1.17.9, then Microsoft will also support 1.17.8. But, anything older, like 1.17.7, is unsupported.
AKS Auto-Upgrade
You can configure Auto-Upgrade on your cluster, if desired. The options are:
Patch: automatically updates to the latest supported Patch version within your current Minor version
Stable: automatically updates to the next Minor version, always keeping you at N-1 (one Minor version behind)
Rapid: automatically updates to the latest Minor version, always keeping you on the most current Minor version (latest and greatest)
Node-Image: automatically updates your Node Images as they are released. This keeps you on the same version of Kubernetes.
Note: Preview versions are not used with Auto-Upgrade
Sources