AKS Edge Essentials is lightweight, local installation of AKS that runs on your on-premises hardware. Unlike other on-premise AKS solutions (like Azure Stack HCI), AKS Edge Essentials has very light hardware requirements and can run on Windows 10 or 11 client operating systems. AKS Edge Essentials supports Azure Arc so that you can connect your local cluster to Azure.
As of the time of this writing, only single-machine AKS-EE clusters are fully GA / supported. You can create scalable AKS-EE clusters across multiple physical machines, but this is an experimental / preview feature. With that being said, building an experimental Scalable Cluster is exactly what I'll be going over in this post. Note: the documentation sometimes calls this type of a setup a "Full Deployment."
Each physical machine can run 1, and only 1, Linux VM. Optionally, each physical machine can also run 1, and only 1, Windows VM as well. The first Linux VM serves double-duty as both the AKS control node and the worker node. If you need to run Windows containers, then you'll also need to configure the optional Windows VM which acts as a worker node.
Installing AKS Edge Essentials
AKS-EE currently comes in two forms: the full Kubernetes installation (K8s), or the more lightweight K3s installation.
Download Links (download everything to the same folder):
K8s Installation: aka.ms/aks-edge/k8s-msi
K3s Installation: aka.ms/aks-edge/k3s-msi
If you want to run the optional Windows VM, then you must also download the Windows Node files: aka.ms/aks-edge/windows-node-zip
Extract the zip to the same folder where you downloaded the msi file
To install AKS-EE run the following command:
msiexec.exe /i AksEdge-kXs-x.x.xxx.x.msi ADDLOCAL=CoreFeature,WindowsNodeFeature
Make sure to replace the placeholder msi filename with the real filename that you downloaded
The ADDLOCAL parameter is only needed if you want to run the optional Windows VM
After installation, check to make sure everything works as expected. If it all looks good, then install some prerequisites onto your physical machine:
The command Install-AksEdgeHostFeatures will install Hyper-V, check power management settings, and a few other things. When this command is finished, you will need to restart.
Repeat this process on every machine that you want to add to your Scalable Cluster.
Creating the Scalable Cluster
Creating and modifying AKS-EE clusters is dependent on JSON configuration files. The general process is as follows:
Run a PowerShell command to create a config file
Manually modify the config file to match your environment
Run a PowerShell command to create/modify your cluster by utilizing the config file that you created earlier
Here are the details for creating your first config file:
As you can see, there are a lot of IP addresses involved in creating a scalable cluster. Specifically, you will need the following IP addresses:
One IP address per physical machine in your AKS-EE scalable cluster
One IP address per Linux Node / VM
One IP address per Windows Node / VM
One IP address that will be used for the Kubernetes Control Plane
A range of IP addresses that will be used by Kubernetes Services
Okay, so now that you have modified the configuration file, how do you actually build the scalable cluster?
If all goes well, then after a few minutes you'll have a new AKS cluster with both a Linux node and a Windows node. Helpfully, AKS-EE also installs the kubectl command-line tool, as well as automatically configures your local kubeconfig file. So, that means you should be able to run this command and see your nodes: kubectl get nodes
Adding machines to the Scalable Cluster
To keep it simple, you follow mostly the same process as before, with a few tweaks:
Run a PowerShell command to create a config file:
This uses a different command than before.
This MUST be done from the first physical machine, as it must capture tokens and other sensitive values that will be used to join new machines to the scalable cluster.
Manually modify the config file to match your environment
Transfer the modified config file to the new machine that you would like to add to the cluster
Run a PowerShell command to add the new machine to the cluster by utilizing the config file that you created earlier
This MUST be done from the new machine
Creating the config file:
Don't forget to copy the config file to your new machine, because the next step will need to be run from the new machine.
Add the second machine to the scalable cluster:
If all goes well, then after a few minutes you'll have a new Linux node and a new Windows node added to the cluster. You can run this command to see your nodes: kubectl get nodes
Repeat this process on every machine that you want to add to your Scalable Cluster.
Here are the nodes from my test cluster. The cluster is made up of 3 physical machines, and each one is running both a Linux Node (with control plane) and a Windows Node.
Under the hood of AKS-EE
AKS-EE scalable clusters run on top of Hyper-V:
AKS-EE creates a new "External" Virtual Switch in Hyper-V called aksedgesw-ext. I can clearly see this in the Hyper-V Manager console.
AKS-EE creates a Linux VM and a Windows VM as well. However, I can not see these in the Hyper-V Manager console!
Even though I can't see these VMs in the Hyper-V Manager console, I can still interact with them by way of hcsdiag.exe. When I run hcsdiag list then I will see both VMs:
I can even connect to the consoles of these VMs using the hcsdiag console command:
There is still more I'd like to cover on AKS-EE, including covering some of the problems I had while building my cluster and the troubleshooting that I went through. I would also like to cover how to connect the cluster to my Azure Subscription using Azure Arc.
Sources: