top of page
Search
  • Writer's pictureNathan

AKS Edge Essentials - Scalable Cluster

Updated: Feb 10

AKS Edge Essentials is lightweight, local installation of AKS that runs on your on-premises hardware. Unlike other on-premise AKS solutions (like Azure Stack HCI), AKS Edge Essentials has very light hardware requirements and can run on Windows 10 or 11 client operating systems. AKS Edge Essentials supports Azure Arc so that you can connect your local cluster to Azure.


As of the time of this writing, only single-machine AKS-EE clusters are fully GA / supported. You can create scalable AKS-EE clusters across multiple physical machines, but this is an experimental / preview feature. With that being said, building an experimental Scalable Cluster is exactly what I'll be going over in this post. Note: the documentation sometimes calls this type of a setup a "Full Deployment."


Each physical machine can run 1, and only 1, Linux VM. Optionally, each physical machine can also run 1, and only 1, Windows VM as well. The first Linux VM serves double-duty as both the AKS control node and the worker node. If you need to run Windows containers, then you'll also need to configure the optional Windows VM which acts as a worker node.

 

Installing AKS Edge Essentials


AKS-EE currently comes in two forms: the full Kubernetes installation (K8s), or the more lightweight K3s installation.


Download Links (download everything to the same folder):


To install AKS-EE run the following command:

  • msiexec.exe /i AksEdge-kXs-x.x.xxx.x.msi ADDLOCAL=CoreFeature,WindowsNodeFeature

  • Make sure to replace the placeholder msi filename with the real filename that you downloaded

  • The ADDLOCAL parameter is only needed if you want to run the optional Windows VM


After installation, check to make sure everything works as expected. If it all looks good, then install some prerequisites onto your physical machine:



The command Install-AksEdgeHostFeatures will install Hyper-V, check power management settings, and a few other things. When this command is finished, you will need to restart.



Repeat this process on every machine that you want to add to your Scalable Cluster.

 

Creating the Scalable Cluster


Creating and modifying AKS-EE clusters is dependent on JSON configuration files. The general process is as follows:

  1. Run a PowerShell command to create a config file

  2. Manually modify the config file to match your environment

  3. Run a PowerShell command to create/modify your cluster by utilizing the config file that you created earlier


Here are the details for creating your first config file:


As you can see, there are a lot of IP addresses involved in creating a scalable cluster. Specifically, you will need the following IP addresses:

  • One IP address per physical machine in your AKS-EE scalable cluster

    • One IP address per Linux Node / VM

    • One IP address per Windows Node / VM

  • One IP address that will be used for the Kubernetes Control Plane

  • A range of IP addresses that will be used by Kubernetes Services


Okay, so now that you have modified the configuration file, how do you actually build the scalable cluster?


If all goes well, then after a few minutes you'll have a new AKS cluster with both a Linux node and a Windows node. Helpfully, AKS-EE also installs the kubectl command-line tool, as well as automatically configures your local kubeconfig file. So, that means you should be able to run this command and see your nodes: kubectl get nodes

 

Adding machines to the Scalable Cluster


To keep it simple, you follow mostly the same process as before, with a few tweaks:

  1. Run a PowerShell command to create a config file:

    1. This uses a different command than before.

    2. This MUST be done from the first physical machine, as it must capture tokens and other sensitive values that will be used to join new machines to the scalable cluster.

  2. Manually modify the config file to match your environment

  3. Transfer the modified config file to the new machine that you would like to add to the cluster

  4. Run a PowerShell command to add the new machine to the cluster by utilizing the config file that you created earlier

    1. This MUST be done from the new machine


Creating the config file:


Don't forget to copy the config file to your new machine, because the next step will need to be run from the new machine.


Add the second machine to the scalable cluster:


If all goes well, then after a few minutes you'll have a new Linux node and a new Windows node added to the cluster. You can run this command to see your nodes: kubectl get nodes


Repeat this process on every machine that you want to add to your Scalable Cluster.


Here are the nodes from my test cluster. The cluster is made up of 3 physical machines, and each one is running both a Linux Node (with control plane) and a Windows Node.


 

Under the hood of AKS-EE


AKS-EE scalable clusters run on top of Hyper-V:

  • AKS-EE creates a new "External" Virtual Switch in Hyper-V called aksedgesw-ext. I can clearly see this in the Hyper-V Manager console.

  • AKS-EE creates a Linux VM and a Windows VM as well. However, I can not see these in the Hyper-V Manager console!



Even though I can't see these VMs in the Hyper-V Manager console, I can still interact with them by way of hcsdiag.exe. When I run hcsdiag list then I will see both VMs:



I can even connect to the consoles of these VMs using the hcsdiag console command:


 

There is still more I'd like to cover on AKS-EE, including covering some of the problems I had while building my cluster and the troubleshooting that I went through. I would also like to cover how to connect the cluster to my Azure Subscription using Azure Arc.

 

Sources:

326 views

Comments


bottom of page