Creating a Kubernetes cluster in Azure with Pulumi (and add Spot Instances)

Kubernetes is an amazing way to create cloud-agnostic applications as well as allowing horizontal scale to happen in a managed way. Almost all the cloud vendors (and even private clouds these days) offer Kubernetes as a managed service, removing the hassle of setting up your own cluster and ensuring it's uptime!

Pulumi Script

Creating the script is quite straightforward and there are some helpful links. To configure the Spot instances, I use the site below to figure out which machine to pick and what the price will be.

https://azureprice.net/?tier=spot&_numberOfCores_min=8&_numberOfCores_max=32&_memoryInMB_min=16&_memoryInMB_max=64&sortField=linuxPrice&sortOrder=true

Next, create an index.ts with the following content:

import * as authorization from "@pulumi/azure-native/authorization/index.js";
import { v20221201 as containerregistry } from "@pulumi/azure-native/containerregistry/index.js";
import * as containerservice from "@pulumi/azure-native/containerservice/index.js";
import { v20220901 as resources } from "@pulumi/azure-native/resources/index.js";
import * as pulumi from "@pulumi/pulumi";

const config = new pulumi.Config();

/**
 * Configure Pulumi Parameters through:
 * pulumi up \
 *    -c glb-location=westeurope \
 *    -c glb-project-name=project-name \
 *    -c glb-project-env=dev
 *
 * You can get the locations with
 * az account list-locations --output table
 */
export const glbLocation = config.get("glb-location");
export const glbProjectName = config.get("glb-project-name");
export const glbProjectEnv = config.get("glb-project-env"); // prd, stg, dev, tst, tmp, ...

/**
 * Fetch the current Azure Subscription Id and Tenant Id
 */
const clientConfig = await authorization.getClientConfig();
export const subscriptionId = clientConfig.subscriptionId;
export const tenantId = clientConfig.tenantId;

// ===============================================================
// Create the main Resource Group
// ===============================================================
const resourceGroup = pulumi
  .all([glbLocation, glbProjectName, glbProjectEnv])
  .apply(([glbLocation, glbProjectName, glbProjectEnv]) => {
    return new resources.ResourceGroup(`rg-${glbProjectName}-${glbProjectEnv}-`, {
      location: glbLocation,
    });
  });

export const rgName = pulumi.interpolate`${resourceGroup.name}`;

// ===============================================================
// Container Registry Configuration
// ===============================================================
const registry = new containerregistry.Registry("acr", {
  resourceGroupName: resourceGroup.name,
  sku: {
    name: "Premium",
  },
  adminUserEnabled: true,
});

const acrCredentials = containerregistry.listRegistryCredentialsOutput({
  resourceGroupName: resourceGroup.name,
  registryName: registry.name,
});

export const acrName = pulumi.interpolate`${registry.name}`;
export const acrLoginServer = pulumi.interpolate`${registry.loginServer}`;
export const acrAdminUser = acrCredentials.apply((credentials) => credentials.username!);
export const acrAdminPass = pulumi.secret(acrCredentials.apply((credentials) => credentials.passwords![0].value!));

// ===============================================================
// Kubernetes Cluster Configuration
// ===============================================================
const k8sCluster = new containerservice.ManagedCluster("aks", {
  resourceGroupName: resourceGroup.name,
  location: resourceGroup.location,

  dnsPrefix: "aks",
  kubernetesVersion: "1.26.3", // az aks get-versions -l eastus -o table
  enableRBAC: true,

  // Assign a managed identity to the cluster
  identity: {
    type: "SystemAssigned",
  },

  // Configure pools
  // 1. Main (normal - frontend, backend, db, etc)
  // 2. Spot (spot)
  agentPoolProfiles: [
    // Main
    {
      name: "main",
      count: 2,
      vmSize: "Standard_B2s", // (2 core, 4GB RAM, 0.041/hour)
      osType: "Linux",
      osSKU: "Ubuntu",
      mode: "System",
    },

    // Spot Instances
    {
      name: "spots",
      vmSize: "Standard_D8as_v5", // (8 vCPU, 32GB RAM, 0.034/hour)
      count: 1,
      minCount: 0,
      maxCount: 3,
      nodeTaints: [
        "kubernetes.azure.com/scalesetpriority=spot:NoSchedule",
      ],
      scaleSetPriority: "Spot",
      spotMaxPrice: 0.05,
      enableAutoScaling: true,
      osType: "Linux",
      osSKU: "Ubuntu",

      // Attach a disk
      osDiskSizeGB: 100,
      osDiskType: "Premium_LRS"
    },
  ],
});


// Export the Kubernetes cluster kubeconfig
const k8sCredentials = containerservice.listManagedClusterUserCredentialsOutput({
  resourceGroupName: resourceGroup.name,
  resourceName: k8sCluster.name,
});

export const kubeconfig = pulumi.secret(k8sCredentials.apply((credentials) => credentials.kubeconfigs![0].value!));

Running Pulumi

Finally, run Pulumi with the below to spin up the Kubernetes cluster

pulumi up \
    --stack composabl \
    -c glb-location=eastus \
    -c glb-project-name=composabl \
    -c glb-project-env=prd

Configuring Kubeconfig

After a bit, our cluster will be created on Azure and we will be able to access it. To do so, create a kubeconfig (from the created secret in the Pulumi script) and configure kubectl to use it:

# Get the kubeconfig file
pulumi stack --stack composabl output kubeconfig --show-secrets | base64 -d > ~/kubeconfig.yaml

# Configure Kubeconfig
export KUBECONFIG=~/kubeconfig.yaml
kubectl get nodes

When running kubectl get nodes we will now see the nodes configured:

NAME                            STATUS   ROLES   AGE     VERSION
aks-main-40673808-vmss000000    Ready    agent   6m7s    v1.26.3
aks-main-40673808-vmss000001    Ready    agent   5m57s   v1.26.3
aks-spots-40673808-vmss000000   Ready    agent   5m42s   v1.26.3

Summary

It is super straightforward to create a Kubernetes cluster! Next up, I'll work and explain how you can use this cluster with Argo to automatically deploy your environment.