Add cloud-provider for Slicer

Slicer launches microVMs on KVM-enabled hosts with K3s and a
join token to add and remove new nodes dynamically, which are
fully isolated through Firecracker.

Signed-off-by: Alex Ellis (OpenFaaS Ltd) <[email protected]>

Author: alexellis

.github/workflows/publish.yaml

@@ -0,0 +1,33 @@
+name: publish
+
+on:
+  push:
+    tags:
+      - "*"
+
+jobs:
+  publish:
+    runs-on: actuated-8cpu-12gb
+    steps:
+      - uses: self-actuated/hub-mirror@master
+      - uses: actions/checkout@v6
+        with:
+          fetch-depth: 1
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v3
+
+      - name: Get TAG
+        id: get_tag
+        run: echo TAG=${GITHUB_REF#refs/tags/} >> $GITHUB_ENV
+
+      - name: Login to Docker Registry
+        uses: docker/login-action@v3
+        with:
+          username: ${{ github.repository_owner }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+          registry: ghcr.io
+
+      - name: Build and push
+        run: |
+          REGISTRY=ghcr.io/openfaasltd BUILD_TAGS=slicer TAG=${{env.TAG}} make ALL_ARCH="amd64 arm64" execute-release
+          REGISTRY=ghcr.io/openfaasltd BUILD_TAGS=slicer TAG=latest make ALL_ARCH="amd64 arm64" execute-release

cluster-autoscaler/.gitignore

@@ -11,3 +11,4 @@
 Session.vim
 .netrwhist
 .vscode
+/kubeconfig

cluster-autoscaler/charts/cluster-autoscaler/templates/clusterrole.yaml

@@ -56,7 +56,7 @@ rules:
 {{- if (eq .Values.cloudProvider "kwok") }}
     - create
 {{- end }}
-{{- if or (eq .Values.cloudProvider "kwok") (eq .Values.cloudProvider "huaweicloud") }}
+{{- if or (eq .Values.cloudProvider "kwok") (eq .Values.cloudProvider "huaweicloud") (eq .Values.cloudProvider "slicer") }}
     - delete
 {{- end }}
     - get

cluster-autoscaler/cloudprovider/builder/builder_all.go

@@ -48,6 +48,7 @@ import (
 	"k8s.io/autoscaler/cluster-autoscaler/cloudprovider/ovhcloud"
 	"k8s.io/autoscaler/cluster-autoscaler/cloudprovider/rancher"
 	"k8s.io/autoscaler/cluster-autoscaler/cloudprovider/scaleway"
+	"k8s.io/autoscaler/cluster-autoscaler/cloudprovider/slicer"
 	"k8s.io/autoscaler/cluster-autoscaler/cloudprovider/tencentcloud"
 	"k8s.io/autoscaler/cluster-autoscaler/cloudprovider/utho"
 	"k8s.io/autoscaler/cluster-autoscaler/cloudprovider/volcengine"
@@ -161,6 +162,9 @@ func buildCloudProvider(opts *coreoptions.AutoscalerOptions,
 		return utho.BuildUtho(opts, do, rl)
 	case cloudprovider.CoreWeaveProviderName:
 		return coreweave.BuildCoreWeave(opts, do, rl)
+	case cloudprovider.SlicerProviderName:
+		return slicer.BuildSlicer(opts, do, rl)
 	}
+
 	return nil
 }

cluster-autoscaler/cloudprovider/cloud_provider.go

@@ -94,6 +94,8 @@ const (
 	RancherProviderName = "rancher"
 	// UthoProviderName gets the provider name of utho
 	UthoProviderName = "utho"
+	// SlicerProviderName gets the provider name of slicer
+	SlicerProviderName = "slicer"
 )
 
 // GpuConfig contains the label, type and the resource name for a GPU.

cluster-autoscaler/cloudprovider/slicer/README.md

@@ -0,0 +1,114 @@
+## Slicer Cloud Provider for Cluster Autoscaler
+
+The cluster autoscaler for [Slicer](https://slicervm.com/) scales nodes on lightweight slicer microVMs.
+
+The architecture is as follows:
+
+* Slicer runs a K3s control plane on one virtualisation host.
+* Slicer runs all agents on one or more additional virtualisation hosts running the Slicer REST API. Starting off with zero microVMs and relying on cluster autoscaler to add new ones as Pods cannot be scheduled to the existing set of nodes.
+
+Check the documentation on [SlicerVM.com](https://docs.slicervm.com/examples/autoscaling-k3s/) for instructions on how to setup the cluster-autoscaler with with Slicer.
+
+## Configuration
+
+The `cluster-autoscaler` with Slicer needs a configuration file to work by using the `--cloud-config` parameter, it is an INI file with the following fields:
+
+| Key | Value | Mandatory | Default |
+|-----|-------|-----------|---------|
+| `global/k3s-url` | The URL of the K3s control plane API server | yes | none |
+| `global/k3s-token` | The K3s join token for adding new agent nodes | yes | none |
+| `global/default-min-size` | Default minimum size of a node group (must be > 0) | no | 1 |
+| `global/default-max-size` | Default maximum size of a node group | no | 8 |
+| `nodegroup \"slicer_host_group_name\"/slicer-url` | The URL of the Slicer API server for this node group | yes | none |
+| `nodegroup \"slicer_host_group_name\"/slicer-token` | The authentication token for the Slicer API server | yes | none |
+| `nodegroup \"slicer_host_group_name\"/min-size` | Minimum size for a specific node group | no | global/defaut-min-size |
+| `nodegroup \"slicer_host_group_name\"/max-size` | Maximum size for a specific node group | no | global/defaut-max-size |
+
+## Development
+
+Follow the instructions in the [slicer docs](https://docs.slicervm.com/examples/autoscaling-k3s/) to setup a K3S cluster and host groups for nodes.
+
+Make sure you are inside the `cluster-autoscaler` path of the [autoscaler repository](https://github.com/kubernetes/autoscaler).
+
+### Run out of cluster
+
+Start the cluster-autoscaler:
+
+```bash
+#!/bin/bash
+go run . \
+    --cloud-provider=slicer \
+    --kubeconfig $HOME/k3s-cp-kubeconfig \
+    --scale-down-enabled=true \
+    --scale-down-delay-after-add=30s \
+    --scale-down-unneeded-time=30s \
+    --expendable-pods-priority-cutoff=-10 \
+    --cloud-config="$HOME/cloud-config.ini" \
+    --v=4
+```
+
+### Run in cluster.
+
+Build and publish an image:
+
+```sh
+REGISTRY=ttl.sh/openfaasltd BUILD_TAGS=slicer TAG=dev make dev-release
+```
+
+Create a the cloud-config secret:
+
+```sh
+kubectl create secret generic cluster-autoscaler-cloud-config \
+  --from-file=cloud-config=cloud-config.ini \
+  -n kube-system
+```
+
+Create a `values.yaml` for the cluster-autoscaler chart:
+
+```yaml
+image:
+  repository: ttl.sh/openfaasltd/cluster-autoscaler-slicer-amd64
+  tag: dev
+
+cloudProvider: slicer
+
+fullnameOverride: cluster-autoscaler-slicer
+
+autoDiscovery:
+  clusterName: k3s-slicer
+
+# Mount the cluster-autoscaler-cloud-config secret
+extraVolumeSecrets:
+  cluster-autoscaler-cloud-config:
+    name: cluster-autoscaler-cloud-config
+    mountPath: /etc/slicer/
+    items:
+      - key: cloud-config
+        path: cloud-config
+
+# All your required parameters
+extraArgs:
+  cloud-config: /etc/slicer/cloud-config
+  # Standard logging
+  logtostderr: true
+  stderrthreshold: info
+  v: 4
+
+  scale-down-enabled: true
+  scale-down-delay-after-add: "30s"
+  scale-down-unneeded-time: "30s"
+  expendable-pods-priority-cutoff: -10
+```
+
+Deploy with Helm:
+
+```sh
+helm install cluster-autoscaler-slicer charts/cluster-autoscaler \
+  --namespace=kube-system \
+  --values=values.yaml
+```
+
+To test the autoscaler do one of the following:
+
+* Scale a deployment higher than can fit on the current set of control-plane nodes, then wait for the autoscaler to scale up the cluster.
+* Or, create a taint / affinity / anti-affinity rule that will prevent a pod from being scheduled to the existing set of nodes, then wait for the autoscaler to scale up the cluster.