MultiKueue via ClusterProfile: Standardize credentials plugins distribution

[hdp617]

What would you like to be added:
MultiKueue with ClusterProfile API requires a credentials plugin to be available to the kueue manager (see KEP-5993). Currently, there is no standardized way to distribute these plugins. This enhancement request aims to standardize the method for distributing and making cloud-provider credential plugins available to the kueue manager.

Proposed Options

Option 1: Include the plugins in the Kueue Manager container image

• Option 1A: Plugins Built and Managed by Kueue. Plugins (e.g., for AWS, GCP, Azure) are compiled and included directly in the official Kueue manager container image, similar to Argo CD.

  • Pros: Offers the best UX as it requires zero extra configuration from the user.
  • Cons: Introduces cloud-provider specific logic in kueue - this might be acceptable as there's prededence from Argo CD and the provider-specific logic is limited to auth.

• Option 1B: Plugins Built and Managed by Customers. Customers are responsible for compiling and building a custom Kueue manager image that includes their required credential plugins.

  • Pros: Keeps cloud-specific logic out of the main Kueue repository.
  • Cons: High friction for users, requiring them to maintain a custom build pipeline.

Option 2: Provide the plugins at run time

Plugins are provided at run time, typically via an initContainer that mounts them into the Kueue manager's volume.

• Pros: Decouples the plugin lifecycle from the Kueue manager release cycle.
• Cons: Adds complexity to the deployment manifest as it requires an initContainer, volume mounts, etc. It also requires maintaining container images for the plugins - it would be useful to have the plugins hosted in a k8s repos and shared among multi-cluster toolings.

Why is this needed:

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[hdp617]

cc @mimowo @zhang-xuebin @knee-berts

[tenzen-y]

I think both should be provided since hosting plugins for major cloud providers by the upstream Kueue controller-manager would be really useful. The cluster admin just installed kueue the same as today.

OTOH, I think that we should open an interface for on-prem or minor regional cloud providers.

At the first step, we can work for the major cloud providers (Google Cloud?).

Plugins are provided at run time, typically via an initContainer that mounts them into the Kueue manager's volume.

I'm not familiar with the ClusterProfile mechanism, but can we host such a mechanism in another Pod? Because we have some of the extensible interfaces like JobFramework, AdmissionChecks, MKDispathcers which typically are implemented in another Por rather than upstream kueue-controller-manager.

[mimowo]

I have added a related topic to the agenda for the next wg-batch (Dec 4th).

While I think it makes sense to provide the plugins along with Kueue for the ease of deployment, I would prefer not to need to compile or maintain them. Just use them as a dependency on the image built by some centralized repository for the plugins. This would also allow to deduplicate the implementations for the plugins with argo-cd.

I think such plugins could be built and published as images by the https://github.com/kubernetes-sigs/cluster-inventory-api project, similarly is Kueue is publishing images.

[tenzen-y]

@kannon92 Could you share if we should implement Openshit dedicated credential mechanism for MultiKueue? Or does Openshit rely on any standardized credential mechanism?

[tenzen-y]

I think such plugins could be built and published as images by the https://github.com/kubernetes-sigs/cluster-inventory-api project, similarly is Kueue is publishing images.

If cluster-inventory-api could manage plugins, it would be better, I think.

[qiujian16]

I'm not familiar with the ClusterProfile mechanism, but can we host such a mechanism in another Pod? Because we have some of the extensible interfaces like JobFramework, AdmissionChecks, MKDispathcers which typically are implemented in another Por rather than upstream kueue-controller-manager

The required plugin is a bin that can be executed in kubeconfig. One way is to start a init container and copy the plugin binary to the volume that kueue manager can mount later. Or I think it is possible to directly mount image with plugin as volume today.

I remember the previous discussion on cluster-inventory-api is it will not maintain plugins for different vendors. But I think it worth a discussion to at least document how to use the plugin. cc @corentone

[mimowo]

Yes, I think it will be possible to mount the plugin directly using the ImageVolume feature which is going to be enabled by default in 1.35