Reduce amount of apiserver writes on workloads that didn't admit.

[mwielgus]

What happened:

Too many api server writes cause slow admission processing on best effort queues with heterogenous CQ (for example containig workloads both requiring and not requiing gpus and corresponding RF with and without GPUs)

What you expected to happen:

Fast processing and admission.

How to reproduce it (as minimally and precisely as possible):

Create large number of CPU and GPU workloads queuing in CQ with 2 RF - one with GPU, one without. Stucked GPU workloads slow down CPU workloads processing.

cc: @mimowo

[mimowo]

I think this is duplicate / closely related to #8081. Im not totally sure we have excessive number of api server writes. Thus i marked the issue as investigation

[sohankunkerkar]

/assign @sohankunkerkar

[sohankunkerkar]

I reproduced it locally. GPU workloads blocked the BestEffortFIFO queue, delaying CPU-only jobs and causing excessive API updates. I’m thinking of a fix that tackles both throughput and API noise.

First, for throughput, the scheduler could look at a small batch of workloads at the front of the queue instead of just one. That way, CPU-only workloads stuck behind GPU-heavy ones aren’t blocked unnecessarily. We’d still enforce same-priority rules so higher-priority jobs always come first.

Second, to reduce API writes, we could skip status updates if nothing has actually changed. That should cut down on repeated “pending” events caused by workloads being retried over and over (UnsetQuotaReservationWithCondition always writes the QuotaReserved=False condition, even if the workload already has that exact condition with the same reason and message).

So overall, this keeps the BestEffort semantics intact while letting the queue make forward progress and reducing unnecessary load.

@mimowo @mwielgus Any thoughts?

[mimowo]

First, for throughput, the scheduler could look at a small batch of workloads at the front of the queue instead of just one.

That is a possibility, but it is tricky. Some environments may have a CPU workload as deep as say 200 positions, so look ahead in scheduler for 200 positions would be costly in itself. Also, it is a complex change than it seems in the context of cohorts.

Second, to reduce API writes, we could skip status updates if nothing has actually changed. That should cut down on repeated “pending” events caused by workloads being retried over and over (UnsetQuotaReservationWithCondition always writes the QuotaReserved=False condition, even if the workload already has that exact condition with the same reason and message).

This sounds interesting and relatively easy, so we could start here. How does it happen if we have this code in place? https://github.com/kubernetes-sigs/kueue/blob/main/pkg/scheduler/scheduler.go#L800-L804 the intention of the code is specifically to avoid sending the request if nothing changed.

[sohankunkerkar]

This sounds interesting and relatively easy, so we could start here. How does it happen if we have this code in place? https://github.com/kubernetes-sigs/kueue/blob/main/pkg/scheduler/scheduler.go#L800-L804 the intention of the code is specifically to avoid sending the request if nothing changed.

IIUC, the existing check is right, but it only patches when something actually changes. The issue is that the inadmissible message content changes between scheduling cycles. The message from https://github.com/kubernetes-sigs/kueue/blob/main/pkg/scheduler/flavorassigner/flavorassigner.go#L988-L989 contains dynamic values like "X more needed" which changes as cluster state changes (e.g., "3 more needed" → "2 more needed" when quota frees up). Since SetStatusCondition compares Status, Reason, Message, and ObservedGeneration, a change in any of these fields triggers an API write, and the Message changes frequently.

[mimowo]

Great news @sohankunkerkar !

I think the next step should be to somehow summarize the baseline timings so that we can assess the improvements etc.

---

Can you please provide some numbers you use for the experiments? Or even it would be great if you could script the experiment, then we can observe the improvements in a quantitative way.

For example, I'm thinking about:

• saturated GPU quota and 100 GPU workloads in the queue
• 1000 CPU workloads running quickly

Then we could measure how long it takes to process the queue of the CPU workloads. Before / after changes.

---

Once we have the experimental setup which demonstrates the issue, I would like to have also results for two experiments:

1. code modified so that the message is always the same (updates skipped)
2. code modified so that the message is always the same, and no event is sent

---

As for the actual fix I think we could for example maintain some in-memory state, and make sure we only send one update in O(1) seconds, say 5s. Yes, this will mean that sometimes the messages for "Pending" are outdated, but (1.) we are always accurate about the "Status" and "Reason", just message might be lagging.

[mimowo]

cc @mwysokin