Description
While scale testing kubernetes-sigs/karpenter, we created thousands of pending pods at one time. We have a cache field indexer on pending pods that we use to get all of the pending pods on the cluster before we start executing our scheduling loop. We noticed (under high load) that the pending pods were deployed to the cluster but Karpenter was not attempting to schedule these pending pods/did not retrieve these pending pods back when pulling them from the field indexer.
In particular, this seemed like the field indexer may be single threaded or may be bottlenecking under high load. What was stranger is that Karpenter has other threads that it runs in its containers where it publishes metrics on pods. These threads (which weren't using the field indexers) observed the pending pods on the cluster much quicker than the thread that was using the field indexers.
Do we know if there is anything that could cause the field indexers to be slow when placed under high load (are they single threaded) and are there ways that we could improve this performance so that we can rely on these indexers? Or should we look to remove them if we care deeply about the performance of retrieving these pods?