[Feature] Multi-cluster priority scheduling

[vie-serendipity]

What would you like to be added:

Enhance old API or introduce a new API, like PreferredClusterAffinities, to implement priority scheduling across multiple affinities and clusters. This indicates that when scheduling, a cluster's priority should be considered first, not just its resources or weight. If the high-priority cluster lacks resources, then lower-priority clusters should be considered subsequently.

Why is this needed:

In multi-clusters scenarios, scheduling shouldn't solely consider cluster's resources. Multi-clusters might have different priorities, considering cloud provider, cost-efficiency, cluster location, etc. Especially in hybrid cloud scenarios, on-premises clusters should be prioritized over cloud-based clusters. The scheduler should fill on-premises clusters first and then utilize the cloud resources.

User Story

1. As a user, I have an on-premises cluster as well as a cloud-based cluster, and I use FHPA for autoscaling. I want my workloads to prioritize using my on-premises cluster, and only utilize the cloud cluster when resources are insufficient to manage costs effectively. It's crucial that during scale-in, the prioritization is maintained, with cloud-based replicas being removed first.
2. As a user, I aim for cluster disaster recovery. Currently, using ClusterAffinities, workloads don't migrate back to the cluster once it's restored.

[vie-serendipity]

/cc @zhzhuang-zju

[zhzhuang-zju]

Thanks! I’ve added this to next week’s meeting agenda—maybe we can discuss it together.

[RainbowMango]

Great, thanks for bringing this up. Love to help enhance this based on more and more users' feedback.