Skip to content

When joining new member cluster, the karmada-scheduler will reschedule some rbs even if their replicas is 0 #6156

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
LivingCcj opened this issue Feb 24, 2025 · 3 comments · May be fixed by #6203
Open

When joining new member cluster, the karmada-scheduler will reschedule some rbs even if their replicas is 0 #6156

LivingCcj opened this issue Feb 24, 2025 · 3 comments · May be fixed by #6203
Assignees
@LivingCcj
Labels
kind/bug Categorizes issue or PR as related to a bug.
Milestone
v1.14

Comments

@LivingCcj
Copy link
Contributor

What happened:

when joining a new member cluster, karmada-schduler will reschedule some resourcebindings(rb). Even if the replica of rb is 0 and point to workload, karmada-scheudler will binding the new member to rb and karmada-controller-manager will create the work and workload.

here is the relevant code

karmada/pkg/scheduler/scheduler.go

Lines 393 to 401 in 006cf70

if rb.Spec.Replicas == 0 ||
rb.Spec.Placement.ReplicaSchedulingType() == policyv1alpha1.ReplicaSchedulingTypeDuplicated {
// Duplicated resources should always be scheduled. Note: non-workload is considered as duplicated
// even if scheduling type is divided.
klog.V(3).Infof("Start to schedule ResourceBinding(%s/%s) as scheduling type is duplicated", namespace, name)
err = s.scheduleResourceBinding(rb)
metrics.BindingSchedule(string(ReconcileSchedule), utilmetrics.DurationInSeconds(start), err)
return err
}

What you expected to happen:
If the replica of rb is 0 and point to workload , karmada-scehduler should ignore to reschedule these rbs.

Anything else we need to know?:
the replicaSchedulingType of their PropagationPolicy is Divided

apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: test-pp
  namespace: test
spec:
  conflictResolution: Abort
  placement:
    clusterAffinity:
      labelSelector:
        matchExpressions:
        - key: zone
          operator: In
          values:
          - test001
    replicaScheduling:
      replicaDivisionPreference: Weighted
      replicaSchedulingType: Divided
      weightPreference:
        dynamicWeight: AvailableReplicas
  preemption: Never
  priority: 0
  propagateDeps: true
  resourceSelectors:
  - apiVersion: apps/v1
    kind: Deployment
    name: test-pp
    namespace: test
  schedulerName: default-scheduler

Environment:

  • Karmada version: 1.9.1
  • kubectl-karmada or karmadactl version (the result of kubectl-karmada version or karmadactl version): v1.4.1
  • kubernetes: 1.20.10
@LivingCcj LivingCcj added the kind/bug Categorizes issue or PR as related to a bug. label Feb 24, 2025
@RainbowMango RainbowMango added this to the v1.14 milestone Feb 24, 2025
@LivingCcj LivingCcj linked a pull request Mar 18, 2025 that will close this issue
Open
@RainbowMango
Copy link
Member

/assign @LivingCcj
In favor of #6203

@LivingCcj
Copy link
Contributor Author

When the ‌karamda scheduler‌ schedules ‌rb objects‌, some scenarios require scheduling decisions based on the type of ‌object‌ that the rb object points to. Therefore, we need to add a field to the rb object to identify the ‌resource type‌ of the target object. Below are several approaches to determine the resource type of the target object when creating an rb resource:

  1. ‌Check replicaRequirements in GetReplicas‌:
    If the target object's replicaRequirements in GetReplicas is ‌non-null‌, it indicates a ‌workload type‌; otherwise, it is a ‌non-workload type‌.
  2. ‌Check for scale subresource‌:
    If the target object has a ‌scale subresource‌, it is considered a ‌workload type‌; otherwise, it is a ‌non-workload type‌.
  3. ‌Use a resource interpreter approach‌:
    Allow users to ‌manually determine the resource type‌ and decide whether it is a ‌workload type‌ based on the returned result.

Here are several proposals that need to be discussed‌ @RainbowMango

@LivingCcj
Copy link
Contributor Author

Append the 4-th approach：Use the exist interpreter HookEnabled to decide whether it is a workload

karmada/pkg/detector/detector.go

Lines 765 to 773 in 2f6ff56

if d.ResourceInterpreter.HookEnabled(object.GroupVersionKind(), configv1alpha1.InterpreterOperationInterpretReplica) {
replicas, replicaRequirements, err := d.ResourceInterpreter.GetReplicas(object)
if err != nil {
klog.Errorf("Failed to customize replicas for %s(%s), %v", object.GroupVersionKind(), object.GetName(), err)
return nil, err
}
propagationBinding.Spec.Replicas = replicas
propagationBinding.Spec.ReplicaRequirements = replicaRequirements
}

And How to use a field IsWorkloadWithReplicas to indicate if the resource is a workload with replicas. Implementated in #6203 @RainbowMango cc @zach593

When the ‌karamda scheduler‌ schedules ‌rb objects‌, some scenarios require scheduling decisions based on the type of ‌object‌ that the rb object points to. Therefore, we need to add a field to the rb object to identify the ‌resource type‌ of the target object. Below are several approaches to determine the resource type of the target object when creating an rb resource:

  1. ‌Check replicaRequirements in GetReplicas‌:
    If the target object's replicaRequirements in GetReplicas is ‌non-null‌, it indicates a ‌workload type‌; otherwise, it is a ‌non-workload type‌.
  2. ‌Check for scale subresource‌:
    If the target object has a ‌scale subresource‌, it is considered a ‌workload type‌; otherwise, it is a ‌non-workload type‌.
  3. ‌Use a resource interpreter approach‌:
    Allow users to ‌manually determine the resource type‌ and decide whether it is a ‌workload type‌ based on the returned result.

Here are several proposals that need to be discussed‌ @RainbowMango

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@LivingCcj LivingCcj

Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
Karmada Overall Backlog
Status: No status
Milestone
v1.14
Development

Successfully merging a pull request may close this issue.

optimize to scheduler some workload with zero replicas LivingCcj/karmada
2 participants
@RainbowMango @LivingCcj