Allow user specify exact resources requests and limits

[baznikin]

• Which image of the operator are you using? e.g. ghcr.io/zalando/postgres-operator:v1.14.0
• Where do you run it - cloud or metal? Kubernetes or OpenShift? DO K8s
• Are you running Postgres Operator in production? yes
• Type of issue? feature request

Proposed behavior:

Operator always sets resources.limits and resources.requests.
We specifically do not want to set CPU limits to avoid CPUThrottling but can't. Only solution is to set resources.cpu.limits to value much higher than request. It works but makes default graphical representation in most monitoring tools unpleasant.

Proposed behavior:

Vanilla StatefulSet behavior is different - it just do not set limits if they are not set. I propose to do same - do not make any assumptions and allow user to decide which resources to requests and which limits to set! User knows better.

Examples:

1. Part of sample STS manifest (I hope it is not required to give full manifests?):

        resources:
          requests:
            cpu: 199m
            memory: 555Mi

Resources in cluster:

    Requests:
      cpu:        199m
      memory:     555Mi

2. Part of sample postgresql manifest:

  resources:
    requests:
      memory: 555Mi

Resources in cluster:

    Limits:
      cpu:     1
      memory:  555Mi
    Requests:
      cpu:      100m
      memory:   555Mi

3. Part of sample postgresql manifest:

  resources:
    requests:
      cpu: "2"
      memory: 555Mi

Resources in cluster:

    Limits:
      cpu:     2
      memory:  555Mi
    Requests:
      cpu:      2
      memory:   555Mi

4. Part of sample postgresql manifest:

  resources: {}

Resources in cluster:

    Limits:
      cpu:     1
      memory:  500Mi
    Requests:
      cpu:      100m
      memory:   100Mi

[bumarcell]

We're also having this problem after having upgraded to v1.14.0.
With v1.12.2 setting default_cpu_limit="0" used to work (no default cpu limit was set)