feat: make startup probe configurable for the sidecar

[tsg]

This makes the startup probe configurable for the sidecar container.

Motivation: Because the sidecar container is an init container, the time it takes for it to be marked ready is added to the total startup time for a cluster, or for waking up an existing cluster from hibernation. The default period was 10 seconds, but the actual starting time is much lower than that.

This PR only makes the startup probe configurable, but doesn't change the defaults, meaning that there shouldn't be a change in the default behavior.

[armru]

Considering two things:

• would it be better to have it inside the Server API, more specifically inside the .spec.instanceSidecarConfiguration?
• should we consider a more generic approach to customize the sidecar to avoid to incur in those issues in future?

Any opinions on this?

Also would love other mantainers feedback

[tsg]

would it be better to have it inside the Server API, more specifically inside the .spec.instanceSidecarConfiguration?

Ah, interesting, I didn't notice that there is some sidecar resources configuration already possible, that is good to know.

I see now that it is on the ObjectStore CRD, rather than the cluster one. I'm not sure about the reason they are there, or all the implications. But it makes sense to me to have all the sidecar config in one place, rather than split between the ObjectStore and the cluster specs. I'm happy to update the code to do this.

should we consider a more generic approach to customize the sidecar to avoid to incur in those issues in future?

You are thinking of something that would take everything from under .spec.instanceSidecarConfiguration and put it in the sidecar spec?

One thing that I can do for sure in this PR is to configure all probes, not only the startup one. I initially wanted to have the minimal code change, but it does feel weird to have one probe configurable but not the others. I'm happy to add them either in this PR on in follow ups. Also, happy to do the generic in-bulk spec copy if you think it's the best approach.

[tsg]

hi @armru, I've pushed two commits to address those two ideas:

• 8c2e72a - Moving the startup probe configuration to the object store under spec.instanceSidecarConfiguration.
• ce7d8e9 - Add support for making the liveness and readiness probes configurable as well.

I think these changes make sense, but happy to revert one or both of them if you think the PR has gotten too large, or want a different approach.

I did testing manually (beyond the unit tests), and covered the following:

• various startup / liveness / readiness configs.
• using multiple replicas with switchover.

Let me know if you have any particular scenarios you'd like me to test.

[armru]

Hello, I think we have to wait for the other maintainers opinion

[tsg]

@armru I have merged main in this branch, and I'd like to join the Community call today to discuss if/how we could progress this PR. Just don't want it to be lost.