[tempo-distributed] More autoscaling configurations

[jordan-simonovski]

Added some more autoscaling configurations to Tempo components:

• Metrics Generator
  • Added a HPA to the metrics generator
  • Added a KEDA scaled object to support custom metric scaling
• Ingester
  • Added a KEDA scaled object to support custom metric scaling
• Distributor
  • Added a KEDA scaled object to support custom metric scaling
• Compactor
  • Updated the query example

Happy to discuss further. We've been running the metrics generator HPA in production without any issues.

The context behind adding KEDA support to the ingester and distributor is to allow for more flexible scaling on external metrics such as pipeline queue backlogs, spikes in ingestion volume, etc.

[jon-rei]

I'm currently optimising our Tempo setup and have just added KEDA to the Compactor. We are also searching for a solution to automatically scale the metrics generator. Therefore, the PR would be a great help.

I do have one remark, though: do you think it would make sense to allow the advanced configuration to be configured in the ScaledObject? This would also allow HPA behaviours to be set.

[jordan-simonovski]

I'm currently optimising our Tempo setup and have just added KEDA to the Compactor. We are also searching for a solution to automatically scale the metrics generator. Therefore, the PR would be a great help.

I do have one remark, though: do you think it would make sense to allow the advanced configuration to be configured in the ScaledObject? This would also allow HPA behaviours to be set.

I'm definitely open to this if the maintainers are happy with it as an additional config.