Provide recommendations for historical deployments

[davidcurrie]

Is your feature request related to a problem? Please describe.
I'd like to be able to use krr to make recommendations in a cluster used for executing test runs but, at least as far as I can tell, it will only make recommendations for currently running resources.

Describe the solution you'd like
Provide a mode in which krr relies only on data from Prometheus so it can be used to provide recommendations for any resource that existed at some point within the history duration.

Describe alternatives you've considered
As things stand, we'd have to modify every CI pipeline to run krr before cleaning up the resources.

Are you interested in contributing a PR for this?
No - sorry - I'm not a Pythonista.

Additional context
None.

[aantn]

Thanks for opening. It's an interesting use case.

We probably wont be able to prioritize this soon, but we're open to PRs as always. (I know you're not a Pythonista :()

[aantn]

Hey, just to update we're working on a mode where KRR can operate only based on Prometheus data. So this will be supported in the near future. (Feel free to help by testing the Prometheus-only mode!)

#266

[davidcurrie]

@aantn, thanks for the heads-up. We tried it, but it still doesn't seem to help for the scenario described in this issue. Does this PR only query Prometheus for currently active workloads? If so, that still doesn't help us, as the test workloads for which we're looking for recommendations are no longer running. The hope was that it would cover any workload that was active at some point during the history duration.

[aantn]

@davidcurrie no, it should fetch historical workloads too. What is the type of workload? If I provide you with a Prometheus query, can you check it manually and see if there is data for that query for the historical workloads?

[JasonSinclair95]

Hi @aantn, i work with @davidcurrie and was the engineer behind testing this feature.

After installing a pod within a namespace containing a workload, I executed the 'krr' command, which detected the new pod. However, upon removing the pod from the namespace, subsequent executions of the command no longer displayed the pod. which tells me this feature operates exclusively in real-time, without having the ability to query historical data and just leverages Prometheus to retrieve its information. Am i wronge and if so can you guide me please

the commands i'm using:

python krr.py simple --mode prometheus --prometheus-url http://localhost:9090/ --namespace test-namespace

[aantn]

Hi @JasonSinclair95,
It is supposed to load the historical data, so that sounds like a bug on our side. What is the workload's type? I will try to reproduce this on my end.