[Question][DB] What is necessary to make Postgres "officially supported"?

[mstrangfeld]

Question

Hey, we are currently starting to use DevLake at my organization and I love it!
However, we usually don't use MySQL and prefer Postgres as we have the tools and knowledge to maintain these databases.

So, my question is:
What is necessary to make Postgres "officially supported" by DevLake?

I ran DevLake local with Postgres and everything seems to work just fine with exception for the predefined Grafana dashboards, of course.

[klesh]

We appreciate you bringing this to our attention and understand the potential value. However, given our current roadmap and resource constraints, investing the effort required for this change is not feasible for us at this point in time. We don't have plans to implement this in the near future.

[mstrangfeld]

Thank you for the update. I completely understand the resource constraints. Since you're not currently planning to work on this, I'd like to offer to contribute myself (with backing from my employer).

I've already validated that DevLake runs with Postgres with only dashboard compatibility issues. I'm willing to put in the work to make Postgres an officially supported option. To make this worthwhile for both sides, could you clarify what would be needed for you to consider merging such a contribution?
Specifically, I'd appreciate guidance on:

• What test coverage would be required (integration tests, smoke tests, etc.)
• Documentation updates needed
• Any design considerations or architectural constraints I should be aware of

We've already standardized on Postgres in our organization, and I believe many other users might benefit from this option as well.

I'm happy to collaborate on making this happen without burdening your core team resources.

[klesh]

Great to hear you're offering to contribute Postgres support, that's very welcome!

You're right, the main hurdle is indeed the Grafana dashboards being MySQL-centric. What do you think we should do to make it "Officially supported" ?

In terms of your questions for merging a contribution:

Testing: We don't have automated dashboard tests currently. For your contribution, demonstrating functional Postgres dashboards will be key. We can discuss validation methods.
Documentation: Yes, installation (Helm/Docker Compose) and general docs will need updates for Postgres.
Design: Ensure installation updates are user-friendly for Postgres selection and configuration.

[Bermos]

Hey @mstrangfeld, we are also working with a Postgres DB and would like to make this officially supported. From our experience the ingestion part works well, the biggest problem are the Grafana dashboards.
I've done some research and stumbled upon this project from Grafana themselves: https://grafana.github.io/grafana-foundation-sdk/next+cog-v0.0.x/go/How-To/custom-query-type/ . It basically allows for defining dashboards with their panels and queries to be defined in code (Go is supported) and then "rendered" as JSON. This would allow us to in a first step to render out both MySQL and Postgres Queries in their respective dashboards. Other things that would make it more practical to use in my opinion:

• Use something like https://github.com/Masterminds/squirrel or https://github.com/doug-martin/goqu/blob/master/docs/dialect.md to build for both dialects
• Validate queries by running them against a DB with known data and match results against expected results
• Instead of reloading the dashboard JSONs form disk in Grafana, use the Grafana API to let DevLake insert/update the dashboards

Another, smaller issue is the helm chart which needed quiet some fiddling to get working properly with a Postgres DB but I'd say that has a lower priority. Feel free to contact me if you'd like to discuss any of those inputs.

[klesh]

One thing came to my mind: I heard that Postgres performance would degrade after collecting too much data many times due to the Vaccum mechansim fail to keep up with the massive Delete+Insert operations.