Drop sidekiq

[coorasse]

This PR is a starting point to discuss where we want to go.

I'd really like to finally drop redis from our dependencies and dropping sidekiq is the first step in that direction.
We had already pretty good experiences with good_jobs in renuo projects and I used successfully solid_queue in personal ones.

Given that, the next question is: separate database or single database?

I recognize the points from dhh on rails/solid_queue#348 (comment) and I think they are pretty valid. I usually just use separate DBs on sqlite.

But with postgres + heroku/deplo.io the situation is different because hosting costs go up very quickly and we have to also take care of multiple backups.

To me the process should be the following:

• are you going with postgres or sqlite?
• if you use postgres, start with a single database (and I'll update this PR with the necessary changes)
• if you use sqlite, use separate databases

Note that also other gems: solid_cache, solid_cable, solid_errors, rails_api_logger, etc... might just go on a separate DB.

In case you go with a single database, we should anyway configure a connection pool for each gem, so that they share the same database, but work on separate transactions.

[schmijos]

On your page. But is good_job described somewhere here in this guide?

Regarding having solid queue in the same Postgres Database: what can go wrong in our setups? Most of our apps are not that job-heavy. So I'd go for a default with solid_queue in the same Postgres database.

Regarding ActionCable, I'm not so sure. Is this still LISTEN/NOTIFY? I wouldn't want Websockets connections to dictate my database performance. Would there be a good prioritization setting available for Postgres? Or can the connection pool on Rails side somehow steer who gets how much of the database?

[coorasse]

Indeed we never described good_job. We experimented it in a couple of apps, but I think the default solid_queue, combined with mission control as a gui, works also pretty well.
One key difference I noticed is this one.
I like good_job, but I don't see a good reason, right now, to deviate from the default. I'd need a good reason and I cannot think of any.

Regarding solid_cable: it does not use LISTEN/NOTIFY but it polls a database table on a configurable interval.
Apps which use turbo stream and turbo page refreshes heavily could be affected negatively by the usage of a single database.
I think this is a scaling issue. One that might appear soon, but not something we should bother thinking on day zero.

TL;DR; Given our history, I believe the Rails default should be good to go, but we could still spin up a separate db or a redis instance when needed.

[simon-isler]

Thanks for the detailed PR description @coorasse 🙌🏼

TL;DR; Given our history, I believe the Rails default should be good to go, but we could still spin up a separate db or a redis instance when needed.

"when needed" + when the client approves it in our case. Spinning up a separate DB would mean switching from a shared DB to a dedicated DB, which is quite a price difference as you mentioned.

To me the process should be the following:
are you going with postgres or sqlite?
if you use postgres, start with a single database (and I'll update this PR with the necessary changes)
if you use sqlite, use separate databases

Fully agree 👍🏼

[rnestler]

Spinning up a separate DB would mean switching from a shared DB to a dedicated DB, which is quite a price difference as you mentioned.

You can just add an additional shared DB, no? And with a dedicated DB-Server on deploio you can add as many DB's as you want.

[rnestler]

In favor of dropping sidekiq and recommend solid queue per default configured with a single DB, as we did for some customers project (which I probably shouldn't name in a public repo)