Set the Django migrate command to run once or throttle when updating multiple instances

[mtinberg]

When deploying/updating Netbox across two hosts that share a backend database everything works great in parallel except for running manage.py migrate, whichever host runs second fails sometimes because the first one is running DDL statements, which fails the rest of the playbook for that host requiring at least a second run to clean up or manual intervention to restart services.

I hotfixed this at my site by adding run_once: true to the migrate task so it only runs on the first host in the batch, which works great when limiting to just prod or qa instances but wouldn't work if someone was changing both at the same time. The serial keyword only works on the playbook level, and it's probably not worth it to make the whole deploy_netbox play serialized, if I wanted that I could just set --forks 1, but the throttle keyword should work for the individual task and may resolve this issue, but I haven't tested that yet.

It may also make sense to make the whole block throttle so that none of the manage.py steps which modify data have any chance to interfere with one another, while many probably can safely run in parallel, that may not be guaranteed.

What do you think?

[lae]

That block has gotten a bit bigger from the original release. It might make sense to move it to its own tasks file and include it, now.

Anyway, using throttle: 1 sounds like it should be fine. It's probably a very rare case that people have enough NetBox hosts where this would cause an issue.

On another note, I feel like it should be possible to smartly group all hosts by (database_host && database_port || database_socket) && database_name and only run a task once per each, but maybe that'd be complicated to implement?

[tyler-8]

run_once seems like the best option to me. It exists for exactly this scenario.

IMO - targeting disparate environments (prod and qa) in a single play run seems like a less than ideal workflow. I would run the plays separately, once for QA and once for Prod.

Using throttle or serial to achieve this behavior seems kind of like a hack around an uncommon workflow.