bug: idsse-1276: Publisher re-connect if connection closed

[mackenzie-grimes-noaa]

Linear Issue

IDSSE-1276

Changes

• In RabbitMQ utils Publisher.blocking_publish(), confirm that RabbitMQ connection/channel is open before trying to publish
  • If it isn't open, re-attempt connection and then publish message
• Default all RabbitMQ Conn to configure a 3600 second heartbeat from the RabbitMQ broker
  • RabbitMQ library default is 60 seconds
  • This might help prevent long-running connections from randomly getting closed by the broker. Might introduce new weird behavior across services though

Explanation

We saw a bug in the dev environment where an instance of the NWSC Gateway had been idle for a while (days). When the Criteria Builder sent it a Support Profile to publish to RabbitMQ, it failed with an error message that the connection was in closed state.

Restarting the Kubernetes pod and re-submitting the Support Profile resolved the issue, but this required the dev team's manual intervention, which is bad.