Check the SMuRF `stream` state when other operations fail

[BrianJKoopman]

We have been seeing a problem when users accidentally leave the SMuRF streams on (whether through user error, or through not realizing an interrupt has left them on -- see #166). This causes all subsequent SMuRF commands, i.e. take_bgmap to fail, as the operations cannot run simultaneously, and so the command can't grab the ocs TimeoutLock, causing it to fail.

In sorunlib this causes us to stop commanding the controller. In practice, initialize() is called again within a few hours, but then another take_bgmap is run, which also fails for the same reason. The result is the stream is never turned off until an observant user notices that this is the reason the controller is being dropped and manually stops them.

The proposed fix for this is to check the SMuRF stream state after an operation fails. If stream is on, turn it off and then drop the slot like usual. If we can't turn it off (up to some timeout period), raise an exception. If stream is off, just drop the slot as normal. We'll still raise an exception if the threshold for number of running slots is crossed.

[BrianJKoopman]

There's a basic implementation of this in #231. One thing that became clear as I wrote this is what likely caused the issue described here is a network issue. That would explain the connection drop to the relay, and the dropping of the SMuRF slots that got left streaming, since if we can't reach the controllers while stopping them we drop the clients:

sorunlib/src/sorunlib/smurf.py

Lines 407 to 411 in 12158db

	# Handles case where agent becomes unreachable
	except ControlClientError as e:
	print(f"Failed to stop stream on {smurf}, removing from targets list.")
	print(e)
	clients_to_remove.append(smurf)

In #231, I just crash the schedule if I can't reach any single controller. This is counter to the rest of the module that drops the client and stops trying to command it if it can't be reached (up the the "function SMuRF threshold"). Is this acceptable?

My other thought is we might need a more complex implementation that keeps trying to reach the client in order to call a 'stop', in the event it comes back online, while dropping it from other operations -- and allowing those other operations to continue.