`pysmurf` performance improvements

[tristpinsm]

A significant upgrade to pymsurf that will remove EPICS as a communication layer in SMuRF has been in the works for a while. It may alleviate some of the lock-up issues that have been a regular occurrence on site (although I don't think the cause of those has been decisively identified yet), but in addition to that it should noticeably improve performance of register operations. This will probably result in some level of speed-up for most operations as-is, but we think we can refactor some of the code to better take advantage of the new capabilities. In particular, tuning resonators had been identified as a bottleneck that could be made a lot faster by introducing parallelisation that is not possible with the current EPICS communication.

However, in a recent call it came up that actually tuning is not thought of as a bottleneck for site operations, but possibly relock or taking IVs is. I don't think this should make us reconsider improving tuning performance, but we could focus also on improving the performance of these other operations if that is useful. Note that this doesn't have to be conditional on the pysmurf overhaul mentioned above, as it may or may not be possible to speed up these operations in the current code.

I don't have much insight into what operations are bottlenecks, how frequently they need to be run, constraints from the scheduler -- in general any requirements for their performance. So specific requests would be helpful.

I tried looking at nextline scripts and pysmurf logs to get an idea of what is being run, and although I found a few variations, the detector calibration that is run periodically looks something like this:

2025-03-24T21:14:03+0000 stream:378 Finished streaming data
2025-03-24T21:14:21+0000 start called for take_bias_steps
2025-03-24T21:15:25+0000 take_bias_steps:379 Status is now "done".
2025-03-24T21:16:30+0000 start called for take_bgmap
2025-03-24T21:17:30+0000 take_bgmap:380 Status is now "done".
2025-03-24T21:17:31+0000 start called for take_noise
2025-03-24T21:18:31+0000 take_noise:381 Status is now "done".
2025-03-24T21:18:31+0000 start called for take_iv
2025-03-24T21:26:24+0000 take_iv:382 Status is now "done".
2025-03-24T21:26:25+0000 start called for bias_dets
2025-03-24T21:26:29+0000 bias_dets:383 Status is now "done".
2025-03-24T21:29:31+0000 start called for take_bias_steps
2025-03-24T21:30:35+0000 take_bias_steps:384 Status is now "done".
2025-03-24T21:30:35+0000 start called for take_noise
2025-03-24T21:31:37+0000 take_noise:385 Status is now "done".
2025-03-24T21:36:03+0000 start called for stream

which takes 15min to run, and half of that time being take_iv, which spends 5min in "cool wait".

The other step I looked at is uxm_relock, which appears to be run daily

2025-03-23T22:31:37+0000 start called for zero_biases
2025-03-23T22:31:37+0000 zero_biases:200 Status is now "done".
2025-03-23T22:33:38+0000 start called for take_noise
2025-03-23T22:34:41+0000 take_noise:202 Finished taking noise
2025-03-23T22:34:46+0000 start called for uxm_relock
2025-03-23T22:40:19+0000 uxm_relock:203 Finished UXM Relock

This takes only about 5min.

I didn't find an instance of tuning (uxm_setup) in the logs, but I know this takes very long, like 45min or so.

What I'd like to know is:

• Tuning, relock, calibration as I've defined them above -- do these span all SMuRF operations on site that we could consider optimising?
• Which of these are currently most constraining for scheduling?
• At what cadence do we need to run these? (if this is known)
• Would we do it more frequently if they were faster? Which steps specifically?

People who may be interested in this topic: @msilvafe @kmharrington @tpsatt @swh76

[kmharrington]

The full discussion of this issue is probably a meeting or two honestly.

But the largest issues with our performance optimization are outside of pysmurf. You are correct that we rarely run uxm_setup. We run uxm_relock once every day to few days and it's short enough to not be significant overhead. The detector setup block every few hours and before every calibration scan. The best summary of what we are doing is here: https://docs.google.com/document/d/1fD5V8bKn7ZEg528BrNpwhoGRK8jqDSfwwtiRG3AUTEw/edit?tab=t.0#heading=h.gjdgxs

But that write-up does not cover why we are doing what we are doing or what we could be doing instead. All of my highest priorities for optimization aren't part of SMuRF or pysmurf (not counting the "fix the communication drop outs and the 20Gb streams of zeros").

My absolute first priority: Where we're very sub-optimal is our detector operations that run on the interface between sodetlib and the UFMs. IVs heat our focal planes, so we know we're biasing sub-optimally and slowly. But the level at which this is true is highly UFM and telescope dependent. The options for rebiasing are limited in sodetlib: we only have biasing dets off IVs and the "rebias with bias steps." The rebias with bias steps can sometimes require overbiasing (re:heating), takes a variable amount of time, and is not setup to run operationally onsite. (both on the pysmurf controller side and the current outputs break data packaging). Significantly improving the computer science / software infrastructure of that method is needed before we can even test it on-sky and see if it works better.

[tristpinsm]

OK, so it doesn't sound like there are any specific things to focus on for performance in the pysmurf upgrade beyond what is being considered already.

Reworking the bias operations in sodetlib is indeed a separate thing to consider if that is a priority.

[msilvafe]

I agree with your conclusion @tristpinsm, we do not tune very frequently (like once a cooldown which could last months-year once stable) except when initially setting up new wafer. Instead we end up relocking daily. That's fairly stable but if we could run setup_notches and it was nearly as fast as running serial_gradient_descent and serial_eta_scan then I would probably opt for that because we would probably get more consistent tracking results and regular checks on the core resonator properties/frequency response which would be helpful to monitor I think. But otherwise I agree that probably most of the updates we could use are in the detector biasing operations on the sodetlib side of things.

[kmharrington]

running setup_notches leads to new channel assignments correct? If so, we want to be very careful about doing that without some level of analysis of how significantly the detector matches change between channel assignments. If we change a channel assignment we change the det match and, when we're in a place where we need both pointing and polarization measurements to create "high quality" det matches, changing them often will make our metadata lives much harder.

[msilvafe]

running setup_notches leads to new channel assignments correct? If so, we want to be very careful about doing that without some level of analysis of how significantly the detector matches change between channel assignments. If we change a channel assignment we change the det match and, when we're in a place where we need both pointing and polarization measurements to create "high quality" det matches, changing them often will make our metadata lives much harder.

No, channel assignments don't change with setup_notches that requires a new tune which requires find_freq