Large overhead for acquisition times using MerlinEM for cRED

[aurorateien]

Hello,

We are using Instamatic v2.0.0 for doing cRED with MerlinEM detector and have discovered an issue related to the acquisition time of each frame.
For example, for an exposure time of 0,5 s for each frame (0,01 s for defocused frames), the acquisition time is typically 0,8 s (high!), which seems a bit peculiar as the readout time for the MerlinEM 24 bit data is only 1,64 ms. The rest of the time included in the definition of the acquisition time is overhead, which typically is only a few ms. Why would these extra 300 ms be added to the acquisition time and can it be corrected for? After what I've understood, this is not an issue for other detectors.

I came across the note on using Instamatic with Merlin that said: "Changing exposure on the fly (for example, in the gui) incurs a small overhead (~300 ms). "
This overhead (of 300 ms) is in fact not small when doing cRED for 3D ED data acquisition, and is seemingly added without us doing any changes on the exposure time in the GUI. If the reason for the added time in fact is overhead, is there any way to avoid it?

[stefsmeets]

Hi @aurorateien , in my testing with the Merlin, the overhead was negligible (<1 ms as you suggested) for both softtrigger and movie mode. Can you tell me which mode you are using or how you are collecting data?

The overhead penalty when changing exposure happens because instamatic has to send a command to change the exposure to Merlin in between collecting images. This was about 300 ms in my testing. If the exposure does not change, this does not get triggered.

[stefsmeets]

I'm going to close this issue now, but feel free to re-open or open a new issue if you have any further questions!

[emichr]

Hi,

Sorry for not following up on this earlier (@aurorateien finished her masters with us last summer and other things got in the way).

I'm not certain, but I believe we are using the softtrigger mode. I must admit I don't know how to check this or how to change it.

In any case, our problem is still that, no matter what our exposure time is, the acquisition time is 0.091-0.095 s. See e.g. these logfiles:

cRED_log_25ms.txt
cRED_log_10ms.txt

Maybe @GearoidM has some experience with this as well?

Any help is greatly appreciated.

[stefsmeets]

No worries. The soft trigger is always going to have some form of overhead, because Instamatic has to retrieve the image from the Merlin software. I imagine Merlin is smart enough, but check if your Merlin instance is set to not save or post-process every frame (i.e. darkfield correction) before sending it. I put some of my notes here: https://instamatic.readthedocs.io/en/latest/merlin/

Another thought is that your exposure time is very short compared to my experiments (usually in the order of 0.1 - 0.5 s). Could you try with 0.3 s and see what happens?

[emichr]

Thanks for your quick reply! Looking at the instamatic logs, it looks like we were triggering it with the software trigger. How can we change to get a continuous movie instead (and can we still use defocus frames in movie mode)?

I'll try with 0.3 s exposure time as well and see what happens. The issue may become that the counts get saturated for some of the pixels, causing later processing steps to fail. How do you usually solve this? We can play a bit with the condenser aperture, but there is not really much we can do with the dose other than that.

[stefsmeets]

Thanks for your quick reply! Looking at the instamatic logs, it looks like we were triggering it with the software trigger. How can we change to get a continuous movie instead (and can we still use defocus frames in movie mode)?

You can't use defocus together with the continuous mode at this moment. Instamatic times the defocus with the exposure time, so it needs the software trigger.

I'll try with 0.3 s exposure time as well and see what happens. The issue may become that the counts get saturated for some of the pixels, causing later processing steps to fail. How do you usually solve this? We can play a bit with the condenser aperture, but there is not really much we can do with the dose other than that.

I usually worked with a very low dose.

[iverks]

Adding to this with some timing data from the same setup as @emichr:

From the timestamps on the images we can see that the median image has an acquisition time of 90-91 ms. Timestamp from here

For most datasets, the first two images take respectively ~1200 ms and ~1000 ms each. This is in combination with another bug i will report what causes the acquisition time in cRED_log.txt to be 95ms.

From the log-files we can see that a lot of the time is spent in receiving data. The lines below are representative for the whole log-file. Log data from here.

2024-11-18 09:28:29,868 | camera_merlin:115 | INFO | Received 1049359 bytes in 30 steps (0.037280 s)
2024-11-18 10:29:17,480 | camera_merlin:115 | INFO | Received 1049359 bytes in 25 steps (0.062942 s)
2024-11-18 11:28:59,406 | camera_merlin:115 | INFO | Received 525071 bytes in 21 steps (0.055101 s)

EDIT: in retrospect I see I misinterpreted the logfile, the lines that are around 0.05s are from the idle stream.
In reality we have 0.04s for 24bit mode with 0.025 exposure and 0.02s for 12bit mode with 0.01s exposure, which does not explain why we end up at 90ms.