Modify get_movie to display frames as they are collected (#121)

* Add `ClickDispatcher` architecture and setup one in `VideoStreamFrame`

* `ClickListener`: make `callback` optional and allow activating via `with` context

* Add `typing_extensions` as a dependency

* Move potential `typing_extensions` types to `instamatic._typing`

* Feature usefulness of new `instamatic._typing.Self`

* Exclude changes irrelevant to the `click_service` branch

* Add a custom `ClickEvent.__repr__`

* Move `instamatic.collections` to `instamatic._collections`

* `ClickListener`: call `callback` only after putting event in `self.queue`

* `_VS`: rename type variable to a more conventional `VideoStream_T`

* `typing_extensions`: uses typing if available so import directly, not via `instamatic._typing`

* `CameraSimu.get_movie`: convert into `Generator`

* `VideoStream.get_movie`: convert into `Generator`

* `VideoStream.get_movie`: Optimize thread release, remove unused event = best fps yet

* `VideoStream.get_movie`: Optimize thread release, remove unused event = best fps yet 2

* `TEMController.get_movie`: convert into `Generator`

* `ctrl.get_image`: fix `header_keys` could hold only 1 key as `str` not `tuple`

* `CalibMovieRate`: first draft of the movie delay calibration

* `TEMController.get_movie`: collect common metadata before the main loop for time precision

* `TEMController`: move ugly explicit `self.cam` check to a method decorator

* `CalibMovieDelays`: simplify and add multi-attempt mechanism

* `CalibMovieDelaysMapping`: add a common class for reading/writing mappings

* `CalibMovieDelaysMapping`: fix I/O, script, bugs, add custom `CalibError`

* `CameraServal.get_movie`: rewrite as generator, streamline by PIL->tifffile

* `CameraBase.get_movie`: rewrite as generator

* `CameraMerlin.get_movie`: rewrite as generator

* `showcase_movie.py`: Add a temp script to feature get_movie streaming feature

* `CameraServal.get_movie`: Actually replace previous implementation this time

* `CalibMovieDelays`: vastly simplify, do for specific exposure only

* `CalibMovieDelays`: fix documentation, warning criteria

* Register `instamatic.calibrate.calibrate_movie_delays` as script

* Fix bugs, typos

* Adapt `test_get_movie` to the new structure

* Add Daniel Tchoń to `CITATION.cff`

* Wrap generators in `try/finally` to correctly close them when not exhausted

* `calibrate_movie_delays.py`: Remove code block comments as requested

Co-authored-by: Stef Smeets <stefsmeets@users.noreply.github.com>

* `calibrate_movie_delays.py`: Remove code block comments as requestedUpdate src/instamatic/calibrate/calibrate_movie_delays.py

Co-authored-by: Stef Smeets <stefsmeets@users.noreply.github.com>

* `calibrate_movie_delays.py`: Remove code block comments as requested

Co-authored-by: Stef Smeets <stefsmeets@users.noreply.github.com>

* `calibrate_movie_delays.py`: Remove code block comments as requested

Co-authored-by: Stef Smeets <stefsmeets@users.noreply.github.com>

* `calibrate_movie_delays.py` reformat `parser.add_argument`

Co-authored-by: Stef Smeets <stefsmeets@users.noreply.github.com>

* `main_entry`: reformat `parser.add_argument` calls

* `CalibMovieDelaysMapping`: improve readability, docstrings as requested

* `CalibMovieDelaysMapping`: save calibrations to yaml instead of json

* `calibrate_movie_delays_live`: warn instead of raising `CalibWarning`

* `calibrate_movie_delays_live`: clear out dead code, improve docstrings, fix issues

* `calibrate_movie_delays_live`: allow passing empty `header_keys(_common)`

* `calibrate_movie_delays_live`: vastly improve/expand docstring

* `CalibMovieDelays`: add docstrings, raise if conditions not found

* `CalibMovieDelaysMapping` de/serialize self from/to a nice yaml list

---------

Co-authored-by: Stef Smeets <stefsmeets@users.noreply.github.com>

Author: Baharis

CITATION.cff

@@ -21,6 +21,11 @@ authors:
     family-names: Rainer
     given-names: Daniel
     orcid: "https://orcid.org/0000-0002-3272-3161"
+  -
+    affiliation: "Czech Academy of Sciences"
+    family-names: Tchoń
+    given-names: Daniel Mariusz
+    orcid: "https://orcid.org/0000-0003-4798-867X"
 cff-version: "1.1.0"
 date-released: 2021-08-09
 doi: "10.5281/zenodo.1090388"

docs/programs.md

@@ -25,6 +25,7 @@ These tools help calibrate instamatic for some experiments.
 - [instamatic.calibrate_beamshift](#instamaticcalibrate_beamshift) (`instamatic.calibrate.calibrate_beamshift:main_entry`)
 - [instamatic.calibrate_directbeam](#instamaticcalibrate_directbeam) (`instamatic.calibrate.calibrate_directbeam:main_entry`)
 - [instamatic.calibrate_stagematrix](#instamaticcalibrate_stagematrix) (`instamatic.calibrate.calibrate_stagematrix:main_entry`)
+- [instamatic.calibrate_movie_delays](#instamaticcalibrate_movie_delays) (`instamatic.calibrate.calibrate_movie_delays:main_entry`)
 - [instamatic.flatfield](#instamaticflatfield) (`instamatic.processing.flatfield:main_entry`)
 - [instamatic.stretch_correction](#instamaticstretch_correction) (`instamatic.processing.stretch_correction:main_entry`)
 
@@ -214,6 +215,35 @@ instamatic.calibrate_beamshift [-h] [IMG [IMG ...]]
 : Show this help message and exit  
 
 
+## instamatic.calibrate_movie_delays
+
+Program to calibrate the delays associated with `ctrl.get_movie` command.
+
+Intuitively, collecting an N-frame movie with X-second exposure should take N*X seconds. However, the hardware specification and software implementation for each detector differ, leading to deviations. This calibration aims to take this effect into account and allow scheduling movies whose frame time better reflects the request.
+
+**Usage:**  
+```bash
+instamatic.calibrate_movie_delays [-h] [-e E] [-a A] [-c C] [-o O]
+```
+
+**Optional arguments:**  
+
+`-h`, `--help`
+: Show this help message and exit  
+
+`-e`, `--exposure`
+: Exposure to test the delays for in seconds (default 1.0)
+
+`-a`, `--variable_headers`
+: Comma-delimited list of variable header keys to calibrate for (for default, see `src/instamatic/controller.py:MOVIE_HEADER_KEYS_VARIABLE`)
+
+`-c`, `--common_headers`
+: Comma-delimited list of common header keys to calibrate for (for default, see `src/instamatic/controller.py:MOVIE_HEADER_KEYS_COMMON`)
+
+`-o`, `--output`
+: Path to the directory where calibration file should be output (default `%appdata%/calib` on Windows or `$AppData/calib` on Unix)
+
+
 ## instamatic.calibrate_directbeam
 
 Program to calibrate the diffraction shift (PLA) to correct for beamshift movements (Deprecated).

pyproject.toml

@@ -102,6 +102,7 @@ publishing = [
 "instamatic.calibrate_beamshift" = "instamatic.calibrate.calibrate_beamshift:main_entry"
 "instamatic.calibrate_directbeam" = "instamatic.calibrate.calibrate_directbeam:main_entry"
 "instamatic.calibrate_stagematrix" = "instamatic.calibrate.calibrate_stagematrix:main_entry"
+"instamatic.calibrate_movie_delays" = "instamatic.calibrate.calibrate_movie_delays:main_entry"
 "instamatic.flatfield" = "instamatic.processing.flatfield:main_entry"
 "instamatic.stretch_correction" = "instamatic.processing.stretch_correction:main_entry"
 # tools

src/instamatic/calibrate/__init__.py

@@ -3,5 +3,7 @@
 from .calibrate_beamshift import CalibBeamShift
 from .calibrate_brightness import CalibBrightness
 from .calibrate_directbeam import CalibDirectBeam
+from .calibrate_movie_delays import CalibMovieDelays
 from .calibrate_stage_lowmag import CalibStage
+
 # from .calibrate_stage_mag1 import CalibStageMag1

src/instamatic/calibrate/calibrate_beamshift.py

@@ -189,7 +189,7 @@ def calibrate_beamshift_live(
             binsize=binsize,
             out=outfile,
             comment=comment,
-            header_keys='BeamShift',
+            header_keys=('BeamShift',),
         )
         img = imgscale(img, scale)
 

src/instamatic/calibrate/calibrate_brightness.py

@@ -110,7 +110,7 @@ def calibrate_brightness_live(ctrl, step=1000, save_images=False, **kwargs):
 
         comment = f'Calib image {i}: brightness={target}'
         img, h = ctrl.get_image(
-            exposure=exposure, out=outfile, comment=comment, header_keys='Brightness'
+            exposure=exposure, out=outfile, comment=comment, header_keys=('Brightness',)
         )
 
         img, scale = autoscale(img)

src/instamatic/calibrate/calibrate_directbeam.py

@@ -250,7 +250,7 @@ def calibrate_directbeam_live(
 
         comment = f'Calib image {i}: dx={dx} - dy={dy}'
         img, h = ctrl.get_image(
-            exposure=exposure, binsize=binsize, out=outfile, comment=comment, header_keys=key
+            exposure=exposure, binsize=binsize, out=outfile, comment=comment, header_keys=(key,)
         )
         img = imgscale(img, scale)