Skip to content

Commit 63155a2

Browse files
Baharisstefsmeets
andauthored
Implement FastADT experiment (#137)
* Add `ClickDispatcher` architecture and setup one in `VideoStreamFrame` * Add `VideoStreamService` architecture and setup one in `VideoStreamFrame` * Rename `VideoStreamEnhancers` to `Overlays`, make `ImageDrawOverlay` editable * `ClickListener`: make `callback` optional and allow activating via `with` context * `MediaGrabber`: fix bug: `get_media` cleared `continuousCollectionEvent` * `calibrate_beamshift.py`: make imports absolute, allow float beamshift * Instamatic GUI: temporarily print coords of clicked point, revert later * Move `instamatic.typing` to `typing_` to avoid import conflicts * `stage`: Add `get_rotation_speed`, improve typing * Add `instamatic.calibrate_stage_rotation` for greater rotation control * Add `instamatic.calibrate_stage_rotation` for greater rotation control 2 * Add first ideas for RATS-like generalized data collection frame * `ImageDrawOverlay`: Don't print `self.operations` on every redraw * GUI: FIX couldn't initialize with non-steamable camera * `VideoStream`: Generalize real & fake streams under common base class * GUI: Allow VideoStreamFrame for non-streamable cameras via the fake * Allow customizing `NoOverwriteDict`'s `KeyOverwriteError` message * Add `ClickDispatcher` architecture and setup one in `VideoStreamFrame` * `ClickListener`: make `callback` optional and allow activating via `with` context * Add `typing_extensions` as a dependency * Move potential `typing_extensions` types to `instamatic._typing` * Feature usefulness of new `instamatic._typing.Self` * Move potential `typing_extensions` types to `instamatic._typing` * Feature usefulness of new `instamatic._typing.Self` * Exclude changes irrelevant to the `click_service` branch * Add a custom `ClickEvent.__repr__` * Add a custom `ClickEvent.__repr__` * Move `instamatic.collections` to `instamatic._collections` * Move `instamatic.collections` to `instamatic._collections` * `ClickListener`: call `callback` only after putting event in `self.queue` * `ClickListener`: call `callback` only after putting event in `self.queue` * `_VS`: rename type variable to a more conventional `VideoStream_T` * `_VS`: rename type variable to a more conventional `VideoStream_T` * `typing_extensions`: uses typing if available so import directly, not via `instamatic._typing` * `typing_extensions`: uses typing if available so import directly, not via `instamatic._typing` * Do not pass numpy objects to the microscope. * `CameraSimu.get_movie`: convert into `Generator` * `VideoStream.get_movie`: convert into `Generator` * `VideoStream.get_movie`: Optimize thread release, remove unused event = best fps yet * `VideoStream.get_movie`: Optimize thread release, remove unused event = best fps yet 2 * `TEMController.get_movie`: convert into `Generator` * `ctrl.get_image`: fix `header_keys` could hold only 1 key as `str` not `tuple` * `CalibMovieRate`: first draft of the movie delay calibration * `TEMController.get_movie`: collect common metadata before the main loop for time precision * `TEMController`: move ugly explicit `self.cam` check to a method decorator * `CalibMovieDelays`: simplify and add multi-attempt mechanism * `CalibMovieDelaysMapping`: add a common class for reading/writing mappings * `CalibMovieDelaysMapping`: fix I/O, script, bugs, add custom `CalibError` * FEI Tecnai upper speed limit seem to be around 0.2 * `CameraServal.get_movie`: rewrite as generator, streamline by PIL->tifffile * `CameraBase.get_movie`: rewrite as generator * `CameraMerlin.get_movie`: rewrite as generator * `showcase_movie.py`: Add a temp script to feature get_movie streaming feature * `CameraServal.get_movie`: Actually replace previous implementation this time * `CalibMovieDelays`: vastly simplify, do for specific exposure only * `CalibMovieDelays`: fix documentation, warning criteria * Register `instamatic.calibrate.calibrate_movie_delays` as script * Fix bugs, typos * Adapt `test_get_movie` to the new structure * Add Daniel Tchoń to `CITATION.cff` * Implement simple cRED protocol in rats frame using `get_movie` * Fix `write_tiff` raising when writing to non-existing directory * Fix `DiffractionRun.middles` off-by-one error * Wrap generators in `try/finally` to correctly close them when not exhausted * RATS cRED: find and rotate with closest possible rotation speed * Reorganize rats frame, lock unusable buttons * `RatsFrame`: clean code formatting * `RatsFrame`: make `Run.experiments` a property * `RatsFrame`: can now extend/collect experiments in parts * These things don't work on Tecnai (fix better) * WHen using center, convert numpy types to python * Disable plotting until its fixed. * Adapt RATS experiment to updated movie calibration * Add typing, align `stage.rotating_speed` to `stage.rotation_speed` to match already-used wording * Fix beamshift deflector so that it passes `int` or `float` instead of `np.int` * Fix wrong definitions of goniometer speed vs pace * Fix mismatched axes order for the purpose of crystal tracking * Generalize image, tracking, diffraction config re/store * Correctly change config during RATS experiment * Add docstrings, streamline blanking * Add `with Beam().blanked()` and `with Beam().unblanked()` * Apply the new beam `un/blanked` context in various places * Various RATS frame fixes needed to run cRED on Tecnai * New console print of carriage return, with debug statements * Remove ANSI ESCAPE characters after all; this version handles tqdm * Remove debug statements * Improve RATS experiment: go to 0 at start, end, also collect image * Add beam blank button, block can when collecting stills, * Systematize `setStagePosition` signature, make `None` a sentinel for speed * Fix TEMController alignments being incorrectly stored and restored * Add generalized handling for PETS input using a dedicated factory * Since we need to support python 3.7 and 3.8, move csv file inside py * Produce just a single PETS file in RATS experiment * Add generalized handling for PETS input using a dedicated factory * Since we need to support python 3.7 and 3.8, move csv file inside py * Fix name of produced pets file to `pets.pts` * Fix name of produced pets file to `pets.pts` * Fix creating `pets.pts` if prefix or suffix are not defined * Fix creating `pets.pts` if prefix or suffix are not defined * Split PETS affix to blocks pre-adding: rejects only duplicate lines, not all affix * Split PETS affix to blocks pre-adding: rejects only duplicate lines, not all affix * Add doc, tests, move csv to resources, bump min Python version to 3.9 * GitHub tests fail for Python 3.13, may be not supported by some libraries yet * Simplify, accelerate the video provider, now `VideoServiceProcessor` * Further simplify, make faster, allow `VideoStreamProcessor` to display figures * Remove `[tool.setuptools.package-data]` from `pyproject.toml` Co-authored-by: Stef Smeets <stefsmeets@users.noreply.github.com> * Fix typo in src/instamatic/processing/ImgConversion.py Co-authored-by: Stef Smeets <stefsmeets@users.noreply.github.com> * Move `PetsInputWarning` to the top of the `PETS_input_factory.py` file * Make `PETS_KEYWORDS` uppercase, fix `PetsKeywords.from_file` type hint * Bump the suggested conda-forge Python version to 3.12 * Improve docstrings in `DeferredImageDraw` & `VideoStreamProcessor` * `on_frame`: scale 2-fold, clear a bunch of empty lines * `ExperimentalRats`: fix beam un/blanker * `ExperimentalRats`: export tracking to its own function * Merge `fix_pets_input` into `video_services` * Revert code unrelated to strictly stream processor updates * Improve `stream.processor.temporary_*`, docstring, add example code * Fix bug: clicking on VideoStreamFrame.panel.image was 2px off * Remove debug timing instructions * Fix clicked point was incorrect if window scrolled or partially visible * Rename `save_image` -> `save_frame`, new `save_image` job saves edited as PNG, remove unused imports * Remove discontinued showcase script * Fix incorrect yield placement in `processor.temporary` context * Revert "Revert code unrelated to strictly stream processor updates" This reverts commit a607a77. * Align with changes suggested in #135 (temporary as common context) * Fix bug: continuous geometry not interpreted correctly for PETS * Revert code left for development showcase purposes only * Simplify, document, rename the RATS frame to FastADT * Add FastADT experiment test (WIP), clear pytest.mark.parametrize w/ `InstanceAutoTracker` * Add changes done to movie delay branch after it split off * Update src/instamatic/experiments/fast_adt/experiment.py Co-authored-by: Stef Smeets <stefsmeets@users.noreply.github.com> * Update src/instamatic/experiments/fast_adt/experiment.py Co-authored-by: Stef Smeets <stefsmeets@users.noreply.github.com> * Update src/instamatic/experiments/fast_adt/experiment.py Co-authored-by: Stef Smeets <stefsmeets@users.noreply.github.com> * Apply selected @stefsmeets suggestions * Use dict literals here, these are much faster. Co-authored-by: Stef Smeets <stefsmeets@users.noreply.github.com> * Adapt FastADT frame to PR 135-137 suggestions * Remove extremely-cool-yet-unused `SubclassRegistryMeta` * Remove rather-cool-yet-unused `timer` context * Document `camera.dead_time` * Add `fast_adt` to settings modules so that it appears to new users * Add FastADT documentation * Bump the lowest supported version of numpy to 1.20 Numpy 1.19 was first numpy version to support python 3.9, so anyone even running on python 3.9 it extremely unlikely to still rely on it. Meanwhile, numpy 1.20 is the first to support very useful numpy.typing: https://numpy.org/devdocs/reference/typing.html * Increase readability, type hints of FastADT `Run` * OK, after 2nd thought, let's support Python3.9 users as we can * Replace custom `RuntimeError` `pass` with docstrings * Does not hurt; Update src/instamatic/experiments/fast_adt/experiment.py Co-authored-by: Stef Smeets <stefsmeets@users.noreply.github.com> * Update src/instamatic/experiments/fast_adt/experiment.py Co-authored-by: Stef Smeets <stefsmeets@users.noreply.github.com> * Update src/instamatic/experiments/fast_adt/experiment.py Co-authored-by: Stef Smeets <stefsmeets@users.noreply.github.com> * Improve the `config.md` description of `camera.dead_time` * Update src/instamatic/experiments/fast_adt/experiment.py Co-authored-by: Stef Smeets <stefsmeets@users.noreply.github.com> * Let pre-commit fix indentation errors in suggested docstring * Add a proper docstring for FastADT `Experiment.start_collection` * Hidden complex bit behind a method to makes things clearer :-) --------- Co-authored-by: Stef Smeets <stefsmeets@users.noreply.github.com>
1 parent 0225245 commit 63155a2

File tree

22 files changed

+907
-70
lines changed

22 files changed

+907
-70
lines changed

docs/app_data/instamatic/config/camera/test.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,3 +8,4 @@ interface: simulate
88
physical_pixelsize: 0.055
99
stretch_amplitude: 2.43
1010
stretch_azimuth: 83.37
11+
dead_time: 0.0

docs/config.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -206,6 +206,9 @@ This file holds the specifications of the camera. This file is must be located t
206206
**interface**
207207
: Give the interface of the camera interface to connect to, for example: `timepix`/`emmenu`/`simulate`/`gatan`/`merlin`. Leave blank or set to `None` to load the camera specs, but do not load the camera module (this also turns off the videostream gui).
208208

209+
**dead_time**
210+
: Set the dead time (i.e. the gap between acquisitions) of the detector; if this value (`camera.dead_time`) is not set but required, Instamatic might attempt to use `CalibMovieDelays.dead_time` value calibrated via `instamatic.calibrate_movie_delays` instead. Typically, Instamatic will not run this calibration automatically: the user needs to either set `camera.dead_time` or call `instamatic.calibrate_movie_delays` themselves.
211+
209212
**default_binsize**
210213
: Set the default binsize, default: `1`.
211214

docs/gui.md

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -154,6 +154,45 @@ Pressing `Start Collection` will collect a series of frames with the given tilt
154154
Data are output to `.tiff` and `.mrc`, including input files to read the data in PETS and REDp.
155155

156156

157+
## FastADT data collection
158+
159+
Module: `fast_adt`
160+
161+
A generic module used to run cRED experiments where the tracking is done a priori i.e. in a swoop directly preceding the diffraction experiment.
162+
163+
Before the experiment, decide on diffraction and tracking modes to be used. The diffraction can be collected as a series of static `stills` or a `continuous` movie. In the latter case, the rotation speed (upon calibration via `instamatic.calib_stage_rotation`) will be established automatically. If desired, a `manual` crystal tracking path can be also requested. Tracking is done just before the experiment. A series of images with crystal positions at different angles will be collected and user will be requested to point on points to measure and then confirm by clicking on the image stream.
164+
165+
![FastADT pane](images/gui_fast_adt.png)
166+
167+
For optimum performance, the FastADT frame uses three separate TEM setting which need to be preemptively set up by the user. The "Image", "Tracking" and "Diffraction" settings are used when collecting the respective kinds of images. They are saved between microscope sessions.
168+
169+
**Diffraction mode**
170+
: `stills` or `continuous`, used to switch between rotation and stills/precession-assisted experiment.
171+
172+
**Diffraction start (deg)**
173+
: The tilt series/swipe starting angle in degrees.
174+
175+
**Diffraction stop (deg)**
176+
: The tilt series/swipe ending angle in degrees.
177+
178+
**Diffraction step (deg)**
179+
: The target spacing between angles at which subsequent diffraction images are collected within the tilt series/swipe in degrees.
180+
181+
**Diffraction exposure (s)**
182+
: The time taken to collect each diffraction image in seconds. In the `continuous` mode it will additionally dictate the rotation speed.
183+
184+
**Tracking mode**
185+
: Dictates whether `none` or `manual` tracking is to be performed at the start of the experiment.
186+
187+
**Tracking step (deg)**
188+
: The target spacing between angles at which subsequent tracking images are collected within the tracking series in degrees.
189+
190+
**Tracking exposure (s)**
191+
: The time taken to collect each tracking image in seconds.
192+
193+
The function of config settings' `store` / `restore` buttons has been described above. The `start` button starts the (tracking and then) experiment. Further information and instructions concerning the `fast_adt` experiment may appear in the window and in the console. Collected data is automatically output in all available formats to the target directory at the end of experiment.
194+
195+
157196
## Machine learning
158197

159198
Module: `machine_learning`

docs/images/gui_fast_adt.png

113 KB
Loading

src/instamatic/_collections.py

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,11 @@
11
from __future__ import annotations
22

3+
import contextlib
4+
import logging
35
import string
6+
import time
47
from collections import UserDict
5-
from typing import Any, Tuple
8+
from typing import Any, Callable
69

710

811
class NoOverwriteDict(UserDict):
@@ -14,14 +17,21 @@ def __setitem__(self, key: Any, value: Any) -> None:
1417
super().__setitem__(key, value)
1518

1619

20+
class NullLogger(logging.Logger):
21+
def __init__(self, name='null'):
22+
super().__init__(name)
23+
self.addHandler(logging.NullHandler())
24+
self.propagate = False
25+
26+
1727
class PartialFormatter(string.Formatter):
1828
"""`str.format` alternative, allows for partial replacement of {fields}"""
1929

2030
def __init__(self, missing: str = '{{{}}}') -> None:
2131
super().__init__()
2232
self.missing: str = missing # used instead of missing values
2333

24-
def get_field(self, field_name: str, args, kwargs) -> Tuple[Any, str]:
34+
def get_field(self, field_name: str, args, kwargs) -> tuple[Any, str]:
2535
"""When field can't be found, return placeholder text instead."""
2636
try:
2737
obj, used_key = super().get_field(field_name, args, kwargs)

src/instamatic/config/camera/simulate.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,3 +18,4 @@ physical_pixelsize: 0.055
1818
possible_binsizes: [1]
1919
stretch_amplitude: 2.43
2020
stretch_azimuth: 83.37
21+
dead_time: 0.0

src/instamatic/config/camera/simulateDLL.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,3 +16,4 @@ interface: simulateDLL
1616
possible_binsizes: [1, 2, 4]
1717
stretch_amplitude: 0
1818
stretch_azimuth: 0
19+
dead_time: 0.0

src/instamatic/config/settings.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,7 @@ modules:
5959
- 'cred'
6060
- 'cred_tvips'
6161
- 'cred_fei'
62+
- 'fast_adt'
6263
- 'sed'
6364
- 'autocred'
6465
- 'red'

src/instamatic/experiments/__init__.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,5 +3,6 @@
33
from .autocred import experiment as autocRED
44
from .cred import experiment as cRED
55
from .cred_tvips import experiment as cRED_tvips
6+
from .fast_adt import experiment as fast_adt
67
from .red import experiment as RED
78
from .serialed import experiment as serialED

src/instamatic/experiments/experiment_base.py

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,11 @@ class ExperimentBase(ABC):
99
"""Experiment base class."""
1010

1111
@abstractmethod
12-
def start_collection(self):
12+
def __init__(self, *args, **kwargs) -> None:
13+
pass
14+
15+
@abstractmethod
16+
def start_collection(self, **kwargs):
1317
pass
1418

1519
def setup(self):

0 commit comments

Comments
 (0)