Merge pull request #283 from GeoscienceAustralia/develop

Release 0.4.3

Author: Matt Garthwaite

.travis.yml

@@ -58,13 +58,12 @@ install:
 
 
 script:
-  - travis_wait 30 pytest --cov-config=.coveragerc --cov-report term-missing:skip-covered --cov=pyrate tests/
-
+  - pytest tests/ -m "slow"
+  - pytest --cov-config=.coveragerc --cov-report term-missing:skip-covered --cov=pyrate tests/ -m "not slow"
 
 after_success:
   - codecov
 
-
 before_deploy:
   - cd docs && make html
 

docs/conf.py

@@ -39,8 +39,8 @@
 source_suffix = ['.rst', '.md']
 # source_suffix = '.rst'
 
-# The master toctree document.
-master_doc = 'index'
+# The main toctree document.
+main_doc = 'index'
 
 # General information about the project.
 project = 'PyRate'
@@ -137,7 +137,7 @@
 # (source start file, target name, title,
 #  author, documentclass [howto, manual, or own class]).
 latex_documents = [
-    (master_doc, 'PyRate.tex', 'PyRate Documentation',
+    (main_doc, 'PyRate.tex', 'PyRate Documentation',
      'Geoscience Australia InSAR team', 'manual'),
 ]
 
@@ -147,7 +147,7 @@
 # One entry per manual page. List of tuples
 # (source start file, name, description, authors, manual section).
 man_pages = [
-    (master_doc, 'pyrate', 'PyRate Documentation',
+    (main_doc, 'pyrate', 'PyRate Documentation',
      [author], 1)
 ]
 
@@ -158,7 +158,7 @@
 # (source start file, target name, title, author,
 #  dir menu entry, description, category)
 texinfo_documents = [
-    (master_doc, 'PyRate', 'PyRate Documentation',
+    (main_doc, 'PyRate', 'PyRate Documentation',
      author, 'PyRate', 'One line description of project.',
      'Miscellaneous'),
 ]

docs/contributing.rst

@@ -80,17 +80,17 @@ Ready to contribute? Here's how to set up `PyRate` for local development.
 
 Now you can make your changes locally.
 
-4. When you have finished making changes, check that your changes pass style and unit
+4. When you have finished making changes, check that your changes pass against unit
    tests. A suite of tests have been developed for use in testing `PyRate` functionality
    and for further code development. The tests use `pytest <http://doc.pytest.org/en/latest/>`__
-   and can be found in the ``PyRate/tests/`` directory. A small test dataset used by the test suite is included in the repository in the ``PyRate/tests/test_data/small_test`` directory.
+   and can be found in the ``PyRate/tests/`` directory. A small test dataset used by
+   the test suite is included in the repository in the ``PyRate/tests/test_data/small_test`` directory.
 
 ::
 
     cd ~/PyRate
-    pytest tests/
-    pip install tox
-    tox
+    # 'not slow' avoids running those tests marked as 'slow'
+    pytest tests/ -m "not slow"
 
 5. If the tests pass, commit your changes and push your branch to GitHub:
 

docs/history.rst

@@ -3,6 +3,33 @@
 Release History
 ===============
 
+0.4.3 (2020-07-28)
+------------------
+Added
++++++
+- Ability to define the order of steps in the ``process`` workflow
+  (default order unchanged).
+  
+Fixed
++++++
+
+Changed
++++++++
+- ``prepifg`` output interferograms are saved as read-only files.
+- ``process`` makes a writable copy of the ``prepifg`` output data
+  at the beginning of each run.
+- The selected reference pixel is saved to disk and re-used on subsequent
+  ``process`` runs.  
+- Saving of incremental time series (``tsincr``) products is optional,
+  controlled by the ``savetsincr`` configuration parameter (default is on).
+
+Removed
++++++++
+- Removed obsolete InSAR terminology from code, docs and test data
+  (changed to `first` and `second` images).
+- Stopped using ``unittest`` unit test framework in favour of exclusively
+  using ``pytest``.
+
 0.4.2 (2020-06-26)
 ------------------
 Added
@@ -16,12 +43,15 @@ Added
   lat/lon and x/y values; mean and standard deviation of reference window samples.
 - Quicklook PNG and KML files are generated for the ``Stack Rate`` error map by default.
 
+Fixed
++++++
+- Ensure ``prepifg`` treats input data files as `read only`.
+- Fix the way that the reference phase is subtracted from interferograms
+  during ``process`` step.
+- Manual entry of ``refx/y`` converted to type ``int``.
+
 Changed
 +++++++
-- Bugfix: ensure ``prepifg`` treats input data files as `read only`.
-- Bugfix: fix the way that the reference phase is subtracted from interferograms
-  during ``process`` step.
-- Bugfix: manual entry of ``refx/y`` converted to type ``int``.
 - User supplies latitude and longitude values when specifying a reference pixel in
   the config file. Pixel x/y values are calculated and used internally.
 - Move ``Stack Rate`` masking to a standalone function ``pyrate.core.stack.mask_rate``,
@@ -55,15 +85,18 @@ Added
 - Improvements to the test suite, including systems-wide tests.
 - Improved logging.
 
+Fixed
++++++
+- Fixed bug in resampling/multi-looking when coherence masking is used.
+  This bugfix will result in significantly fewer ``nan`` pixels in the outputs.
+- Fixed a bug in how NaNs are handled during coherence masking and multi-looking.
+  Output rasters will contain ``nan`` as the nodata value.
+
 Changed
 +++++++
 - ``Linear Rate`` algorithm has been renamed ``Stack Rate``.
 - User supplies full paths to input files in respective file lists.
 - All files generated by `PyRate` saved to user-defined ``outdir`` directory.
-- Fixed a bug in how NaNs are handled during coherence masking and multi-looking.
-  Output rasters will contain ``nan`` as the nodata value.
-- Fixed bug in resampling/multi-looking when coherence masking is used.
-  This bugfix will result in significantly fewer ``nan`` pixels in the outputs.
 - Renamed ``slcfilelist`` parameter to ``hdrfilelist``.
 - Log files are generated in the ``outdir`` and every `PyRate` step produces independent log files.
 
@@ -129,7 +162,7 @@ Changed
 +++++++
 - Requirements now managed by ``requirements.txt`` file, parsed by ``setup.py``.
 - Requirements now split across base ``requirements.txt`` and separate files
-  for dev (``requirements-dev.txt``) and testing ('requirements-test.txt').
+  for dev (``requirements-dev.txt``) and testing (``requirements-test.txt``).
 - Moved default config files to top level source directory.
 - Pinned Python dependencies to specific versions.
 - Travis build now installs GDAL from apt.

input_parameters.conf

@@ -18,6 +18,9 @@ tscal:      1
 # Optional save of numpy array files for output products (MERGE)
 savenpy:    0
 
+# Optional save of incremental time series products (PROCESS/MERGE)
+savetsincr: 0
+
 #%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 # Multi-threading parameters used by stacking/timeseries/prepifg
 # gamma prepifg runs in parallel on a single machine if parallel = 1

pyrate/configuration.py

@@ -19,11 +19,12 @@
 """
 from configparser import ConfigParser
 from pathlib import Path, PurePath
+from typing import Union
 from pyrate.constants import NO_OF_PARALLEL_PROCESSES
 from pyrate.default_parameters import PYRATE_DEFAULT_CONFIGURATION
 from pyrate.core.algorithm import factorise_integer
 from pyrate.core.shared import extract_epochs_from_filename, InputTypes
-from pyrate.core.config import parse_namelist, ConfigException
+from pyrate.core.config import parse_namelist, ConfigException, ORB_ERROR_DIR, TEMP_MLOOKED_DIR
 
 
 def set_parameter_value(data_type, input_value, default_value, required, input_name):
@@ -78,28 +79,38 @@ def validate_file_list_values(file_list, no_of_epochs):
 
 class MultiplePaths:
     def __init__(self, out_dir: str, file_name: str, ifglksx: int = 1, ifgcropopt: int = 1,
-                 input_type: InputTypes = InputTypes.IFG):
+                 input_type: InputTypes = InputTypes.IFG,  tempdir: Union[Path, str] = TEMP_MLOOKED_DIR):
         self.input_type = input_type
         b = Path(file_name)
         if b.suffix == ".tif":
             self.unwrapped_path = None
             converted_path = b  # original file
             self.sampled_path = Path(out_dir).joinpath(
-                b.stem + '_' + str(ifglksx) + "rlks_" + str(ifgcropopt) + "cr.tif").as_posix()
+                b.stem + '_' + str(ifglksx) + "rlks_" + str(ifgcropopt) + "cr.tif")
         else:
             self.unwrapped_path = b.as_posix()
             converted_path = Path(out_dir).joinpath(
                 b.stem.split('.')[0] + '_' + b.suffix[1:] + input_type.value).with_suffix('.tif')
             self.sampled_path = converted_path.with_name(
-                converted_path.stem + '_' + str(ifglksx) + "rlks_" + str(ifgcropopt) + "cr.tif").as_posix()
+                converted_path.stem + '_' + str(ifglksx) + "rlks_" + str(ifgcropopt) + "cr.tif")
+        if not isinstance(tempdir, Path):
+            tempdir = Path(out_dir).joinpath(tempdir)
+        self.tmp_sampled_path = tempdir.joinpath(self.sampled_path.name).as_posix()
+        self.sampled_path = self.sampled_path.as_posix()
         self.converted_path = converted_path.as_posix()
 
     def __str__(self):  # pragma: no cover
-        return"""
-            unwrapped_path = """ + self.unwrapped_path+""" 
+        st = ""
+        if self.unwrapped_path is not None:
+            st += """\nunwrapped_path = """ + self.unwrapped_path
+        else:
+            st += """\nunwrapped_path = None"""
+        st += """
             converted_path = """ + self.converted_path+""" 
             sampled_path = """ + self.sampled_path+"""    
+            tmp_sampled_path = """ + self.tmp_sampled_path+"""
             """
+        return st
 
 
 class Configuration:
@@ -112,12 +123,26 @@ def __init__(self, config_file_path):
         with open(config_file_path) as stream:
             parser.read_string("[root]\n" + stream.read())
 
-        for key, value in parser._sections["root"].items():
+        for key, value in parser["root"].items():
             self.__dict__[key] = value
 
         # make output path, if not provided will error
         Path(self.outdir).mkdir(exist_ok=True, parents=True)
 
+        # custom process sequence if 'process' section is provided in config
+        if 'process' in parser and 'steps' in parser['process']:
+            self.__dict__['process'] = list(filter(None, parser['process'].get('steps').splitlines()))
+        else:
+            self.__dict__['process'] = [
+                'orbfit',
+                'refphase',
+                'mst',
+                'apscorrect',
+                'maxvar',
+                'timeseries',
+                'stack'
+            ]
+
         # Validate required parameters exist.
 
         required = {k for k, v in PYRATE_DEFAULT_CONFIGURATION.items() if v['Required']}
@@ -162,6 +187,14 @@ def __init__(self, config_file_path):
 
         self.tmpdir.mkdir(parents=True, exist_ok=True)
 
+        # create orbfit error dir
+        self.orb_error_dir = Path(self.outdir).joinpath(ORB_ERROR_DIR)
+        self.orb_error_dir.mkdir(parents=True, exist_ok=True)
+
+        # create temp multilooked files dir
+        self.temp_mlooked_dir = Path(self.outdir).joinpath(TEMP_MLOOKED_DIR)
+        self.temp_mlooked_dir.mkdir(parents=True, exist_ok=True)
+
         # var no longer used
         self.APS_ELEVATION_EXT = None
         self.APS_INCIDENCE_EXT = None
@@ -194,4 +227,13 @@ def __init__(self, config_file_path):
     def __get_files_from_attr(self, attr, input_type=InputTypes.IFG):
         val = self.__getattribute__(attr)
         files = parse_namelist(val)
-        return [MultiplePaths(self.outdir, p, self.ifglksx, self.ifgcropopt, input_type=input_type) for p in files]
+        return [
+            MultiplePaths(self.outdir, p, self.ifglksx, self.ifgcropopt, input_type=input_type,
+                          tempdir=self.temp_mlooked_dir) for p in files
+        ]
+
+
+def write_config_parser_file(conf: ConfigParser, output_conf_file: Union[str, Path]):
+    """replacement function for write_config_file which uses dict instead of a ConfigParser instance"""
+    with open(output_conf_file, 'w') as configfile:
+        conf.write(configfile)

pyrate/core/algorithm.py

@@ -125,7 +125,7 @@ def unit_vector(incidence, azimuth):
 
 def ifg_date_lookup(ifgs, date_pair):
     """
-    Returns the Interferogram which has the master and slave dates given
+    Returns the Interferogram which has the first and second dates given
     in 'date_pair'.
 
     :param list ifgs: List of interferogram objects to search
@@ -136,10 +136,10 @@ def ifg_date_lookup(ifgs, date_pair):
     """
 
     if len(date_pair) != 2:
-        msg = "Need (datetime.date, datetime.date) master/slave pair"
+        msg = "Need (datetime.date, datetime.date) first/second image pair"
         raise IfgException(msg)
 
-    # check master/slave dates are in order
+    # check first/second dates are in order
     try:
         # TODO: Clarify: Is the comparison here for a different date?
         # Then it should be written in a more pythonic way
@@ -151,17 +151,17 @@ def ifg_date_lookup(ifgs, date_pair):
         raise ValueError("Bad date_pair arg to ifg_date_lookup()")
 
     for i in ifgs:
-        if date_pair == (i.master, i.slave):
+        if date_pair == (i.first, i.second):
             return i
 
-    raise ValueError("Cannot find Ifg with "
-                     "master/slave of %s" % str(date_pair))
+    raise ValueError("Cannot find Ifg with first/second"
+                     "image dates of %s" % str(date_pair))
 
 
 def ifg_date_index_lookup(ifgs, date_pair):
     """
-    Returns the Interferogram index which has the master and slave dates
-    given in 'date_pair'.
+    Returns the Interferogram index which has the first and second image
+    dates given in 'date_pair'.
 
     :param list ifgs: List of interferogram objects to search
     :param tuple date_pair: A (datetime.date, datetime.date)
@@ -171,21 +171,21 @@ def ifg_date_index_lookup(ifgs, date_pair):
     """
 
     if len(date_pair) != 2:
-        msg = "Need (datetime.date, datetime.date) master/slave pair"
+        msg = "Need (datetime.date, datetime.date) first/second image date pair"
         raise IfgException(msg)
 
-    # check master/slave dates are in order
+    # check first/second image dates are in order
     try:
         if date_pair[0] > date_pair[1]:
             date_pair = date_pair[1], date_pair[0]
     except:
         raise ValueError("Bad date_pair arg to ifg_date_lookup()")
 
     for i, _ in enumerate(ifgs):
-        if date_pair == (ifgs[i].master, ifgs[i].slave):
+        if date_pair == (ifgs[i].first, ifgs[i].second):
             return i
 
-    raise ValueError("Cannot find Ifg with master/slave of %s" % str(date_pair))
+    raise ValueError("Cannot find Ifg with first/second image dates of %s" % str(date_pair))
 
 
 def get_epochs(ifgs: Union[Iterable, Dict]) -> Tuple[EpochList, int]:
@@ -211,18 +211,18 @@ def get_epochs(ifgs: Union[Iterable, Dict]) -> Tuple[EpochList, int]:
 
 def get_all_epochs(ifgs):
     """
-    Returns a sequence of all master and slave dates in given interferograms.
+    Returns a sequence of all image dates used to form given interferograms.
 
     :param list ifgs: List of interferogram objects
 
-    :return: master and slave dates
+    :return: list of all image dates
     :rtype: list
     """
 
-    return [ifg.master for ifg in ifgs] + [ifg.slave for ifg in ifgs]
+    return [ifg.first for ifg in ifgs] + [ifg.second for ifg in ifgs]
 
 
-def master_slave_ids(dates):
+def first_second_ids(dates):
     """
     Returns a dictionary of 'date:unique ID' for each date in 'dates'.
     IDs are ordered from oldest to newest, starting at 0.