Merge pull request #295 from GeoscienceAustralia/develop

Release 0.5.0

Author: Matt Garthwaite

README.rst

@@ -1,4 +1,5 @@
 .. image:: docs/PyRate_logo_50.png
+   :alt: PyRate logo
 
 Python tool for InSAR Rate and Time-series Estimation
 =====================================================
@@ -21,21 +22,16 @@ The full PyRate documentation is available at http://geoscienceaustralia.github.
 Dependencies
 ------------
 
-The following dependencies need to be on your system prior to PyRate installation:
+The following system dependencies are required by PyRate:
 
 - `Python <https://www.python.org/downloads/>`_, versions 3.6, 3.7 or 3.8.
 - `GDAL <https://gdal.org/download.html>`_, versions 3.0.2 or 3.0.4
 - `Open MPI <https://www.open-mpi.org/software/ompi/v4.0/>`_, versions 2.1.6, 3.0.4, 3.1.4 or 4.0.2
 
 The versions of each package stated above have been tested to work.
 
-Python dependencies are:
+Python dependencies for PyRate are::
 
-::
-
-    Cython==0.29.16
-    Pillow==7.1.1
-    glob2==0.7
     joblib==0.14.1
     mpi4py==3.0.3
     networkx==2.4
@@ -49,15 +45,11 @@ Install
 
 Details of all install options are given in the `PyRate documentation <http://geoscienceaustralia.github.io/PyRate>`_.
 
-`PyRate` and its Python dependencies can be installed directly from the `Python Package Index (PyPI) <https://pypi.org/project/Py-Rate/>`_:
-
-::
+PyRate and its Python dependencies can be installed directly from the `Python Package Index (PyPI) <https://pypi.org/project/Py-Rate/>`_::
 
     pip install Py-Rate
 
-To install from source in Linux, enter these commands in a terminal:
-
-::
+Alternatively, to install from source and create an executable program in Linux, enter these commands in a terminal::
 
     cd ~
     git clone https://github.com/GeoscienceAustralia/PyRate.git
@@ -66,34 +58,49 @@ To install from source in Linux, enter these commands in a terminal:
     cd ~/PyRate
     python3 setup.py install
 
-This will install the above-listed Python dependencies and compile the executable program ``pyrate``:
-
-::
+This will install the above-listed Python dependencies and compile the executable program ``pyrate``.
+To learn more about using PyRate, type ``pyrate`` command in the terminal::
 
     >> pyrate --help
     usage: pyrate [-h] [-v {DEBUG,INFO,WARNING,ERROR}]
-                  {conv2tif,prepifg,process,merge,workflow} ...
+              {conv2tif,prepifg,correct,timeseries,stack,merge,workflow} ...
 
     PyRate workflow:
 
         Step 1: conv2tif
         Step 2: prepifg
-        Step 3: process
-        Step 4: merge
+        Step 3: correct
+        Step 4: timeseries
+        Step 5: stack
+        Step 6: merge
 
     Refer to https://geoscienceaustralia.github.io/PyRate/usage.html for
     more details.
 
     positional arguments:
-      {conv2tif,prepifg,process,merge,workflow}
+      {conv2tif,prepifg,correct,timeseries,stack,merge,workflow}
         conv2tif            Convert interferograms to geotiff.
-        prepifg             Perform multilooking and cropping on geotiffs.
-        process             Main processing workflow including corrections, time series and stacking computation.
+        prepifg             Perform multilooking, cropping and coherence masking to interferogram geotiffs.
+        correct             Calculate and apply corrections to interferogram phase data.
+        timeseries          Timeseries inversion of interferogram phase data.
+        stack               Stacking of interferogram phase data.
         merge               Reassemble computed tiles and save as geotiffs.
-        workflow            Run all the PyRate processes
+        workflow            Sequentially run all the PyRate processing steps.
 
     optional arguments:
       -h, --help            show this help message and exit
       -v {DEBUG,INFO,WARNING,ERROR}, --verbosity {DEBUG,INFO,WARNING,ERROR}
                             Increase output verbosity
 
+Test
+----
+
+To run the test suite, enter these commands in the terminal::
+
+   pip install -r requirements-test.txt
+   python3 -m pytest -m "not slow" tests/
+
+To run the tests for a single module (e.g. test_timeseries.py), use this command::
+
+   python3 -m pytest tests/test_timeseries.py
+

docs/PyRate_plot_screenshot.png

@@ -5,7 +5,7 @@
 import sphinx_rtd_theme
 
 
-__version__ = "0.4.0"
+__version__ = "0.5.0"
 # -- General configuration ------------------------------------------------
 
 # If your documentation needs a minimal Sphinx version, state it here.

docs/conf.py

@@ -1,16 +1,20 @@
 Dependencies
 ------------
 
-The following dependencies need to be on your system prior to `PyRate` installation:
+The following dependencies need to be on your system (or in your working
+environment) prior to `PyRate` installation:
 
 - Python_, versions 3.6, 3.7 or 3.8.
 - GDAL_, versions 3.0.2 or 3.0.4
 - `Open MPI`_, versions 2.1.6, 3.0.4, 3.1.4 or 4.0.2
 
-The versions of each package stated above have been tested to work.
+The versions of each package stated above have been tested to work using
+`Travis CI`_ continuous integration testing.
 
 .. _Python: https://www.python.org/downloads/
 .. _GDAL: https://gdal.org/download.html
 .. _`Open MPI`: https://www.open-mpi.org/software/ompi/v4.0/
+.. _`Travis CI`: https://travis-ci.org/github/GeoscienceAustralia/PyRate
 
-Other Python dependencies that will be installed are listed in ``PyRate/requirements.txt``.
+Other Python dependencies that will be installed are listed in
+``PyRate/requirements.txt``.

docs/dependencies.rst

@@ -1,13 +1,46 @@
 Docker
 ^^^^^^
 
-Docker can be used to run `PyRate` on Linux, Windows and MacOS.
-Please follow the instructions at https://hub.docker.com/ to download 
-and install Docker on your machine. Then run:
+Docker can be used to run `PyRate` on Linux, Windows and MacOS systems.
 
-::
+.. note::
+
+    - Docker is the recommended method to use `PyRate` under Windows.
+
+The only system pre-requisites for using docker are Git and Docker Desktop
+(e.g. for Windows https://docs.docker.com/docker-for-windows/ ).
+The other system dependencies required by PyRate are installed in to the
+docker container itself during the build operation.
+Once Git and Docker Desktop are installed, and Docker Desktop is running,
+execute the following at a command line prompt (e.g. PowerShell in Windows 10)::
 
     git clone git@github.com:GeoscienceAustralia/PyRate.git
     cd PyRate
-    docker build -t pyrate-image:1.0 .
-    docker run -it --rm -v %cd%:/PyRate pyrate-image:1.0 /bin/bash
+    docker build -t pyrate-image .
+    docker run -it pyrate-image
+
+.. note::
+
+    - The image name “pyrate-image” is not mandatory.
+      You are free to name the docker image whatever you choose.
+
+The ``docker build`` command builds the docker container described in the
+``Dockerfile`` in the `PyRate` repo. It first starts by setting up an Ubuntu linux
+system image, and then installs the required system and python dependencies.
+
+Once the docker container is running (you will see a different-looking command
+prompt in the PowerShell), execute the following commands::
+
+    source /usr/local/bin/virtualenvwrapper.sh
+    workon pyrate
+    cd PyRate
+    python3 setup.py install
+    pyrate --help
+    pyrate workflow -f input_parameters.conf
+
+The ``pyrate`` executable program is built as part of the ``docker build`` step.
+If the ``pyrate`` executable is reported as "not found", re-run the compilation::
+
+    cd PyRate
+    python3 setup.py install
+

docs/docker.rst

@@ -3,6 +3,48 @@
 Release History
 ===============
 
+0.5.0 (2020-09-08)
+------------------
+Added
++++++
+- New functionality "``linear_rate``" to calculate linear regression of
+  cumulative displacement time series for every pixel as part of the ``timeseries`` step.
+- Script for plotting ``timeseries`` and ``linear_rate`` output geotiff products using Matplotlib.
+  To use, additional dependencies listed in ``requirements-plot.txt`` are required.
+- Correction data (except ``maxvar`` and ``vcmt``) applied to the ifg data is saved to disk
+  and re-used on subsequent repeat runs. Corrections are only re-calculated if config
+  parameters change between runs.
+- MPI parallelisation of APS spatio-temporal filter correction.
+- Unit test coverage for refpixel lat/lon to x/y conversion and ``aps`` module.
+
+Fixed
++++++
+- Re-enable ``ifglksx`` and ``ifglksy`` to be different values, resulting in different
+  resolutions in x and y dimensions in multi-looked interferograms.
+- Re-enable ``orbfitlksx`` and ``orbfitlksy`` to be different values, resulting in different
+  resolutions in x and y dimensions during network orbit correction.
+- Screen messages from main process only during MPI runs.
+
+Changed
++++++++
+- ``process`` step has been renamed ``correct``. Stacking and timeseries have been removed from
+  this step and are now invoked by separate ``timeseries`` and ``stack`` command line options.
+- Processing of coherence files by ``conv2tif`` and ``prepifg`` is now triggered by the presence
+  of ``cohfilelist`` in the config file. If the list is present, multilooked/cropped coherence
+  files are saved to disk, regardless of whether ``cohmask`` is 0 or 1.
+- Parallelisation capability is refactored - MPI and multiprocessing both now use a common
+  tiling framework for ``stack``, ``timeseries`` and ``mst`` algorithms.
+- Introduced a simplified and standardised file naming format for files produced by the
+  ``prepifg`` step. Information previously saved in the filename (e.g. ``ifglksx``, ``ifglksy``,
+  and ``ifgcropopt`` values) is now saved to the geotiff header instead.
+
+Removed
++++++++
+- Redundant ``tscal`` config parameter was deprecated - not needed now there is a ``timeseries``
+  step invokable on the command line.
+- Unused ``Pillow``, ``cython`` and ``glob2`` dependencies.
+- Deprecated function ``pyrate.prepifg_helper.prepare_ifgs``, which is no longer needed.
+
 0.4.3 (2020-08-04)
 ------------------
 Added

docs/history.rst

@@ -7,47 +7,34 @@ HPC platforms may differ.
 
 .. _`National Computational Infrastructure (NCI)`: https://nci.org.au/ 
 
-Login to Gadi and clone the `PyRate` repository:
-
-::
+Login to Gadi and clone the `PyRate` repository::
 
     ssh <user_name>@gadi.nci.org.au
     cd ~
     git clone https://github.com/GeoscienceAustralia/PyRate.git
 
-Load the required Gadi modules (this will also remove the default NCI GDAL
-Python bindings so we can build and use our own):
-
-::
-
-    source ~/PyRate/scripts/nci_load_modules.sh
-
-Create a Python virtual environment:
-
-::
+Create a Python virtual environment::
 
     python3 -m venv ~/PyRateVenv
     source ~/PyRateVenv/bin/activate
 
-Install `PyRate`:
+Load the required Gadi modules (this will also remove the default NCI GDAL
+Python bindings so we can build and use our own)::
+
+    source ~/PyRate/scripts/nci_load_modules.sh
 
-::
+Install `PyRate`::
 
     cd ~/PyRate
     python3 setup.py install
 
-
 Following this, `PyRate` will be available for PBS jobs. To verify the 
-installation, first run an interactive session:
-
-::
+installation, first run an interactive session::
 
     qsub -I -q express -l walltime=01:00:00,mem=16Gb,ncpus=4,wd
 
 Once the session has started, you will need to reactivate your virtual 
-environment and reload the required modules:
-
-::
+environment and reload the required modules::
 
     source ~/PyRateVenv/bin/activate
     source ~/PyRate/scripts/nci_load_modules.sh

docs/hpc.rst

@@ -2,7 +2,8 @@ Installation
 ============
 
 This is an installation guide to get `PyRate` running on various platforms.
-Follow the instructions to install `PyRate` and run a small toy example.
+Follow the instructions to install `PyRate` and run a small toy example using
+the test dataset included in the repository.
 
 .. include:: dependencies.rst
 
@@ -12,14 +13,13 @@ Platforms
 PyPI
 ^^^^
 
-`PyRate` and its Python dependencies can be installed directly from the Python Package Index (PyPI_):
-
-.. _PyPI: https://pypi.org/project/Py-Rate/
-
-::
+`PyRate` and its Python dependencies can be installed directly from the
+Python Package Index (PyPI_)::
 
     pip install Py-Rate
 
+.. _PyPI: https://pypi.org/project/Py-Rate/
+
 .. include:: ubuntu.rst
 .. include:: docker.rst
 .. include:: hpc.rst
@@ -28,19 +28,17 @@ PyPI
 Verify Installation
 -------------------
 
-To verify `PyRate` has been successfully installed, run the workflow with the
-example config file and the small dataset included in the repository:
+To verify `PyRate` has been successfully installed, run the full processing
+workflow with the example config file and the small dataset included
+in the repository. If you compiled the ``pyrate`` executable program::
+
+    pyrate workflow -f input_parameters.conf
 
-::
+If you installed from the Python Package Index (PyPI_)::
 
-    pyrate conv2tif -f input_parameters.conf
-    pyrate prepifg -f input_parameters.conf
-    pyrate process -f input_parameters.conf
-    pyrate merge -f input_parameters.conf
+    python3 pyrate/main.py workflow -f input_parameters.conf
 
 If the installation has been successful, this workflow will complete without 
-errors and geotiff files will be available in:
+errors and geotiff files will be available in::
 
-:: 
-    
     ~/PyRate/out

docs/installation.rst

@@ -1,7 +1,7 @@
-PyRate Process Script
+PyRate Correct Script
 ========================
 
-.. automodule:: pyrate.process
+.. automodule:: pyrate.correct
     :members:
     :undoc-members:
     :show-inheritance:

docs/pyrate.process.rst renamed to docs/pyrate.correct.rst

@@ -1,12 +1,14 @@
 PyRate Main Script
 ==================
 
-.. automodule:: pyrate.__main__
+.. automodule:: pyrate.main
 
    .. rubric:: Functions
    .. autosummary::
 
       conv2tif
       prepifg
-      process
+      correct
+      timeseries
+      stack
       merge

docs/pyrate.__main__.rst renamed to docs/pyrate.main.rst

@@ -6,6 +6,6 @@ PyRate Scripts
 
    pyrate.conv2tif
    pyrate.prepifg
-   pyrate.process
+   pyrate.correct
    pyrate.merge
-   pyrate.__main__
+   pyrate.main