Skip to content

Commit 29ce59c

Browse files
author
Matt Garthwaite
authored
Merge pull request #274 from GeoscienceAustralia/develop
Release 0.4.2
2 parents ceabb05 + 8b4fee0 commit 29ce59c

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

60 files changed

+1361
-2450
lines changed

.travis.yml

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -53,14 +53,11 @@ install:
5353
- pip install GDAL==$(gdal-config --version)
5454
- python setup.py install
5555
- rm -rf Py_Rate.egg-info # remove the local egg
56-
- export PYRATEPATH=$(pwd)
5756
- export PYTHONPATH=$PYRATEPATH:$PYTHONPATH
5857
- chmod 444 tests/test_data/small_test/tif/geo_070709-070813_unw.tif # makes the file readonly, used in a test
5958

6059
# command to run tests, e.g. python setup.py test
6160
script:
62-
# - python scripts/update_placeholder_paths.py
63-
- mpirun -n 3 pytest tests/test_mpi.py
6461
- pytest --cov-config=.coveragerc --cov-report term-missing:skip-covered --cov=pyrate tests/
6562

6663

@@ -79,7 +76,7 @@ deploy:
7976
verbose: true
8077
on:
8178
branch: master
82-
condition: $GDALVERSION="3.0.2" && $TRAVIS_PYTHON_VERSION=3.8.*
79+
python: 3.8
8380
github_token: $GITHUB_TOKEN2
8481
local_dir: docs/_build/html
8582
project_name: PyRate

docs/history.rst

Lines changed: 45 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,50 @@
22
33
Release History
44
===============
5+
6+
0.4.2 (2020-06-26)
7+
------------------
8+
Added
9+
+++++
10+
- Save full-res coherence files to disk in ``conv2tif`` step if ``cohmask = 1``.
11+
- Save multi-looked coherence files to disk in ``prepifg`` step if ``cohmask = 1``.
12+
- Additional ``DATA_TYPE`` geotiff header metadata for above coherence files.
13+
- ``conv2tif`` and ``prepifg`` output files have a tag applied to filename dependent
14+
on data type, i.e. ``_ifg.tif``, ``_coh.tif``, ``_dem.tif``.
15+
- Metadata about used reference pixel is added to interferogram geotiff headers:
16+
lat/lon and x/y values; mean and standard deviation of reference window samples.
17+
- Quicklook PNG and KML files are generated for the ``Stack Rate`` error map by default.
18+
19+
Changed
20+
+++++++
21+
- Bugfix: ensure ``prepifg`` treats input data files as `read only`.
22+
- Bugfix: fix the way that the reference phase is subtracted from interferograms
23+
during ``process`` step.
24+
- Bugfix: manual entry of ``refx/y`` converted to type ``int``.
25+
- User supplies latitude and longitude values when specifying a reference pixel in
26+
the config file. Pixel x/y values are calculated and used internally.
27+
- Move ``Stack Rate`` masking to a standalone function ``pyrate.core.stack.mask_rate``,
28+
applied during the ``merge`` step and add unit tests.
29+
- Skip ``Stack Rate`` masking if threshold parameter ``maxsig = 0``.
30+
- Provide log message indicating the percentage of pixels masked by
31+
``pyrate.core.stack.mask_rate``.
32+
- Refactor ``pyrate.core.stack`` module; expose two functions in documentation:
33+
i) single pixel stacking algorithm, and
34+
ii) loop function for processing full ifg array.
35+
- Refactor ``pyrate.merge`` script; remove duplicated code and create reusable
36+
generic functions.
37+
- Colourmap used to render quicklook PNG images is calculated from min/max values of
38+
the geotiff band.
39+
- Updated ``test`` and ``dev`` requirements.
40+
41+
Removed
42+
+++++++
43+
- Deprecate unused functions in ``pyrate.core.config`` and corresponding tests.
44+
- Static colourmap ``utils/colourmap.txt`` that was previously used to render
45+
quicklook PNG images is removed.
46+
547
0.4.1 (2020-05-19)
6-
-----------------------
48+
------------------
749
Added
850
+++++
951
- Python 3.8 support.
@@ -31,7 +73,7 @@ Removed
3173
- Deprecate ``parallel = 2`` option; splitting image via rows for parallelisation.
3274

3375
0.4.0 (2019-10-31)
34-
-----------------------
76+
------------------
3577
Added
3678
+++++
3779
- Python 3.7 support.
@@ -64,7 +106,7 @@ Removed
64106
- Unused tests for legacy api.
65107

66108
0.3.0 (2019-07-26)
67-
-----------------------
109+
------------------
68110
Added
69111
+++++
70112
- ``utils/apt_install.sh`` script that lists Ubuntu/apt package requirements.

input_parameters.conf

Lines changed: 14 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -4,24 +4,27 @@
44
# Optional ON/OFF switches - ON = 1; OFF = 0
55

66
# Coherence masking (PREPIFG)
7-
cohmask: 1
7+
cohmask: 0
88

99
# Orbital error correction (PROCESS)
10-
orbfit: 1
10+
orbfit: 1
1111

1212
# APS correction using spatio-temporal filter (PROCESS)
13-
apsest: 0
13+
apsest: 0
1414

1515
# Time series calculation (PROCESS)
16-
tscal: 1
16+
tscal: 1
17+
18+
# Optional save of numpy array files for output products (MERGE)
19+
savenpy: 0
1720

1821
#%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
1922
# Multi-threading parameters used by stacking/timeseries/prepifg
2023
# gamma prepifg runs in parallel on a single machine if parallel = 1
2124
# parallel: 1 = parallel, 0 = serial
22-
parallel: 0
25+
parallel: 0
2326
# number of processes
24-
processes: 8
27+
processes: 8
2528

2629
#%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
2730
# Input/Output file locations
@@ -85,8 +88,8 @@ nan_conversion: 1
8588
# refnx/y: number of search grid points in x/y image dimensions
8689
# refchipsize: size of the data window at each search grid point
8790
# refminfrac: minimum fraction of valid (non-NaN) pixels in the data window
88-
refx:
89-
refy:
91+
refx: 150.941666654
92+
refy: -34.218333314
9093
refnx: 5
9194
refny: 5
9295
refchipsize: 5
@@ -149,9 +152,9 @@ ts_pthr: 10
149152
#------------------------------------
150153
# Stacking calculation parameters
151154

152-
# pthr: minimum number of coherent ifg connections for each pixel
153-
# nsig: n-sigma used as residuals threshold for iterative least squares stacking
154-
# maxsig: maximum residual used as a threshold for values in the rate map
155+
# pthr: threshold for minimum number of ifg observations for each pixel
156+
# nsig: threshold for iterative removal of observations
157+
# maxsig: maximum sigma (std dev) used as an output masking threshold applied in Merge step. 0 = OFF.
155158
pthr: 5
156159
nsig: 3
157160
maxsig: 1000

pyrate/configuration.py

Lines changed: 19 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -13,13 +13,16 @@
1313
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1414
# See the License for the specific language governing permissions and
1515
# limitations under the License.
16+
"""
17+
This Python module contains utilities to validate user input parameters
18+
parsed in a PyRate configuration file.
19+
"""
1620
from configparser import ConfigParser
1721
from pathlib import Path, PurePath
18-
import re
1922
from pyrate.constants import NO_OF_PARALLEL_PROCESSES
2023
from pyrate.default_parameters import PYRATE_DEFAULT_CONFIGURATION
2124
from pyrate.core.algorithm import factorise_integer
22-
from pyrate.core.shared import extract_epochs_from_filename
25+
from pyrate.core.shared import extract_epochs_from_filename, InputTypes
2326
from pyrate.core.config import parse_namelist, ConfigException
2427

2528

@@ -44,17 +47,17 @@ def validate_parameter_value(input_name, input_value, min_value=None, max_value=
4447
if min_value is not None:
4548
if input_value < min_value: # pragma: no cover
4649
raise ValueError(
47-
"Invalid value for " + str(input_name) + " supplied: " + str(input_value) + ". Please provided a valid value greater than " + str(min_value) + ".")
50+
"Invalid value for " + str(input_name) + " supplied: " + str(input_value) + ". Provide a value greater than or equal to " + str(min_value) + ".")
4851
if input_value is not None:
4952
if max_value is not None:
5053
if input_value > max_value: # pragma: no cover
5154
raise ValueError(
52-
"Invalid value for " + str(input_name) + " supplied: " + str(input_value) + ". Please provided a valid value less than " + str(max_value) + ".")
55+
"Invalid value for " + str(input_name) + " supplied: " + str(input_value) + ". Provide a value less than or equal to " + str(max_value) + ".")
5356

5457
if possible_values is not None:
5558
if input_value not in possible_values: # pragma: no cover
5659
raise ValueError(
57-
"Invalid value for " + str(input_name) + " supplied: " + str(input_value) + ". Please provided a valid value from with in: " + str(possible_values) + ".")
60+
"Invalid value for " + str(input_name) + " supplied: " + str(input_value) + ". Provide a value from: " + str(possible_values) + ".")
5861
return True
5962

6063

@@ -74,7 +77,9 @@ def validate_file_list_values(file_list, no_of_epochs):
7477

7578

7679
class MultiplePaths:
77-
def __init__(self, out_dir, file_name, ifglksx=1, ifgcropopt=1):
80+
def __init__(self, out_dir: str, file_name: str, ifglksx: int = 1, ifgcropopt: int = 1,
81+
input_type: InputTypes = InputTypes.IFG):
82+
self.input_type = input_type
7883
b = Path(file_name)
7984
if b.suffix == ".tif":
8085
self.unwrapped_path = None
@@ -83,7 +88,8 @@ def __init__(self, out_dir, file_name, ifglksx=1, ifgcropopt=1):
8388
b.stem + '_' + str(ifglksx) + "rlks_" + str(ifgcropopt) + "cr.tif").as_posix()
8489
else:
8590
self.unwrapped_path = b.as_posix()
86-
converted_path = Path(out_dir).joinpath(b.stem + '_' + b.suffix[1:]).with_suffix('.tif')
91+
converted_path = Path(out_dir).joinpath(
92+
b.stem.split('.')[0] + '_' + b.suffix[1:] + input_type.value).with_suffix('.tif')
8793
self.sampled_path = converted_path.with_name(
8894
converted_path.stem + '_' + str(ifglksx) + "rlks_" + str(ifgcropopt) + "cr.tif").as_posix()
8995
self.converted_path = converted_path.as_posix()
@@ -171,20 +177,21 @@ def __init__(self, config_file_path):
171177
if self.cohfilelist is not None:
172178
# if self.processor != 0: # not roipac
173179
validate_file_list_values(self.cohfilelist, 1)
174-
self.coherence_file_paths = self.__get_files_from_attr('cohfilelist')
180+
self.coherence_file_paths = self.__get_files_from_attr('cohfilelist', input_type=InputTypes.COH)
175181

176-
self.header_file_paths = self.__get_files_from_attr('hdrfilelist')
182+
self.header_file_paths = self.__get_files_from_attr('hdrfilelist', input_type=InputTypes.HEADER)
177183

178184
self.interferogram_files = self.__get_files_from_attr('ifgfilelist')
179185

180-
self.dem_file = MultiplePaths(self.outdir, self.demfile, self.ifglksx, self.ifgcropopt)
186+
self.dem_file = MultiplePaths(self.outdir, self.demfile, self.ifglksx, self.ifgcropopt,
187+
input_type=InputTypes.DEM)
181188

182189
# backward compatibility for string paths
183190
for key in self.__dict__:
184191
if isinstance(self.__dict__[key], PurePath):
185192
self.__dict__[key] = str(self.__dict__[key])
186193

187-
def __get_files_from_attr(self, attr):
194+
def __get_files_from_attr(self, attr, input_type=InputTypes.IFG):
188195
val = self.__getattribute__(attr)
189196
files = parse_namelist(val)
190-
return [MultiplePaths(self.outdir, p, self.ifglksx, self.ifgcropopt) for p in files]
197+
return [MultiplePaths(self.outdir, p, self.ifglksx, self.ifgcropopt, input_type=input_type) for p in files]

pyrate/constants.py

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,6 @@
2727
PROCESS = 'process'
2828
MERGE = 'merge'
2929

30-
REF_COLOR_MAP_PATH = os.path.join(PYRATEPATH, "utils", "colourmap.txt")
3130
# distance division factor of 1000 converts to km and is needed to match legacy output
3231
DISTFACT = 1000
3332
# mappings for metadata in header for interferogram

pyrate/conv2tif.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,9 +22,11 @@
2222
from typing import Tuple, List
2323
from joblib import Parallel, delayed
2424
import numpy as np
25+
from pathlib import Path
2526

2627
from pyrate.core.prepifg_helper import PreprocessError
2728
from pyrate.core import shared, mpiops, config as cf, gamma, roipac
29+
from pyrate.core import ifgconstants as ifc
2830
from pyrate.core.logger import pyratelogger as log
2931
from pyrate.configuration import MultiplePaths
3032
from pyrate.core.shared import mpi_vs_multiprocess_logging
@@ -103,7 +105,9 @@ def _geotiff_multiprocessing(unw_path: MultiplePaths, params: dict) -> Tuple[str
103105
header = roipac.roipac_header(unw_path.unwrapped_path, params)
104106
else:
105107
raise PreprocessError('Processor must be ROI_PAC (0) or GAMMA (1)')
108+
header[ifc.INPUT_TYPE] = unw_path.input_type
106109
shared.write_fullres_geotiff(header, unw_path.unwrapped_path, dest, nodata=params[cf.NO_DATA_VALUE])
110+
Path(dest).chmod(0o444) # readonly output
107111
return dest, True
108112
else:
109113
log.warning(f"Full-res geotiff already exists in {dest}! Returning existing geotiff!")

pyrate/core/aps.py

Lines changed: 16 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -19,32 +19,32 @@
1919
signals.
2020
"""
2121
# pylint: disable=invalid-name, too-many-locals, too-many-arguments
22-
import logging
2322
import os
2423
from copy import deepcopy
2524
from collections import OrderedDict
2625
import numpy as np
2726
from numpy import isnan
2827
from scipy.fftpack import fft2, ifft2, fftshift, ifftshift
2928
from scipy.interpolate import griddata
29+
from pyrate.core.logger import pyratelogger as log
3030

3131
from pyrate.core import shared, ifgconstants as ifc, mpiops, config as cf
3232
from pyrate.core.covariance import cvd_from_phase, RDist
3333
from pyrate.core.algorithm import get_epochs
3434
from pyrate.core.shared import Ifg
3535
from pyrate.core.timeseries import time_series
36-
from pyrate.merge import _assemble_tiles
37-
38-
log = logging.getLogger(__name__)
36+
from pyrate.merge import assemble_tiles
3937

4038

4139
def wrap_spatio_temporal_filter(ifg_paths, params, tiles, preread_ifgs):
4240
"""
4341
A wrapper for the spatio-temporal filter so it can be tested.
4442
See docstring for spatio_temporal_filter.
4543
"""
46-
if not params[cf.APSEST]:
47-
log.info('APS correction not required.')
44+
if params[cf.APSEST]:
45+
log.info('Doing APS spatio-temporal filtering')
46+
else:
47+
log.info('APS spatio-temporal filtering not required')
4848
return
4949

5050
# perform some checks on existing ifgs
@@ -101,16 +101,15 @@ def _calc_svd_time_series(ifg_paths, params, preread_ifgs, tiles):
101101
new_params[cf.TIME_SERIES_METHOD] = 2 # use SVD method
102102

103103
process_tiles = mpiops.array_split(tiles)
104-
output_dir = params[cf.TMPDIR]
105104

106105
nvels = None
107106
for t in process_tiles:
108107
log.debug('Calculating time series for tile {} during APS '
109108
'correction'.format(t.index))
110109
ifg_parts = [shared.IfgPart(p, t, preread_ifgs, params) for p in ifg_paths]
111-
mst_tile = np.load(os.path.join(output_dir, 'mst_mat_{}.npy'.format(t.index)))
110+
mst_tile = np.load(os.path.join(params[cf.TMPDIR], 'mst_mat_{}.npy'.format(t.index)))
112111
tsincr = time_series(ifg_parts, new_params, vcmt=None, mst=mst_tile)[0]
113-
np.save(file=os.path.join(output_dir, 'tsincr_aps_{}.npy'.format(t.index)), arr=tsincr)
112+
np.save(file=os.path.join(params[cf.TMPDIR], 'tsincr_aps_{}.npy'.format(t.index)), arr=tsincr)
114113
nvels = tsincr.shape[2]
115114

116115
nvels = mpiops.comm.bcast(nvels, root=0)
@@ -125,11 +124,14 @@ def _assemble_tsincr(ifg_paths, params, preread_ifgs, tiles, nvels):
125124
"""
126125
Helper function to reconstruct time series images from tiles
127126
"""
127+
# pre-allocate dest 3D array
128128
shape = preread_ifgs[ifg_paths[0]].shape + (nvels,)
129129
tsincr_g = np.empty(shape=shape, dtype=np.float32)
130+
# shape of one 2D time-slice array
131+
s = preread_ifgs[ifg_paths[0]].shape
132+
# loop over the time slices and assemble dest 3D array
130133
for i in range(nvels):
131-
for n, t in enumerate(tiles):
132-
_assemble_tiles(i, n, t, tsincr_g[:, :, i], params[cf.TMPDIR], 'tsincr_aps')
134+
tsincr_g[:, :, i] = assemble_tiles(s, params[cf.TMPDIR], tiles, out_type='tsincr_aps', index=i)
133135

134136
return tsincr_g
135137

@@ -184,7 +186,7 @@ def spatial_low_pass_filter(ts_lp, ifg, params):
184186
:return: ts_hp: filtered time series data of shape (ifg.shape, n_epochs)
185187
:rtype: ndarray
186188
"""
187-
log.info('Applying APS spatial low-pass filter')
189+
log.info('Applying spatial low-pass filter')
188190
if params[cf.SLPF_NANFILL] == 0:
189191
ts_lp[np.isnan(ts_lp)] = 0 # need it here for cvd and fft
190192
else:
@@ -281,7 +283,7 @@ def temporal_low_pass_filter(tsincr, epochlist, params):
281283
:return: tsfilt_incr: filtered time series data, shape (ifg.shape, nepochs)
282284
:rtype: ndarray
283285
"""
284-
log.info('Applying APS temporal low-pass filter')
286+
log.info('Applying temporal low-pass filter')
285287
nanmat = ~isnan(tsincr)
286288
tsfilt_incr = np.empty_like(tsincr, dtype=np.float32) * np.nan
287289
intv = np.diff(epochlist.spans) # time interval for the neighboring epoch
@@ -298,7 +300,7 @@ def temporal_low_pass_filter(tsincr, epochlist, params):
298300
func = mean_filter
299301

300302
_tlpfilter(cols, cutoff, nanmat, rows, span, threshold, tsfilt_incr, tsincr, func)
301-
log.debug("Finished applying temporal low pass filter")
303+
log.debug("Finished applying temporal low-pass filter")
302304
return tsfilt_incr
303305

304306
# Throwaway function to define Gaussian filter weights

0 commit comments

Comments
 (0)