Remove and or mask outer rows and columns from all data being read in

[matthew-mizielinski]

Migrated Jira Issue CDDSO-73
Jira creation date 21/04/2022 15:24

MIP Convert now has options to mask and cut off rows and columns from data before writing out, see the section on masking and the halo_removal section after

If we remove the halo removal from extract and get configure to provide the relevant information in the mip convert config files we can
(a) minimise load on MASS during extraction of data
(b) enable easier use of CDDS on JASMIN

In Extract this would mean removing these lines and adding code to the get_halo_removal_attributes() and get_masking_attribute() functions in configure to default to the information in the plugins (as used by extract).

We need to be confident that this does change makes no difference to the output for ocean & seaice data at multiple model resolutions.

Updates would be needed to the model config files within the CDDS plugins

[matthew-mizielinski]

Testing with the variable zostoga has shown that it is tricky to combine ancillary data with haloes removed and model data with haloes included -- identifying exactly when haloes need to be removed and when they don't isn't immediately straightforward.

An option that I've looked at is adding a global attribute named haloes_removed to ancillary files and then only removing haloes if that attribute doesn't exist. This seems to work provided the inner loop in _apply_removal() looks something like

        for key, cube in self.input_variables.items():
                # print(f"Processing cube: {cube}")
                
                new_cube = cube.copy()
                if 'haloes_removed' in new_cube.attributes:
                    logger.debug(f'{new_cube} has the "haloes_removed" attribute, skipping halo removal')
                else:
                    if cube.coords("latitude"):
                        new_cube = self._remove_latitude_halo(new_cube, stream_removal)
                    if cube.coords("longitude"):
                        new_cube = self._remove_longitude_halo(new_cube, stream_removal)

                self.input_variables[key] = new_cube

For testing I've added $CDDS_ETC/ancil_testing on disk where I've added the "haloes_removed" attribute to each UKESM1/HadGEM3 LL ocean ancillary file.