185 docs

docs/index.md

@@ -19,9 +19,12 @@ convient GUI.
 - [Quick Start Installation :rocket:](#quick-start-installation-)
 - [How to use pyBDY :mechanical_arm:](#how-to-use-pybdy-)
 - [Function List :scroll:](#function-list-)
+-
 
 ## Dependencies :globe_with_meridians:
 
+[Back to top](#pybdy-documentation)
+
 pyBDY is installed under a conda/mamba environment to aid wider distribution and to facilitate development.
 The key dependecies are listed below:
 
@@ -42,6 +45,8 @@ A recent JAVA installation is also required.
 
 ## Quick Start Installation :rocket:
 
+[Back to top](#pybdy-documentation)
+
 To get started, check out and set up an instance of the pyBDY GitHub [repository](https://github.com/NOC-MSM/pyBDY):
 
 ```sh
@@ -111,6 +116,8 @@ conda deactivate
 
 ## How to use pyBDY :mechanical_arm:
 
+[Back to top](#pybdy-documentation)
+
 In this documentation "bdy points" refer to the output boundary points generated by pyBDY.
 Fist follow the installation instructions [Quick Start Installation](#quick-start-installation-).
 
@@ -277,8 +284,12 @@ This command line tool reads a BDY file, extracts boundary data and prepares the
 
 ## Function List :scroll:
 
+[Back to top](#pybdy-documentation)
+
 ## Tidal Boundary Conditions Generation
 
+[Back to top](#pybdy-documentation)
+
 By providing a global tidal model dataset (TPXO and FES are currently supported) pyBDY can generate boundary conditions for the NEMO configuration supplied using the namelist file.
 
 ### Namelist options
@@ -354,6 +365,105 @@ Phase Exceedance = 5.052 * pyBDY Amplitude ^ -0.60
 
 ## CMEMS Downloader
 
+[Back to top](#pybdy-documentation)
+
+IMPORTANT The CMEMS downloader has not been tested recently (May 2025).
+IMPORTANT The CMEMS downloader has only been tested with the GLOBAL_ANALYSIS_FORECAST_PHY_001_024 model and specifcally the hourly SSH and U V product. This also has temperature stored within it, but not salinity. Other models and products should work but are currently likely to need some changes to the code to cope with different variable names within the data. This will be fixed in a later release of pyBDY that is able to handle different variable and tracer names.
+
+pyBDY has a CMEMS downloading function incorporated within it, this will download a section of the CMEMS global model (more models to be added) ‘GLOBAL_ANALYSIS_FORECAST_PHY_001_024-TDS’ for the defined time period in the namelist file
+
+To use the downloading function, the following command is used:
+
+```
+$ pynemo -d namelist.bdy
+```
+
+Where the -d flag tells pyBDY to use the CMEMS downloader and download data as specified in the namelist file. The log file that pyBDY produces provides a log of what the downloader does. The CMEMS MOTU system is prone to disconnects and failure so there is download retry and error handling built in. Most of the options required should not need editing and are there for future use in case URL’s and filenames on CMEMS change.
+
+The options that can be configured are described in further detail below:
+
+```
+!------------------------------------------------------------------------------
+!  I/O
+!------------------------------------------------------------------------------
+   sn_src_dir = '/Users/thopri/Projects/pyBDY/inputs/CMEMS.ncml' ! src_files/'
+   sn_dst_dir = '/Users/thopri/Projects/pyBDY/outputs'
+
+   sn_fn      = 'NNA_R12'             ! prefix for output files
+   nn_fv      = -1e20                 !  set fill value for output files
+   nn_src_time_adj = 0                ! src time adjustment
+   sn_dst_metainfo = 'CMEMS example'
+
+!------------------------------------------------------------------------------
+!  CMEMS Data Source Configuration
+!------------------------------------------------------------------------------
+   ln_use_cmems             = .true.
+   ln_download_cmems        = .true.
+   sn_cmems_dir             = '/Users/thopri/Projects/pyBDY/inputs/' ! where to download CMEMS input files (static and variable)
+   ln_download_static       = .true.
+   ln_subset_static         = .true.
+   nn_num_retry             = 4 ! how many times to retry CMEMS download after non critical errors?
+!------------------------------------------------------------------------------
+!  CMEMS MOTU Configuration (for Boundary Data)
+!------------------------------------------------------------------------------
+   sn_motu_server           = 'http://nrt.cmems-du.eu/motu-web/Motu'
+   sn_cmems_config_template = '/Users/thopri/Projects/pyBDY/pynemo/config/motu_config_template.ini'
+   sn_cmems_config          = '/Users/thopri/Projects/pyBDY/pynemo/config/motu_config.ini'
+   sn_cmems_model           = 'GLOBAL_ANALYSIS_FORECAST_PHY_001_024-TDS'
+   sn_cmems_product         = 'global-analysis-forecast-phy-001-024'
+   sn_dl_prefix             = 'subset'
+!------------------------------------------------------------------------------
+!  CMEMS FTP Configuration (for Static Files)
+!------------------------------------------------------------------------------
+    sn_ftp_server            = 'nrt.cmems-du.eu'
+    sn_static_dir            = '/Core/GLOBAL_ANALYSIS_FORECAST_PHY_001_024/global-analysis-forecast-phy-001-024-statics'
+    sn_static_filenames      = 'GLO-MFC_001_024_coordinates.nc GLO-MFC_001_024_mask_bathy.nc GLO-MFC_001_024_mdt.nc'
+    sn_cdo_loc               = '/opt/local/bin/cdo' ! location of cdo executable can be found by running "where cdo"
+!------------------------------------------------------------------------------
+!  CMEMS Extent Configuration
+!------------------------------------------------------------------------------
+    nn_latitude_min          = 40
+    nn_latitude_max          = 66
+    nn_longitude_min         = -22
+    nn_longitude_max         = 16
+    nn_depth_min             = 0.493
+    nn_depth_max             = 5727.918000000001
+```
+
+Some of the options define the behaviour of the downloader, others define locations to save files and others detail models and grid files to download. Finally the spatial extent to download is also required.
+
+#### I/O and NCML file
+
+The location of the NCML file is listed a string defining the source directory or “sn_src_dir”. The output folder is also defined here as “sn_dst_dir”, NOTE if this directory does not exist it will need to be created and permissoned correctly for pyBDY to run properly. The NCML file details the input files to agregate and what the variable names are. Variable names are in [https://resources.marine.copernicus.eu/?option=com_csw&task=results](https://resources.marine.copernicus.eu/?option=com_csw&task=results) For more information please read the ncml part.
+
+NOTE A NCML file must be used and it also must use a regular expression. The CMEMS downloader uses this regular expression to determine what grid a given variable is part of e.g. temperature and salinity on the T grid. The example CMEMS.ncml file includes: an implementation of how to define temperature, SSH and U and V components of ocean currents.
+
+Firstly, the string “sn_fn” defines the prefix for the output files. The number “nn_fv” defines the fill value, and the number “nn_src_time_adj” defines the source time adjustment. The rest of the boxes are CMEMS specific.
+
+#### Data Source Configuration
+
+The first section defines the CMESM data source configuration. The boolean “ln_use_cmems” when set to true will use the CMEMS downloader function to download the requested data, this is defined in the ncml file which can be generated using the NCML generator. Among other things this file defines what data variables to download. This term also changes the variable names to CMEMS specific ones e.g. thetao for temperature and so for salinity. This is in contrast to the NEMO specific ones such as Votemper and Vosaline. When set to false no download occurs and variable names are kept to NEMO specific.
+
+#### MOTU Configuration
+
+In the next section when set to true “ln_download_cmems” will download the boundary tracer data, e.g. time series of temperature and saliniy. When set to false pyBDY will skip this download. The string “sn_cmems_dir” defines where to save these downloaded files. pyBDY requires grid data, this isn’t possible to download using the same method as the tracer data which uses the MOTU python client. To get the grid data, an ftp request is made to download the global grids which are then subset to the relevent size. The booleans “ln_downlad_static” and “ln_subset_static” determine this behavior. Finally there is an int named “nn_num_retry” this defines the number of times to retry downloading the CMEMS data. The data connections are prone to failure so if a non critical error occurs the function will automatically try to redownload. This int defines how many times it will try to do this. Typically this static data and subsetting are only required once so these can be set to true for first download and then set to false when more time series data is required.
+
+As mentioned previously, the time series boundary data is downloaded using MOTU, this is an efficent and robust web server that handles, extracts and transforms oceanographic data. By populating a configuration file, this can be sent to the MOTU server which will return the requested data in the requested format. The section CMEMS MOTU configuration sets this up. Most of these options should not need changing. The location of the MOTU server for CMEMS is defined here, and the location of the config template file and also the location of the config file to submit. The only options that should require changing are the model, product and prefix options. These define which CMEMS model and product to download and the prefix is a user defined string to prefix the downloads. A catalogue of the CMEMS model and products can be found at https://resources.marine.copernicus.eu/?option=com_csw&task=results Currently pyBDY has only been tested using the physical global forecast model although the downloader should be able to download other models and products, it has not been tested and their are known issues with other products that restrict seamless download. e.g. the NorthWest Atlantic model is not currently compatible due to differences in how the model variables are stored.
+
+#### FTP Configuration for Static and Grid files
+
+The next section CMEMS FTP configuration, defines which FTP server, remote directory and files to download. This should require modification unless CMEMS changes the file structure or names. Note it is important that the filenames are separated by a space as this is what pyBDY is expecting. Finally the location of CDO executable which should be installed to enable subsetting to occur. This can be found by running:
+
+```
+$ where cdo
+```
+
+#### Extent configuration
+
+Finally the last box, this is where the extent to download is configured, it is up to the user to decide but it is suggested this is at least 1 degree wider than the destination or child configuration. The depth range to request is also defined here. This information can be extracted from the CMEMS catalogue. Once set for a given configuration this will not need to be edited.
+
 ## Troubleshooting
 
+[Back to top](#pybdy-documentation)
+
 Always check the pyBDY log file. This is usually saved in the working directory of pyBDY as nrct.log. It gives helpful information which may help to diagnose issues. E.g. ValueErrors that are result of a THREDDS server being down and unable to provide data files.