-
Notifications
You must be signed in to change notification settings - Fork 0
01. Create coordinates, bathymetry and domain files
Welcome to the UK500 wiki!
git clone https://github.com/micdom/UK500
cd UK500
cd SCRIPTS/
. ./make_paths.sh
. ./make_directories.sh
. ./make_xios.sh
. ./make_nemo.sh
. ./make_tools_wad.sh
cp /work/n01/n01/micdom/SEVERN-SWOT/DOWNLOADS/coordinates_AMM15.nc ../DOWNLOADS/
vi ../BUILD_CFG/DOMAIN/namelist.input_UK500
&nesting
imin = 570
imax = 1125
jmin = 220
jmax = 1200
rho = 3
rhot = 3
bathy_update = false
/
Then run
. ./make_coordinates_from_parent.sh
File created in /work/n01/n01/micdom/UK500/BUILD_CFG/DOMAIN
NOTE:
I had to comment some lines in
/work/n01/n01/micdom/UK500/BUILD_EXE/NEMO/4.0.6/tools/NESTING/src/agrif_create_coordinates.f90
because it was giving segmentation fault
I changed the creation of the netcdf to NETCDF4 in
/work/n01/n01/micdom/UK500/BUILD_EXE/NEMO/4.0.6/tools/NESTING/src/agrif_interpolation.f90
/work/n01/n01/micdom/UK500/BUILD_EXE/NEMO/4.0.6/tools/NESTING/src/agrif_readwrite.f90
but it does not seem to solve the segmentation fault in this case, so the hacked agrif_create_coordinates.f90 file is still needed.
cp /work/n01/n01/micdom/UK500_working/BUILD_EXE/NEMO/4.0.6/tools/NESTING/src/agrif_create_coordinates.f90 /work/n01/n01/micdom/UK500/BUILD_EXE/NEMO/4.0.6/tools/NESTING/src/agrif_create_coordinates.f90
cp /work/n01/n01/micdom/UK500_working/BUILD_EXE/NEMO/4.0.6/tools/NESTING/src/agrif_interpolation.f90 /work/n01/n01/micdom/UK500/BUILD_EXE/NEMO/4.0.6/tools/NESTING/src/agrif_interpolation.f90
cp /work/n01/n01/micdom/UK500_working/BUILD_EXE/NEMO/4.0.6/tools/NESTING/src/agrif_readwrite.f90 /work/n01/n01/micdom/UK500/BUILD_EXE/NEMO/4.0.6/tools/NESTING/src/agrif_readwrite.f90
Recompile
. ./make_tools_wad.sh
(say no to everything)
The hacked agrif_create_coordinates.f90 might have an impact on nav_lon, nav_lat in the the coordinates.nc file, but that does not seem to be an issue.
Script is /projectsa/CHAMFER/EMODNET/interpo_EMODnet_bathy_UK_wide_v2.m I have downloaded the 6 EMODNET tiles covering the UK domain from the EMODNET website, I've chosen the ascii file referenced to MSL (EMODNET does now the conversion from LAT to MSL): E5_MSL_2020.asc,D5_MSL_2020.asc, E4_MSL_2020.asc,D4_MSL_2020.asc, E3_MSL_2020.asc, D3_MSL_2020.asc
I've stitched the tiles together using Matlab (there are 4 points of overlap in both directions in the tiles!), the bathymetry is negative at sea, positive on land, nan/9999 where undefined.
I've interpolated the bathymetry using griddedInterpolant function.
The bathymetry has been smoothed with a Shapiro filter.
Land points >10 m (not many) have been lowered to 10 m.
The land/sea mask is built from the EMODNET bathymetry: I have assigned NAN where EMODNET bathymetry has a missing values (i.e. land) and interpolated (bi-linear) (if one of the 4 values used for interpolation is nan then is land)*.
Isolated wet cells, unconnected narrow channels and lakes have been removed using the remove_lake function available from the https://github.com/NOAA-EMC/gridgen software package, created for generating grids for WAVEWATCH III.
I've saved the bathymetry in a netcdf (bathy_meter9999.nc) and transferred to ARCHER2:
scp bathy_meter9999.nc $USER@login.archer2.ac.uk:/work/n01/n01/$USER/UK500/BUILD_CFG/DOMAIN
On ARCHER2, in $DOMAIN I changed the bathy to positive at sea, negative on land, and -9999 where undefined.
module load load-epcc-module
module load nco/5.1.6
ncflint --fix_rec_crd -w -1.0,0.0 bathy_meter9999.nc bathy_meter9999.nc tmp.nc
mv tmp.nc bathy_meter.nc
NOTE: I've tried to use the AGRIF tools to produce the bathymetry, but without success. The bathymetry file was too big, AGRIF scripts can be run only in serial and it was exceeding the permitted time on ARCHER2. I did some changes to the AGRIF subroutines to be able to handle big netcdf files.
I've changed all the nf_create and nf90_create to generate NETCDF4.
/work/n01/n01/micdom/UK500/BUILD_EXE/NEMO/4.0.6/tools/WEIGHTS/src
Scripts modified are:
scripinterp_mod.F90
scripgrid_mod.F90
scripshape.F90.orig
remap_write.f90
scripinterp_mod.F90.orig
scripshape.F90
The AGRIF tools are were run on the serial queue, from SCRIPTS:
sbatch submitagriftools_emodnet.slurm
Finally generate the domain_cfg.nc file using the coordinates.nc, bathy_meter.nc. This is done in the tools directory using make_domain_cfg.sh script in the SCRIPTS directory. The tools used are the DOMAINcfg and REBUILD_NEMO tools. The DOMAINcfg tool is submitted as a slurm job.
If making changes to the configuration, these are controlled by the namelist_cfg file. In this configuration, it is $DOMAIN/"$REPO"_s-sig_DOMAINcfg_namelist_cfg, as per $WDIR/SCRIPTS/make_domain_cfg.sh. Ensure this has the appropriate parameters and number of lat,lon,depth levels etc set.
For example the lateral size of the domain can be extracted from bathy_meter.nc using ncdump -h bathy_meter.nc, then namelist can be edited accordingly e.g.: for the UK500 domain
vi namelist_cfg
...
jpidta = 1668 ! 1st lateral dimension ( >= jpi )
jpjdta = 2943 ! 2nd " " ( >= jpj )
jpkdta = 31 ! number of levels ( >= jpk )
...
The model is sigma coordinates only:
rn_rmax = 0.0 ! maximum cut-off r-value allowed (0<r_max<1)
When ready, run:
cd SCRIPTS
. ./make_domain_cfg.sh
For the UK500 domain this script runs the domain decomposition in parallel on 128 processors, the tiles are then rebuilt together to create domain_cfg.nc, last step is to copy it in $DOMAIN.
In this experiment an unforced ocean is initialised from rest. It is initialised with constant density. Any velocities that are generated are the result of model errors. These can be either i) genuine code bugs (hopefully zero), or ii) numerical errors. This can be a useful check that things are working as expected.
The model can be initialised with constant density either by prescribing it as an initial condition, or by compiling it into the code as an analytic function. Here we do the latter.
export CONFIG=NEMOconstTS # NEMO exec with hardwired constant T,S
It will select and compile with a prescribed stratification given in $NEMO/cfgs/$CONFIG/MY_SRC/usrdef_istate.F90. This has been done already during compilation.
Go to the experiment folder cd /RUN_DIRECTORIES/EXP_unforced_constTS
Verify that the boundary condition in namelist_cfg is false
vi namelist_cfg
ln_bdy=.false.
Run the experiment from SCRIPTS folder
. ./run_unforced_constTS.sh
NB the run_unforced_constTS.sh script assumes you can submit jobs to the n01-CHAMFER ARCHER2 account. Edit submit.slurm accordingly.
Be efficient with HPC resource: see notes on processor decomposition