Where are all the ocean variables?

[rabernat]

I started to look at the LENS AWS data. I discovered there is very little available

import intake_esm
import intake
url = 'https://raw.githubusercontent.com/NCAR/cesm-lens-aws/master/intake-catalogs/aws-cesm1-le.json'
col = intake.open_esm_datastore(url)
col.search(component='ocn').df

	component	frequency	experiment	variable	path
0	ocn	monthly	20C	SALT	s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-SAL...
1	ocn	monthly	20C	SSH	s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-SSH...
2	ocn	monthly	20C	SST	s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-SST...
3	ocn	monthly	CTRL	SALT	s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-SA...
4	ocn	monthly	CTRL	SSH	s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-SS...
5	ocn	monthly	CTRL	SST	s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-SS...
6	ocn	monthly	RCP85	SALT	s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-S...
7	ocn	monthly	RCP85	SSH	s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-S...
8	ocn	monthly	RCP85	SST	s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-S...

There are only 3 variables: SALT (3D), SSH (2D), and SST (2D).

At minimum, I would also like to have THETA (3D), UVEL (3D), VVEL (3D), and WVEL (3D), and all the surface fluxes of heat and freshwater. Beyond that, it would be ideal to also have the necessary variables to reconstruct the tracer and momentum budgets.

Are there plans to add more data?

[jeffdlb]

Ryan-

You are correct there is not much ocean data there. The plan was to add additional data on request, so thanks for your request. We do have room in our AWS allocation to add more, and I would be glad for us to do so. I will discuss this with some team members at our meeting on Thursday. Besides the THETA, UVEL, VVEL, WVEL fields you mentioned, can you indicate which specific other variables you desire? Gary Strand's list of variable names is at http://www.cgd.ucar.edu/ccr/strandwg/CESM-CAM5-BGC_LENS_fields.html

[rabernat]

Thanks @jeffdlb! I will review the list of fields and get back to you. Our general interest is ocean heat and salt budgets.

[jeffdlb]

PS. We currently only have monthly ocean data, whereas the other realms also have some daily or 6-hour data. Is monthly sufficient for your use case?

[rabernat]

For coarse-resolution (non eddy-resolving) models, the oceans tend not to have too much sub-monthly variability. If we did need daily data, it would just be surface fluxes. Monthly should be fine for the other stuff.

[rabernat]

Ok, here is my best guess at identifying the variables we would need for the heat and salt budgets. Would be good for someone with more POP experience (e.g. @matt-long) to verify.

THETA
UVEL
UVEL2
VVEL
VVEL2
WVEL
FW
HDIFB_SALT
HDIFB_TEMP
HDIFE_SALT
HDIFE_TEMP
HDIFN_SALT
HDIFN_TEMP
HMXL
HOR_DIFF
KAPPA_ISOP
KAPPA_THIC
KPP_SRC_SALT
KPP_SRC_TEMP
RESID_S
RESID_T
QFLUX
SHF
SHF_QSW
SFWF
SFWF_WRST
SSH
TAUX
TAUX2
TAUY
TAUY2
VNT_ISOP
VNT_SUBM
UES
UET
VNS
VNT
WTS
WTT

[jeffdlb]

Ryan- Many of these variables do not seem present in the monthly ocean data on GLADE: 1310> pwd /glade/collections/cdg/data/cesmLE/CESM-CAM5-BGC-LE/ocn/proc/tseries 1311> while read ln; do ls -d "monthly/$ln"; done < ~/oceanVars.txt ls: cannot access monthly/THETA: No such file or directory monthly/UVEL ls: cannot access monthly/UVEL2: No such file or directory monthly/VVEL ls: cannot access monthly/VVEL2: No such file or directory monthly/WVEL ls: cannot access monthly/FW: No such file or directory ls: cannot access monthly/HDIFB_SALT: No such file or directory ls: cannot access monthly/HDIFB_TEMP: No such file or directory ls: cannot access monthly/HDIFE_SALT: No such file or directory ls: cannot access monthly/HDIFE_TEMP: No such file or directory ls: cannot access monthly/HDIFN_SALT: No such file or directory ls: cannot access monthly/HDIFN_TEMP: No such file or directory monthly/HMXL ls: cannot access monthly/HOR_DIFF: No such file or directory ls: cannot access monthly/KAPPA_ISOP: No such file or directory ls: cannot access monthly/KAPPA_THIC: No such file or directory ls: cannot access monthly/KPP_SRC_SALT: No such file or directory ls: cannot access monthly/KPP_SRC_TEMP: No such file or directory ls: cannot access monthly/RESID_S: No such file or directory ls: cannot access monthly/RESID_T: No such file or directory monthly/QFLUX monthly/SHF monthly/SHF_QSW monthly/SFWF ls: cannot access monthly/SFWF_WRST: No such file or directory monthly/SSH monthly/TAUX monthly/TAUX2 monthly/TAUY monthly/TAUY2 ls: cannot access monthly/VNT_ISOP: No such file or directory ls: cannot access monthly/VNT_SUBM: No such file or directory monthly/UES ls: cannot access monthly/UET: No such file or directory monthly/VNS ls: cannot access monthly/VNT: No such file or directory monthly/WTS ls: cannot access monthly/WTT: No such file or directory Jeff de La Beaujardiere, PhD Director, NCAR/CISL Information Systems Division https://staff.ucar.edu/users/jeffdlb https://orcid.org/0000-0002-1001-9210

…

On Wed, Feb 12, 2020 at 1:46 PM Ryan Abernathey ***@***.***> wrote: Ok, here is my best guess at identifying the variables we would need for the heat and salt budgets. Would be good for someone with more POP experience (e.g. @matt-long <https://github.com/matt-long>) to verify. THETA UVEL UVEL2 VVEL VVEL2 WVEL FW HDIFB_SALT HDIFB_TEMP HDIFE_SALT HDIFE_TEMP HDIFN_SALT HDIFN_TEMP HMXL HOR_DIFF KAPPA_ISOP KAPPA_THIC KPP_SRC_SALT KPP_SRC_TEMP RESID_S RESID_T QFLUX SHF SHF_QSW SFWF SFWF_WRST SSH TAUX TAUX2 TAUY TAUY2 VNT_ISOP VNT_SUBM UES UET VNS VNT WTS WTT — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#34?email_source=notifications&email_token=ABF4W4V7YH2OHFOZ3IW4DLTRCRNY3A5CNFSM4KPJYHB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELSKIBY#issuecomment-585409543>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABF4W4TKKWSQ4WIUUTK5N7TRCRNY3ANCNFSM4KPJYHBQ> .

[matt-long]

TEMP is the POP variable name for potential temperature (not THETA).

Many of the data not available on glade are available on HPSS
[HSI]: /CCSM/csm/CESM-CAM5-BGC-LE/ocn/proc/tseries/monthly

And (possibly) on NCAR Campaign storage here
/glade/campaign/cesm/collections/cesmLE
(accessible on Casper)

Note that to close a tracer budget you need
Advection
UE{S,T}
VN{S,T}
WT{S,T}

Lateral diffusion (GM, submeso)
HDIF{E,N,B}_{S,T}

Vertical mixing
KPP_SRC_{SALT,TEMP}
DIA_IMPVF_{SALT,TEMP} # implicit diabatic vertical mixing, I think you missed this one

Surface fluxes # some choices
SHF_QSW # I don't think we save fully 3D QSW, unfortunately
QSW_HTP # top layer
QSW_HBL # boundary layer
SHF
QFLUX
SFWF

Inventory
TEMP
SALT
SSH # free-surface deviations impact the tracer mass in top layer, where dz = dz + SSH

[bonnland]

According to Gary Strand, all of the LENS data is available on Glade here:

/glade/collections/cdg/data/cesmLE/CESM-CAM5-BGC-LE

When I look at what is available for the ocean, I find this:

casper26:/glade/collections/cdg/data/cesmLE/CESM-CAM5-BGC-LE/ocn/proc/tseries$ ls annual/
DIA_IMPVF_DIC/		HDIFB_DOC/	    HDIFE_O2/		J_Fe/		      KPP_SRC_Fe/      VN_DIC/		WT_DOC/
DIA_IMPVF_DIC_ALT_CO2/	HDIFB_Fe/	    HDIFN_DIC/		J_NH4/		      KPP_SRC_O2/      VN_DIC_ALT_CO2/	WT_Fe/
DIA_IMPVF_DOC/		HDIFB_O2/	    HDIFN_DIC_ALT_CO2/	J_NO3/		      UE_DIC/	       VN_DOC/		WT_O2/
DIA_IMPVF_Fe/		HDIFE_DIC/	    HDIFN_DOC/		J_PO4/		      UE_DIC_ALT_CO2/  VN_Fe/
DIA_IMPVF_O2/		HDIFE_DIC_ALT_CO2/  HDIFN_Fe/		J_SiO3/		      UE_DOC/	       VN_O2/
HDIFB_DIC/		HDIFE_DOC/	    HDIFN_O2/		KPP_SRC_DIC/	      UE_Fe/	       WT_DIC/
HDIFB_DIC_ALT_CO2/	HDIFE_Fe/	    J_ALK/		KPP_SRC_DIC_ALT_CO2/  UE_O2/	       WT_DIC_ALT_CO2/

casper26:/glade/collections/cdg/data/cesmLE/CESM-CAM5-BGC-LE/ocn/proc/tseries$ ls monthly
ALK/		   DIC_ALT_CO2/       get	      MELTH_F/	       photoC_diat/  SENH_F/	 TAUY/			      VISOP/
ATM_CO2/	   DOC/		      HBLT/	      MOC/	       photoC_diaz/  SFWF/	 TAUY2/			      VNS/
BSF/		   DpCO2/	      HMXL/	      N_HEAT/	       photoC_sp/    SHF/	 TBLT/			      VVEL/
CFC11/		   DpCO2_ALT_CO2/     IAGE/	      N_SALT/	       PREC_F/	     SHF_QSW/	 TEMP/			      WISOP/
CFC_ATM_PRESS/	   ECOSYS_ATM_PRESS/  IFRAC/	      O2/	       QFLUX/	     SNOW_F/	 tend_zint_100m_ALK/	      WTS/
CFC_IFRAC/	   ECOSYS_IFRAC/      INT_DEPTH/      O2_CONSUMPTION/  QSW_HBL/      spCaCO3/	 tend_zint_100m_DIC/	      WVEL/
CFC_XKW/	   ECOSYS_XKW/	      IOFF_F/	      O2_PRODUCTION/   QSW_HTP/      spChl/	 tend_zint_100m_DIC_ALT_CO2/  XBLT/
CO2STAR/	   EVAP_F/	      Jint_100m_ALK/  O2SAT/	       RHO/	     SSH/	 tend_zint_100m_DOC/	      XMXL/
DCO2STAR/	   FG_ALT_CO2/	      Jint_100m_DIC/  O2_ZMIN/	       RHO_VINT/     SSH2/	 tend_zint_100m_O2/	      zsatarag/
DCO2STAR_ALT_CO2/  FG_CO2/	      Jint_100m_DOC/  O2_ZMIN_DEPTH/   ROFF_F/	     SST/	 TLT/			      zsatcalc/
DIA_DEPTH/	   FvICE_ALK/	      Jint_100m_O2/   pCO2SURF/        SALT/	     STF_CFC11/  TMXL/
diatChl/	   FvICE_DIC/	      LWDN_F/	      PD/	       SALT_F/	     STF_O2/	 UES/
diazChl/	   FvPER_ALK/	      LWUP_F/	      PH/	       SCHMIDT_CO2/  TAUX/	 UISOP/
DIC/		   FvPER_DIC/	      MELT_F/	      PH_ALT_CO2/      SCHMIDT_O2/   TAUX2/	 UVEL/

casper26:/glade/collections/cdg/data/cesmLE/CESM-CAM5-BGC-LE/ocn/proc/tseries$ ls daily
CaCO3_form_zint/  diazC_zint_100m/  ECOSYS_XKW_2/  nday1/	      spCaCO3_zint_100m/  SST/	     TAUY_2/	zooC_zint_100m/
diatChl_SURF/	  DpCO2_2/	    FG_CO2_2/	   photoC_diat_zint/  spChl_SURF/	  SST2/      WVEL_50m/
diatC_zint_100m/  ecosys/	    HBLT_2/	   photoC_diaz_zint/  spC_zint_100m/	  STF_O2_2/  XBLT_2/
diazChl_SURF/	  ECOSYS_IFRAC_2/   HMXL_2/	   photoC_sp_zint/    SSH_2/		  TAUX_2/    XMXL_2/

casper26:/glade/collections/cdg/data/cesmLE/CESM-CAM5-BGC-LE/ocn/proc/tseries$ 

If someone can help decipher these variables and determine if they are worth publishing, I would be happy to work on getting them onto AWS.

[bonnland]

@jeffdlb @rabernat On second glance, it appears that my directory listing above shows that some variables are missing. I'm waiting to hear from Gary to get some clarification.

[rabernat]

@bonnland -- were you the one who produced the original S3 LENS datasets? If so, it would be nice to build on that effort. My impression from @jeffdlb is that they have a pipeline set up, they just need to find the data! Maybe you were part of that...sorry for my ignorance.

As for the missing variables, I guess I would just request that you take the intersection between my requested list and what is actually available. I think that the list of monthly and daily variables you showed above is a great start. I would use nearly all of it.

[bonnland]

@rabernat I just got word from Gary; I was originally in a slightly different folder. I have the correct folder path now, and all 273 monthly ocean variables appear to be present.

I was part of the original data publishing, so I know parts of the workflow. The most time consuming part is creating the CSV file describing an intake-esm catalog, which I did not originally take part in. The catalog is used to load data into xarray and then write out to Zarr. We have the file paths now; I just need to research how to construct the remaining fields for the CSV file.

[bonnland]

@rabernat I've loaded some variables, and the datasets are big. A single variable will take over 2TB. Here are some stats for five of the variables:

Note that these sizes are uncompressed sizes, and they will be smaller on disk.

Is there a priority ordering that makes sense if we can initially publish just a subset? Anderson believes that if the available space on AWS has not changed, we have around 30 TB available.

@jeffdlb Do you know more exactly how much space is left on S3, and when we might get more?

[jeffdlb]

Brian-

On Thu, Mar 5, 2020 at 5:06 PM bonnland ***@***.***> wrote: @jeffdlb <https://github.com/jeffdlb> Do you know more exactly how much space is left on S3, and when we might get more?

We are currently using 61.5 TB for 905023 objects in ncar-cesm-lens bucket, the vast majority of which is for atmosphere data. Ocean data only use 1.2 TB at present. I don't think there is an automatically-enforced limit to the allocation, so nothing will prevent writing objects after 100 TB. However, we should as a courtesy notify AWS if we plan to go over. They have already said we can use more if needed, within reason. Is 20.95TB the total for all the new ocean variables, or only for a subset? If subset, can you estimate the total (uncompressed) for all the new vars?

…

-Jeff Jeff de La Beaujardiere, PhD Director, NCAR/CISL Information Systems Division https://staff.ucar.edu/users/jeffdlb https://orcid.org/0000-0002-1001-9210

[andersy005]

Is 20.95TB the total for all the new ocean variables, or only for a subset?
If subset, can you estimate the total (uncompressed) for all the new vars?

The 20.95 TB is for only 5 variables.

Of the 39 variables listed in #34 (comment), we found 38 variables. A back of the envelope calculation shows that their total uncompressed size would be ~ 170TB.

[rabernat]

Do we have any idea what typical zarr + zlib compression rates are for these datasets? I would not be surprised to see a factor for 2 or more.