Using NetCDF file attributes to filter dataset in ERDDAP

[pkraska]

Hello,

I'm not sure this is the appropriate forum for this question, but I have not found an answer for this yet.

I am working with a group of researchers looking to compile nearshore temperature logger data into a single dataset on our local ERDDAP server. The data is converted into NetCDF files so that the deployment level metadata (Lat/Long/Depth/Project/Location Name/Logger ID) is retained in each file as a global attribute without being repeated for each row of the timeseries.

My question is this: Is there a way to filter the data using these file attributes (i.e. only select records from a particular logger ID or location name) without inflating file sizes by creating a variable from them and having it repeat for the entirety of the dataset? The simplest solution is to keep the data as a CSV and repeat information, but I guess I'm looking for a more elegant approach to not artificially inflate datasets.

Any help would be appreciated. Thanks for your time and consideration!

[ChrisJohnNOAA]

I believe you want to take a look at the documentation for Pseudo source names:
https://erddap.github.io/docs/server-admin/datasets#pseudo-sourcenames

Specifically what you describe sounds like global: sourceNames to me, but there are several options there to use.

Your data files can specify the information as attributes (not as a column), but ERDDAP will treat them like a column, and so users will be able to filter on the fields.

[pkraska]

Thanks for the link, @ocefpaf. (Un)fortunately I use R (but am learning Python when I can), I'll have a read through as it seems promising for some workflows!

@ChrisJohnNOAA , I assume this is me not being very familiar with editing the datasets.xml file, but any tips on how to format the pseudo source name as this isn't working for me?

<dataVariable>
        <sourceName>global:project_lead</sourceName>
        <dataType>String</dataType>
</dataVariable>

[ChrisJohnNOAA]

You may need to add the ioos_category attribute (depending on your settings it may be required for all variables. For an example from one of our test datasets:

<dataVariable>
  <sourceName>global:wmo_platform_code</sourceName>
  <destinationName>wmo_platform_code</destinationName>
  <dataType>int</dataType>
  <addAttributes>
    <att name=\"ioos_category\">Identifier</att>
    <att name=\"long_name\">WMO Platform Code</att>
    <att name=\"missing_value\" type=\"int\">2147483647</att>
  </addAttributes>
</dataVariable>

[pkraska]

Thanks for the help @ChrisJohnNOAA, that did it. In case it helps anyone in the future (and to remind me what I did next time):

the global attribute:

// global attributes:
                :project_lead = "Someone" ;

and the datasets.xml data variable chunk for that global attribute to be viewed as a variable:

<dataVariable>
            <sourceName>global:project_lead</sourceName>
            <destinationName>project_lead</destinationName>
            <dataType>String</dataType>
            <addAttributes>
                <att name="ioos_category">Identifier</att>
            </addAttributes>
</dataVariable>

Thanks everyone!

[rmendels]

@pkraska Can you point to the URL of the dataset, I knew you could do this but I think this is the first time I have ever seen it done.

[pkraska]

@rmendels Unfortunately, this is a bit of a proof of concept/stop-gap measure until our department gets it's ERDDAP server up and running (I really hope that's soon, but we'll see), so it's running on our internal network with the aim to shift the datasets and associated xml dataset files when the time comes.

If you'd like I can send you a few .nc files and the dataset xml for you to try out?

[rmendels]

@pkraska For my benefit could you post the netcdf header info from a file - that is what you would get from "ncdump -h" Also perhaps what you want is to set the data as a table, not a grid, using EDDTableFromNcFiles. You do not have to convert the files to a table, ERDDAP will.

[pkraska]

The header, as requested. Essentially just looking to be able to filter based on the global attributes without repeating them ~18k times per file. I'm sure this "user ignorance" issue more than anything.

netcdf \2023May16_2024May13_BaddeckWharf_10392956 {
dimensions:
        lon = 1 ;
        lat = 1 ;
        time = 18157 ;
variables:
        double lon(lon) ;
                lon:units = "degrees_east" ;
                lon:long_name = "Longitude" ;
        double lat(lat) ;
                lat:units = "degrees_north" ;
                lat:long_name = "Latitude" ;
        double time(time) ;
                time:units = "seconds since 1970-01-01" ;
                time:long_name = "Time" ;
                time:calendar = "standard" ;
        double PRSTPR01(time, lat, lon) ;
                PRSTPR01:units = "kPa" ;
                PRSTPR01:_FillValue = NaN ;
                PRSTPR01:long_name = "Pressure (measured variable) exerted by the water body plus atmosphere by semi-fixed in-situ pressure sensor" ;
        double PPSBPR01(time, lat, lon) ;
                PPSBPR01:units = "m" ;
                PPSBPR01:_FillValue = NaN ;
                PPSBPR01:long_name = "Depth below surface of the water body by semi-fixed in-situ pressure sensor and correction to zero at sea level and conversion to depth using unspecified algorithm" ;
        double TEMPPR01(time, lat, lon) ;
                TEMPPR01:units = "degrees C" ;
                TEMPPR01:_FillValue = NaN ;
                TEMPPR01:long_name = "The degree of hotness of the water column expressed against a standard scale. Includes both IPTS-68 and ITS-90 scales." ;

// global attributes:
                :project_lead = "Someone" ;
                :cf_title = "Hobo temperature logger data" ;
                :cf_institute = "BIO" ;
                :cf_source = "2023May16_2024May13_BaddeckWharf_10392956.csv" ;
                :cf_author = "PKraska" ;
                :creation_date = 1752661825.68167 ;
                :license = "Open Government License - Canada (https://open.canada.ca/en/open-government-licence-canada)" ;
}

[rmendels]

@pkraska Thanks. No need to send it. I have test data I can try it out on using the ERDDAP I have on my laptop.