|
1 | | -# Generating a dataset config file |
| 1 | +# Configuration files |
2 | 2 |
|
3 | | -The recommended approach to adding support for a new dataset is to start with an existing config. The best way to obtain the information for these fields is through a combination of looking at a sample data granule and looking at the dataset's documentation. Here we'll walk through the config for `AMSR-2_OSI-408`, looking at the various sections. |
| 3 | +## Dataset configs |
4 | 4 |
|
5 | | -## Dataset |
| 5 | +Config file per dataset contains all information needed for a dataset to be run through the pipeline from harvesting through aggregation. See README in `ecco_pipeline/conf/ds_configs` for more information. `ecco_pipeline/conf/ds_configs/deprecated` contains configs for datasets no longer supported, typically because they have been supplanted by a newer version. |
6 | 6 |
|
7 | | -``` |
8 | | -ds_name: AMSR-2_OSI-408 # Name for dataset |
9 | | -start: "19800101T00:00:01Z" # yyyymmddThh:mm:ssZ |
10 | | -end: "NOW" # yyyymmddThh:mm:ssZ for specific date or "NOW" for...now |
11 | | -``` |
| 7 | +## global_settings.py |
12 | 8 |
|
13 | | -- `ds_name` is the internal name for the dataset. We recommend using the dataset's shortname or something similar if shortname is not available. |
14 | | -- `start` and `end` are the isoformatted (with Z!) date ranges that the pipeline should try to process. The `end` field can also be the string "NOW" which will set the end field to the current datetime at runtime. |
15 | | - |
16 | | -## Harvester |
17 | | -This section contains fields that dictate which harvester the data should be pulled from. There are different fields required depending on the harvester. These are the fields consistent across all harvesters. |
18 | | -``` |
19 | | -harvester_type: osisaf |
20 | | -filename_date_fmt: "%Y%m%d" |
21 | | -filename_date_regex: '\d{8}' |
22 | | -``` |
23 | | -- `harvester_type` can be one of `cmr`, `osisaf`, `nisdc`, or `catds`. |
24 | | -- `filename_date_fmt` is the string format of the date in the filename. |
25 | | -- `filename_date_regex` is the regex format of the date in filename. |
26 | | - |
27 | | -The section where `cmr` is the harvester_type includes the following: |
28 | | -``` |
29 | | -cmr_concept_id: C2491756442-POCLOUD |
30 | | -provider: "archive.podaac" |
31 | | -``` |
32 | | -- `cmr_concept_id` is the unique concept_id identifier in CMR for the datatset |
33 | | -- `provider` is the provider of the data, used to select the download URLs. Typically this is set to "archive.podaac" although it will be different for NSIDC datasets. |
34 | | - |
35 | | - |
36 | | -The section `osisaf` or `catds` is the harvester includes: |
37 | | -``` |
38 | | -ddir: "ice/amsr2_conc" |
39 | | -``` |
40 | | -- `ddir` is the subdirectory where the specific data can be found. This is only required for non-cmr harvesters, and is not required for the non-cmr NSIDC harvester. |
41 | | - |
42 | | -## Metadata |
43 | | -This section includes details specific to the data. |
44 | | -``` |
45 | | -data_time_scale: "daily" # daily or monthly |
46 | | -hemi_pattern: |
47 | | - north: "_nh_" |
48 | | - south: "_sh_" |
49 | | -fields: |
50 | | - - name: ice_conc |
51 | | - long_name: Sea ice concentration |
52 | | - standard_name: sea_ice_area_fraction |
53 | | - units: " " |
54 | | - pre_transformations: [] |
55 | | - post_transformations: ["seaice_concentration_to_fraction"] |
56 | | -original_dataset_title: "Global Sea Ice Concentration (AMSR-2)" |
57 | | -original_dataset_short_name: "Global Sea Ice Concentration (AMSR-2)" |
58 | | -original_dataset_url: "https://osi-saf.eumetsat.int/products/osi-408" |
59 | | -original_dataset_reference: "https://osisaf-hl.met.no/sites/osisaf-hl.met.no/files/user_manuals/osisaf_cdop2_ss2_pum_amsr2-ice-conc_v1p1.pdf" |
60 | | -original_dataset_doi: "OSI-408" |
61 | | -``` |
62 | | -- `data_time_scale` is the time scale of the data, either daily or monthly. Monthly data is considered data averaged per month - all other data is considered daily. |
63 | | -- `hemi_pattern` sets the filename pattern for data split by hemisphere. This section can be omitted for datasets that don't do this. |
64 | | -- `fields` is the list of data variables that should be transformed as part of the pipeline. You need to manually provide the field's `name`, `long_name`, `standard_name`, and `units`. The `pre_transformations` and `post_transformations` are the names of functions to be applied to the specific data field. Some examples are units conversion, or data masking. Functions are defined in `ecco_pipeline/utils/processing_utils/ds_functions.py`. |
65 | | -- The five `original_*` fields are dataset level metadata that will be included in transformed file metadata. |
66 | | - |
67 | | -## Transformation |
68 | | -This section contains fields required for transformation and contain information on data resolution etc. For hemispherical data, `area_extent`, `dims`, and `proj_info` must be defined for each hemispher as below. It is not unusual for the values in this section to be determined iteratively. The testing notebooks in `tests/quicklook_notebooks/` are a useful tool in determining the validity of the values provided. |
69 | | -``` |
70 | | -t_version: 2.0 # Update this value if any changes are made to this file |
71 | | -data_res: 10/111 # Resolution of dataset |
72 | | -
|
73 | | -# Values for non split datasets (for datasets split into nh/sh, append '_nh'/'_sh') |
74 | | -area_extent_nh: [-3845000, -5345000, 3745000, 5845000] |
75 | | -area_extent_sh: [-3950000, -3950000, 3945000, 4340000] |
76 | | -dims_nh: [760, 1120] |
77 | | -dims_sh: [790, 830] |
78 | | -proj_info_nh: |
79 | | - area_id: "3411" |
80 | | - area_name: "polar_stereographic" |
81 | | - proj_id: "3411" |
82 | | - proj4_args: "+init=EPSG:3411" |
83 | | -proj_info_sh: |
84 | | - area_id: "3412" |
85 | | - area_name: "polar_stereographic" |
86 | | - proj_id: "3412" |
87 | | - proj4_args: "+init=EPSG:3412" |
88 | | -
|
89 | | -notes: "" |
90 | | -``` |
91 | | -- `t_version` is a metadata field used internally in the pipeline. Modifying the value will trigger retransformation. |
92 | | -- `data_res` is the spatial resolution of the dataset in degrees |
93 | | -- `area_extent` is the area extent specific to this data in the form: lower_left_x, lower_left_y, upper_right_x, upper_right_y |
94 | | -- `dims` is the size of longitude or x coordinate, latitude or y coordinate |
95 | | -- `proj_info` contains projection information used by pyresample |
96 | | -- `notes` is an optional string to include in global metadata in output files |
97 | | - |
98 | | -## Aggregation |
99 | | -This section contains fields required for aggregating data. |
100 | | -``` |
101 | | -a_version: 1.3 # Update this value if any changes are made to this file |
102 | | -remove_nan_days_from_data: False # Remove empty days from data when aggregating |
103 | | -do_monthly_aggregation: True |
104 | | -skipna_in_mean: True # Controls skipna when calculating monthly mean |
105 | | -``` |
106 | | -- `a_version` is a metadata field used internally in the pipeline. Modifying the value will trigger reaggregation. |
107 | | -- `remove_nan_days_from_data` will remove nan days from aggregated outputs |
108 | | -- `do_monthly_aggregation` will also compute monthly averages when aggregating annual files |
109 | | -- `skipna_in_mean` is used when calculating the monthly mean |
| 9 | +Script that contains some settings that are used globally throughout the pipeline, such as the location of the output directory. This file must be manually set up after cloning the repo. |
0 commit comments