Skip to content

Commit f69ed99

Browse files
authored
fix: v1.0.1 refactor links
- ReadMe.md - SUMMARY.md - instructions.md - Detailed_Instructions_Data_Dictionary_File.md - Detailed_Instructions_Methods_File.md - Detailed_Instructions_Data_File.md
1 parent 06edf29 commit f69ed99

6 files changed

+44
-40
lines changed

Detailed_Instructions_Data_Dictionary_File.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -2,15 +2,15 @@
22

33
It is required to include a data dictionary (\_dd.csv) for each csv file in a dataset, or a single data dictionary file for an entire dataset. As mentioned in the detailed instructions for the terminology file, it may be useful to generate a master terminology file that includes the required (and optional) fields for all csv files and then pull from there the relevant fields for each \_dd.csv file. Figure 1 shows an example of how a master terminology file could be structured to comply with the \_dd.csv template as well as the terminology template (i.e., by including the “Unit” and “Column_or_Row_Long_Name” fields, as well as “Term_Type”) in order to allow for a single master file collection of all terminology used within a dataset/project/team etc. Note that the Column header “Term” should be replaced by “Column_or_Row_Name” in all \_dd.csv files.
44

5-
The following instructions are from the [ESS-DIVE file level metadata reporting format](https://github.com/ess-dive-community/essdive-file-level-metadata) and are designed to facilitate File-Level Metadata extraction of some fields using the [File Level Metadata Extractor](https://code.ornl.gov/ngee-arctic/ess-dive-meta).
5+
The following instructions are from the [ESS-DIVE file level metadata reporting format](https://github.com/ess-dive-workspace/essdive-file-level-metadata) and are designed to facilitate File-Level Metadata extraction of some fields using the [File Level Metadata Extractor](https://code.ornl.gov/ngee-arctic/ess-dive-meta).
66

7-
1. Create a CSV data dictionary using the [CSV_dd_template](https://github.com/ess-dive-community/essdive-file-level-metadata/blob/51f1a64260323d25754a693bce9a6f836ec3164f/CSV_dd/CSV_dd_template.xlsx). You can create either:
7+
1. Create a CSV data dictionary using the [CSV_dd_template](https://github.com/ess-dive-workspace/essdive-file-level-metadata/blob/main/CSV_dd/CSV_dd_template.csv). You can create either:
88

99
a. One data dictionary for each data file or
1010

1111
b. One data dictionary representing all data files in your dataset
1212

13-
2. See the [CSV_dd_quick_guide](https://github.com/ess-dive-community/essdive-file-level-metadata/blob/master/CSV_dd/csv_dd_quick_guide.md) for more details about completing template
13+
2. See the [CSV_dd_quick_guide](https://github.com/ess-dive-workspace/essdive-file-level-metadata/blob/main/CSV_dd/csv_dd_quick_guide.md) for more details about completing template
1414

1515
3. Enter a new row in the data dictionary for each column/row name in your data matrix
1616

@@ -19,9 +19,9 @@ The following instructions are from the [ESS-DIVE file level metadata reporting
1919
b. For example - if the same description applies to all spectra data column names in the data file(s)
2020
1. one row in the data dictionary can refer to "wave_\*" and this definition will be understood to apply to multiple column names within a data sheet
2121

22-
4. View an example [CSV_dd_example](https://github.com/ess-dive-community/essdive-file-level-metadata/blob/master/CSV_dd/csv_dd_example.md)
22+
4. View an example [CSV_dd_example](https://github.com/ess-dive-workspace/essdive-file-level-metadata/blob/main/CSV_dd/csv_dd_example.md)
2323

24-
5. Save the CSV_dd template following the [CSV Reporting Format guidance](https://github.com/ess-dive-community/essdive-csv-structure) with the filename "dd.csv" or "\*\_dd.csv". Alternatively you may choose to name you data dictionary:
24+
5. Save the CSV_dd template following the [CSV Reporting Format guidance](https://github.com/ess-dive-workspace/essdive-csv-structure) with the filename "dd.csv" or "\*\_dd.csv". Alternatively you may choose to name you data dictionary:
2525

2626
a. With the same name as the associated data file, but include '\_dd.csv' at the end of the file name
2727

Detailed_Instructions_Data_File.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,22 @@
11
# Detailed Instructions for Data Files
22

3-
We strongly recommend using the [Data_template](https://github.com/ess-dive-community/essdive-water-soil-sed-chem/blob/main/templates/Data_opt_template.csv) for your data files, as it will automatically ensure compliance with the reporting format and minimize issues with uploading your data package(s) to ESS-DIVE. The template was constructed to allow for ample flexibility in how to structure the data and, hence, be tailored to existing best practices or preferences used by data producers. Should you choose not to use the template, you will need to carefully follow all instructions and refer to the term_lists for this reporting format, as well as for those upstream of it ([Sample Metadata](https://ess-dive.gitbook.io/sample-id-and-metadata/), [csv](https://ess-dive.gitbook.io/csv-file-structure-reporting-format/), and [FLMD](https://ess-dive.gitbook.io/file-level-metadata-reporting-format/) reporting formats). It is also strongly recommended that you discuss your data file structure with an ESS-DIVE team member if you plan to deviate from the template.
3+
We strongly recommend using the [Data_template](templates/Data_opt_template.csv) for your data files, as it will automatically ensure compliance with the reporting format and minimize issues with uploading your data package(s) to ESS-DIVE. The template was constructed to allow for ample flexibility in how to structure the data and, hence, be tailored to existing best practices or preferences used by data producers. Should you choose not to use the template, you will need to carefully follow all instructions and refer to the term_lists for this reporting format, as well as for those upstream of it ([Sample Metadata](https://ess-dive.gitbook.io/sample-id-and-metadata/), [csv](https://ess-dive.gitbook.io/csv-file-structure-reporting-format/), and [FLMD](https://ess-dive.gitbook.io/file-level-metadata-reporting-format/) reporting formats). It is also strongly recommended that you discuss your data file structure with an ESS-DIVE team member if you plan to deviate from the template.
44

55
**Minimum requirement template**
66

77
Figure 1 shows a screenshot of the excel version of the data template in its simplest form (i.e. with only required fields included), unfilled (top) and with example data (bottom). In this case, all cells that are not pre-populated with N/A, -9999, or are unshaded (i.e. appear white) must be populated with data/metadata:
88

99
1. Columns B (Sample_Name) and C (Material) must be populated with the exact same values as recorded in the Sample Metadata file for the samples included in your dataset. (This applies to all rows - i.e. row 10 and below - that contain a measurement value in Column D (and any additional columns with a Variable_Name).
1010
2. Replace Column D header (Variable_Name) with the name of the variable you are reporting data for. If you want to include data for more than one variable in the same file, simply add more columns to the right (columns E, F, G, etc.) and name them appropriately.
11-
3. Add the required metadata for each variable (rows 4-9 for column D and each column to the right with a Variable_Name). Descriptions, format recommendations, and examples for each metadata row are found in the [term_list_dataFile](https://github.com/ess-dive-community/essdive-water-soil-sed-chem/blob/main/term_lists/term_list_dataFile.md).
11+
3. Add the required metadata for each variable (rows 4-9 for column D and each column to the right with a Variable_Name). Descriptions, format recommendations, and examples for each metadata row are found in the [term_list_dataFile](term_lists/term_list_dataFile.md).
1212
4. Add your measurement values starting on row 10 in column D (and any additional columns with a Variable_Name).
1313
5. Save your populated and finalized file in csv-format and make sure your file name ends with “\_data.csv”. Note that only letters, numbers, underscores “\_”, and hyphens “-” are allowed in file names. It is recommended to maintain a consistent, informative, and simple naming structure for your files, as it will simplify the use of the “\*” wild card for associated data dictionary files (\_dd.csv) and potential automatic file level metadata generation. For example, all data from a specific field site, sampling campaign, and/or year would follow the same naming structure either in the front or end of the file names - e.g., “soil_chem_Riverton_2019_data.csv” and “groundwater_chem_Riverton_2019_data.csv” or “Riverton_2019_soil_chem_data.csv” and “Riverton_2019_groundwater_chem_data.csv”.
1414

1515
![Diagram showing the empty data file template with only required fields.](.gitbook/assests/Data_File_Figure_1_Top.png)
1616
![Diagram showing a filled data file template with data and required metadata fields.](.gitbook/assests/Data_File_Figure_1_Bottom.png)
1717

1818
**Figure 1.**
19-
*Top:* Data_template with only required fields included. Blue shading indicates required fields, gray shading indicates cells that will never contain data (note that they still need to be filled with “N/A” (text fields) or “-9999” (numerical fields), as no empty cells are allowed). *Bottom:* Example of populated data file with only the minimum required fields for a dataset with raw (i.e. straight off the instrument) dissolved organic carbon measurements in water samples. This example assumes a separate method file is included in the data package with explanations for the MethodID_Analysis code (NPOC_0). Also note the -9999 entry on row 12, which indicates missing data, but no information about why it is missing. Although this is allowed, it is strongly recommended to add a column to explain the reason for missing data. To do this, add column E and entitle it “Dissolved_Organic_Carbon_dataFlags” or “Dissolved_Organic_Carbon_Notes” and add information on row 12 (all other rows in column E would be filled with N/A in this case) either as a dataFlag (i.e. a code that is explained in a separate [terminology file](https://github.com/ess-dive-community/essdive-water-soil-sed-chem/blob/main/Detailed_Instructions_Terminology_File.md)) or a note - see instructions and examples below.
19+
*Top:* Data_template with only required fields included. Blue shading indicates required fields, gray shading indicates cells that will never contain data (note that they still need to be filled with “N/A” (text fields) or “-9999” (numerical fields), as no empty cells are allowed). *Bottom:* Example of populated data file with only the minimum required fields for a dataset with raw (i.e. straight off the instrument) dissolved organic carbon measurements in water samples. This example assumes a separate method file is included in the data package with explanations for the MethodID_Analysis code (NPOC_0). Also note the -9999 entry on row 12, which indicates missing data, but no information about why it is missing. Although this is allowed, it is strongly recommended to add a column to explain the reason for missing data. To do this, add column E and entitle it “Dissolved_Organic_Carbon_dataFlags” or “Dissolved_Organic_Carbon_Notes” and add information on row 12 (all other rows in column E would be filled with N/A in this case) either as a dataFlag (i.e. a code that is explained in a separate [terminology file](Detailed_Instructions_Terminology_File.md)) or a note - see instructions and examples below.
2020

2121
**All optional and required fields template**
2222

Detailed_Instructions_Methods_File.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
The methods file template is structured to allow for collecting all types of methods into one master methods library, independent of the kind of method - e.g., sampling, measurements, sensors, preservation, data processing etc. Thus, it is totally acceptable (and even recommended) to reuse the same methods file for several or all datasets from a specific research group or project, as long as all methods used and referred to within any (meta)data files are included with enough details to meet the requirements for all relevant reporting formats.
44

5-
However, to simplify things, the following instructions and examples only include information on how to structure a Methods file for methods related to the water-soil-sed-chem-reporting format. For this purpose, it is strongly recommended to use the [Methods_template](https://github.com/ess-dive-community/essdive-water-soil-sed-chem/blob/main/templates/Methods_opt_template.csv) for your Methods file(s), as it will automatically ensure compliance with this reporting format and minimize issues with uploading your datasets to ESS-DIVE. Should you choose not to use the template, you will need to carefully follow all instructions and refer to the term_lists for this reporting format, as well as for those upstream of it (Sample Metadata, csv, and FLMD reporting formats). It is also strongly recommended that you discuss your method file(s) structure with an ESS-DIVE team member if you plan to deviate from the template.
5+
However, to simplify things, the following instructions and examples only include information on how to structure a Methods file for methods related to the water-soil-sed-chem-reporting format. For this purpose, it is strongly recommended to use the [Methods_template](templates/Methods_opt_template.csv) for your Methods file(s), as it will automatically ensure compliance with this reporting format and minimize issues with uploading your datasets to ESS-DIVE. Should you choose not to use the template, you will need to carefully follow all instructions and refer to the term_lists for this reporting format, as well as for those upstream of it (Sample Metadata, csv, and FLMD reporting formats). It is also strongly recommended that you discuss your method file(s) structure with an ESS-DIVE team member if you plan to deviate from the template.
66
Figure 1 shows examples of the minimum (top) and optional/recommended (bottom) metadata fields included in the methods file template. Note that the optional/recommended fields break out metadata that is also included in the Method_Description, but that can be useful for data users to use as basis for data searches (e.g. if looking for data from a specific type of instrument or from anoxically preserved or field moist samples).
77

88
Figure 1 shows examples of the minimum (top) and optional/recommended (bottom) metadata fields included in the methods file template. Note that the optional/recommended fields break out metadata that is also included in the Method_Description, but that can be useful for data users to use as basis for data searches (e.g. if looking for data from a specific type of instrument or from anoxically preserved or field moist samples).

README.md

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,21 @@
1-
# ESS-DIVE Reporting Format for Water-Soil-Sediment Chemistry Data v1.0.0
1+
# ESS-DIVE Reporting Format for Water-Soil-Sediment Chemistry Data v1.0.1
22

33
## About the reporting format
44
This reporting format is recommended for data from chemical concentration measurements of water, soil, and sediment samples.
55

66
## Quickstart Guide
7-
1. **Read** the instructions for each file type in this reporting format: [main instructions](https://github.com/ess-dive-community/essdive-water-soil-sed-chem/blob/main/instructions.md), [terminology](https://github.com/ess-dive-community/essdive-water-soil-sed-chem/blob/main/Detailed_Instructions_Terminology_File.md), [methods](https://github.com/ess-dive-community/essdive-water-soil-sed-chem/blob/main/Detailed_Instructions_Methods_File.md), [data](https://github.com/ess-dive-community/essdive-water-soil-sed-chem/blob/main/Detailed_Instructions_Data_File.md), [data dictionary](https://github.com/ess-dive-community/essdive-water-soil-sed-chem/blob/main/Detailed_Instructions_Data_Dictionary_File.md)
8-
2. **Review** [examples of filled in template files](https://github.com/ess-dive-community/essdive-water-soil-sed-chem/tree/main/examples)
9-
3. **Download** [blank templates](https://github.com/ess-dive-community/essdive-water-soil-sed-chem/tree/main/templates)
10-
4. **Use [term list](https://github.com/ess-dive-community/essdive-water-soil-sed-chem/tree/main/term_lists)** to fill in templates
7+
1. **Read** the instructions for each file type in this reporting format: [main instructions](instructions.md), [terminology](Detailed_Instructions_Terminology_File.md), [methods](Detailed_Instructions_Methods_File.md), [data](Detailed_Instructions_Data_File.md), [data dictionary](Detailed_Instructions_Data_Dictionary_File.md)
8+
2. **Review** [examples of filled in template files](examples)
9+
3. **Download** [blank templates](templates)
10+
4. **Use [term list](term_lists)** to fill in templates
1111
5. **Save** all templates as CSV files and **upload to ESS-DIVE** as dataset
12-
12+
13+
## Updates in v1.0.1
14+
In March 2025, a patch version of the Water-Soil-Sediment Chemistry reporting format was made to improve the overall experience with the associated reporting format documentation. Additionally, the standard reporting format keyword and standard term were added to the instructions.md file.
1315

1416
## How to contribute
1517

16-
This reporting format is evolving and adapting to meet the needs of the community. Feedback and new contributions are welcome. If you would like to suggest a change, please submit a GitHub issue using one of the [templates we provide](https://github.com/ess-dive-community/essdive-water-soil-sed-chem/issues/new/choose).
18+
This reporting format is evolving and adapting to meet the needs of researchers. Feedback and new contributions are welcome. If you would like to suggest a change, please submit a GitHub issue using one of the [templates we provide](https://github.com/ess-dive-workspace/essdive-water-soil-sed-chem/issues/new/choose).
1719

1820
Though our intent is to provide a reporting format that will enable data producers to gain wider recognition for their data and data users to gain more streamlined options for automated download and synthesis of datasets, the result is a reporting format that will not fully please anyone and that leaves a lot to be desired from everybody.
1921

@@ -32,4 +34,4 @@ Boye, K., Crystal-Ornelas, R., O’Ryan, D., Goldman, A. E., Damerow, J., Varadh
3234

3335
## References
3436

35-
Our reporting format was loosely based on those used by the EPA and USGS through the [Water Quality Portal (WQP)](https://www.waterqualitydata.us/), but were also inspired from other standards, best practices, and recommendations within the water quality, soil science, and geological survey communities. Most importantly the reporting format and templates were developed to fit with needs, recommendations, and advice from the DOE-BER-ESS community including both data producers and data users – a summary of the community member feedback and list of consulted members can be found as part of the dataset cited above.
37+
Our reporting format was loosely based on those used by the EPA and USGS through the [Water Quality Portal (WQP)](https://www.waterqualitydata.us/), but were also inspired from other standards, best practices, and recommendations within the water quality, soil science, and geological survey communities. Most importantly the reporting format and templates were developed to fit with needs, recommendations, and advice from the DOE-BER-ESS researchers including both data producers and data users – a summary of the DOE ESS member feedback and list of consulted members can be found as part of the dataset cited above.

0 commit comments

Comments
 (0)