[Proposal] Either include all codelist in the schema or keep them all external (csv)

[matamadio]

There's trade-off between flexibility vs simplicity. Let me analyze both approaches:

External CSV Files

• Pros:
  • Independent updates without schema versioning
  • Domain experts can contribute easily
  • Cleaner separation of concerns

• Cons:
  • Additional HTTP requests (slower loading)
  • Network dependency
  • Complex error handling
  • Potential version mismatches
  • Inconsistent with existing enums

Codelists Integrated into Schema

• Pros:
  • Single source of truth
  • No network dependencies after load
  • Guaranteed consistency
  • Better performance
  • Simpler implementation
  • Works offline

• Cons:
  • Need schema updates for codelist changes
  • Harder for non-technical contributors

[stufraser1]

Codelists Integrated into Schema - additional pro: metadata editor requires the codelist to be in the schema, to provide a drop-down list to users.

[stufraser1]

At the moment the closed lists are integrated into the schema, and open lists are not - maintained only as CSV files, which enables people to add their own values. If the open codelists were integrated into the schema, would they cease to function as open codelists in the metadata spreadsheet / validation process? I.e. would users still be able to add their own values, and would those fields with open codelists allow those new values to pass the checks against the enum list?

Currently, I would be in favour of keeping codelists as both csv and in the schema, for flexibility of use, with a potentially scripted check to check periodically that they are synchronised / matching

[matamadio]

For open codelists, the dropdown would be a text input+suggestions+existing value. So you can see what is already listed and also add your own value if not present.
This is something I'm already testing in a all-codelists-embedded version of the schema.

Potentially the metadata editor could look up those csv to build the dropdown; I can't test it out because those csv are currently only present in the source repo, aka not accessible through the published website like https://riskdatalibrary.org/schema/codelists.

Else, another way is to make the json schema autobuilt from the csv codelists after any list update. This way we ensure consistency between the two. Csv would act as middle-step between change requests and schema version.

PR request to add a value to open codelist > PR approved > csv is updated > auto workflow rebuilds the schema (adding +1 to subversion, e.g. 2.2, 2.3)

In this scenario, those additional value entered by user should avoid the validation step - else he would need to wait for the PR to be approved and the schema updated to finalise his metadata, that sounds like a bottleneck to avoid.

Overall, I don't see any major advantage in this csv strategy compared to just editing the json file.

Ultimately, my suggestion is to keep open codelists at a minimum. License, media type, data formats, roles, location gazzetter, quantity kind and a few others can become more inclusive of options and closed.

[matamadio]

More in depth overview of alternatives for codelist management

Embedded VS csv codelists

There are pros and cons to each. An hybrid approach could be:

• Closed enums: Embed directly in the schema (for simplicity and validation).

• Open or very large enums: Reference external sources (with documentation and tooling support).

  An open enum is a list of recommended or commonly used values — but not exhaustive. You want users to be able to:

  • Choose from a suggested list (e.g., for UI dropdowns)
  • Or supply their own value (e.g., a new crop type or job title)
    • If the value is in the enum list → ✅ OK
    • If it's not in the enum list but still a valid string → ✅ also OK

But JSON Schema does not have a built-in "open enum" type, so we need to work around it.

Suggested method: anyOf with enum + type

{
  "type": "object",
  "properties": {
    "cropType": {
      "anyOf": [
        { "type": "string", "enum": ["wheat", "rice", "maize"] },
        { "type": "string" }
      ],
      "description": "Suggested values are wheat, rice, maize — but other strings are allowed"
    }
  },
  "required": ["cropType"]
}

What happens:

• "wheat" → ✅ valid
• "bananas" → ✅ valid
• 42 → ❌ invalid (not a string)

Let’s break this down step by step from both the user and admin perspectives.

🧑‍💻 User Side — Suggesting a New Value

Scenario: The user wants to submit a new value for an open list (e.g., a new cropType called "teff" not found in the current list).
How it might work:

1. User fills a form or uses a schema-driven UI (e.g., JSON editor or API input).
2. They enter "teff" in the cropType field.
3. The schema accepts it (due to the open enum setup with anyOf or no strict enum).
4. The system detects that "teff" is not in the current version of the codelist and:
   • Flags it as "user-defined"
   • Stores it with metadata (submittedBy, date, reason, etc.)
5. The system submits the new term for moderation/approval.

👩‍⚖️ Admin Side — Reviewing and Approving New Values

1. Admin sees a dashboard or a GitHub issue/PR with pending new value requests:

   • Suggested value
   • Submitted metadata
   • Usage context or justification

2. Admin reviews each suggestion:

   • Is it relevant?
   • Is it a duplicate or alias of an existing value?
   • Does it meet naming/formatting standards?

3. Admin approves, modifies, or rejects:

   • Approved values are added to the official source of truth, e.g., a CSV or JSON file storing the codelist
   • The codelist is versioned (e.g., v1.1.0 → v1.1.2)
   • A new version of the schema is generated (automatically)
   • Rejected values may be flagged or hidden from further use

[matamadio]

Codelists dependency

In addition to the above discussion about codelist integration, there is the opportunity for the metadata editor to enforce the dependency between attributes.
E.g., if Hazard type = Drought, the choice of Hazard process should be just between drought processes.

This can be applied for several codelists and would likely reduce the chance of nonsensical selections before schema validation.

In terms of implementation, it is done easily using values mapping:

  "hazard_process_mappings": {
    "drought": ["agricultural_drought", "hydrological_drought", "meteorological_drought", "socioeconomic_drought"],
    "flood": ["fluvial_flood", "pluvial_flood", "groundwater_flood", "coastal_flood"],
    "earthquake": ["primary_rupture", "secondary_rupture", "ground_motion", "liquefaction"],
    "extreme_temperature": ["extreme_cold", "extreme_heat"],
    "strong_wind": ["extratropical_cyclone", "tropical_cyclone", "tornado"],
    "wildfire": ["wildfire"],
    "landslide": ["snow_avalanche", "landslide_general", "landslide_rockslide", "landslide_mudflow", "landslide_rockfall"],
    "tsunami": ["tsunami"],
    "volcanic": ["ashfall", "volcano_ballistics", "lahar", "lava", "pyroclastic_flow"],
    "coastal_flood": ["coastal_flood", "storm_surge"],
    "convective_storm": ["tornado"]
  },

This mapping can be part of the schema itself (requiring all codelists to be embedded) or we can use an hybrid approach, with both codelists and mapping sitting on external data.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "Risk dataset (Risk Data Library Schema v2.1)",
  
  // Keep dependency mappings in schema (lightweight)
  "codelist_dependencies": {
    "hazard_processes": {
      "depends_on": "hazard_type",
      "codelist_url": "https://docs.riskdatalibrary.org/en/latest/codelists/process_type.csv",
      "mappings_url": "https://docs.riskdatalibrary.org/en/latest/codelists/hazard_process_mappings.json"
    },
    "intensity_measures": {
      "depends_on": "hazard_type", 
      "codelist_url": "https://docs.riskdatalibrary.org/en/latest/codelists/IMT.csv",
      "mappings_url": "https://docs.riskdatalibrary.org/en/latest/codelists/hazard_imt_mappings.json"
    }
  },
  
  // Standard property definitions reference external codelists
  "properties": {
    "hazard": {
      "properties": {
        "type": {
          "type": "string",
          "$ref": "#/$defs/hazard_type",
          "codelist": "hazard_type.csv"
        },
        "processes": {
          "type": "array", 
          "$ref": "#/$defs/process_type",
          "codelist": "process_type.csv",
          "depends_on": "type"  // <-- New dependency marker
        }
      }
    }
  }
}

External Mapping Files

hazard_process_mappings.json:

{
  "drought": ["agricultural_drought", "hydrological_drought", "meteorological_drought", "socioeconomic_drought"],
  "flood": ["fluvial_flood", "pluvial_flood", "groundwater_flood", "coastal_flood"],
  "earthquake": ["primary_rupture", "secondary_rupture", "ground_motion", "liquefaction"],
  "extreme_temperature": ["extreme_cold", "extreme_heat"],
  "strong_wind": ["extratropical_cyclone", "tropical_cyclone", "tornado"],
  "wildfire": ["wildfire"],
  "landslide": ["snow_avalanche", "landslide_general", "landslide_rockslide", "landslide_mudflow", "landslide_rockfall"],
  "tsunami": ["tsunami"],
  "volcanic": ["ashfall", "volcano_ballistics", "lahar", "lava", "pyroclastic_flow"],
  "coastal_flood": ["coastal_flood", "storm_surge"],
  "convective_storm": ["tornado"]
}

process_type.csv:

Code,Title,Description
agricultural_drought,Agricultural Drought,Conditions affecting crop production
hydrological_drought,Hydrological Drought,Below-normal water levels in aquifers
meteorological_drought,Meteorological Drought,Extended period of deficient precipitation
...

[matamadio]

Changes applied in the 2.1 version

[duncandewhurst]

Changes applied in the 2.1 version

Please could you link to the PR in which the changes were made?