BUFR processor to read NOMADS BUFR files into NNJA-AI format

[jacobbieker]

It would be great, although probably a bit more out of scope for this repo, to be able to read the BUFR files from NOMADS into the same format as the NNJA-AI archive.

Detailed Description

NNJA-AI archive is now version 1, and has a large historical archive available to training models. The NNJA source archive will be updated with a 72 hour delay, and the public NNJA-AI archive seems like it will follow that eventually. From this comment it doesn't seem like there is a plan to release the near-real time converted files that is available in NOMADS, or, alternatively, the code the read the BUFR files and decode to the parquet format.

Context

For real-time forecasting or data assimilation, being able to read from NOMADS is somewhat crucial, as otherwise the 72 hour delay makes the observations pointless for a operational system.

Possible Implementation

It would be quite a bit of engineering work, I think, to make this. It should probably be in its own repo if so, and I don't have that much experience with BUFR files currently. There are some BUFR reading python libraries that could be used. Performance would be somewhat of a lesser importance, as this would be only to read the real-time data, not back process the NNJA archive.

[jacobbieker]

There are also some caveats for the processing that might make it easier, as the bufr files only come every 6 hours, so for the most up to date, and quickest access data, an initial prototype would want to focus on the hard-to-get or slow data sources.

Primarily, this would mean we want to decode BUFR files for (currently):

1. ADPUPA (IGRA, as another source, also has a 72 hour delay)
2. CrIS - Easy to get from AWS, but haven't found easy way to process it, so NOMADS BUFR seems the best
3. IASI - Similar to CrIS, can process IASI data from EUMETSAT with the Data Store, or SatPy, but possibly not the easiest?

The geostationary radiances can be fairly easily processed with OCF's satellite-consumer, especially the fork https://github.com/jacobbieker/satellite-consumer and generally have a delay of a few minutes (EUMETSAT) or 1.5 hours (the rest on AWS).

AMSU-A and other MetOp ones can be downloaded with a 2.5ish hour delay from the API.
ATMS and MHS are similar.

[yuvraajnarula]

Hi @jacobbieker, long time!
I just wanted to know how you envision this implementation? Given the critical nature of near-real-time data for operational forecasting (where the 72-hour delay essentially makes observations unusable), here's what I'm thinking:

• BUFR Decoding Library: Start with ecCodes (via Python bindings) or pybufr-ecmwf. Which one have you found more reliable for operational use? ecCodes seems more robust but I'm open to alternatives if you've had better experience elsewhere.
• Priority Targets (in order of operational impact):
  • ADPUPA first - upper-air soundings are critical for forecast initialization
  • CrIS/IASI next - hyperspectral soundings for data assimilation
  • Other sources as needed
• Poll NOMADS every 6 hours (or more frequently if sub-6hr data becomes available)
• Immediate processing upon BUFR availability - aiming for <30 min latency from file drop to decoded Parquet
• Output to NNJA-AI compatible Parquet format for seamless integration
• Schema Compatibility: Match the NNJA-AI Parquet structure exactly so decoded BUFR data can drop directly into your operational pipeline.

[yuvraajnarula]

I have some doubts. If you could answer these:

• Should this be standalone (separate repo as you mentioned), or would it make sense to extend your satellite-consumer fork to create a unified real-time observation ingestion system?
• What's the maximum acceptable delay between BUFR availability on NOMADS and having usable Parquet files? Is 30 minutes realistic, or do you need faster?

I would love to be of some help on this interesting issue. So let me know if I could give this a try

[jacobbieker]

I would add it is a module in this repo first, and then maybe split it out later into its own thing. Ideally, its as soon as the files appear on NOMADS, but this implementation shouldn't deal with actually getting the files from NOMADS, what is important is the BUFR conversion. The actual handling of getting the files, etc. will be handled by some external process. So yeah, if you want to work on this, just focus on BUFR conversion and being able to match the NNJA-AI Parquet from a NNJA BUFR file.

[yuvraajnarula]

Okay, got the vision! I will push a PR soon for this. Thank you!

[yuvraajnarula]

@jacobbieker, should there be a pytest file for this bufr_processor?

[jacobbieker]

Yeah, ideally, one that takes an example bufr file from the nnja archive, and passes if it matches exactly the corresponding NNJA-AI parquet file