Download failure for PREFIRE data

[greg9q]

gregq@lemon:~/Desktop/prefire-viirs$ cmrfetch collections -s PREFIRE_SAT*_2B-FLX
┌─────────────────────┬─────────┬────────────────────────┬─────────────┬────────────┐
│ SHORTNAME           │ VERSION │ CONCEPT                │ REVISION_ID │ PROVIDER   │
├─────────────────────┼─────────┼────────────────────────┼─────────────┼────────────┤
│ PREFIRE_SAT1_2B-FLX │ R01     │ C3518594632-LARC_CLOUD │ 12          │ LARC_CLOUD │
│ PREFIRE_SAT2_2B-FLX │ R01     │ C3499202317-LARC_CLOUD │ 11          │ LARC_CLOUD │
└─────────────────────┴─────────┴────────────────────────┴─────────────┴────────────┘
gregq@lemon:~/Desktop/prefire-viirs$ cmrfetch granules -c C3518594632-LARC_CLOUD -t 2025-08-21T22:54:00Z,2025-08-21T22:54:00Z
┌─────────────────────────────────────────────────────┬─────────┬─────────────────────────────────────────────────────┬────────────────────────┬─────────────┐
│ NAME                                                │ SIZE    │ NATIVE_ID                                           │ CONCEPT_ID             │ REVISION_ID │
├─────────────────────────────────────────────────────┼─────────┼─────────────────────────────────────────────────────┼────────────────────────┼─────────────┤
│ PREFIRE_SAT1_2B-FLX_R01_P00_20250821223824_06708.nc │ 4833 KB │ PREFIRE_SAT1_2B-FLX_R01_P00_20250821223824_06708.nc │ G3671893497-LARC_CLOUD │ 1           │
└─────────────────────────────────────────────────────┴─────────┴─────────────────────────────────────────────────────┴────────────────────────┴─────────────┘
gregq@lemon:~/Desktop/prefire-viirs$ cmrfetch granules -c C3518594632-LARC_CLOUD -t 2025-08-21T22:54:00Z,2025-08-21T22:54:00Z --download .
1 results
failed! https://data.asdc.earthdata.nasa.gov/asdc-prod-protected/PREFIRE/PREFIRE_SAT1_2B-FLX_R01/2025.08.21/PREFIRE_SAT1_2B-FLX_R01_P00_20250821223824_06708.nc error=fetching: expected one of md5, sha256, sha512, got SHA-256

Seems like it's caused by an unexpected format for the checksum type.

[greg9q]

A couple thoughts beyond fixing the specific case of "SHA-256":

1. Are there other unexpected values for that checksum type string in CMR that we should be handling?
2. Perhaps an unexpected type should result in a warning and still download the file.

[bmflynn]

The algorithm name is documented in https://wiki.earthdata.nasa.gov/display/CMR/Archive+And+Distribution+Information+for+Granules.

The value SHA-256 is valid, however, it just seems to be handled incorrectly here

cmrfetch/internal/fetch.go

Line 32 in 42c889c

func newHash(alg string) (hash.Hash, error) {

.

I believe all it needs is an additional value for each case that includes the -, e.g., case "sha256", "sha-256"

I could definitely see it just being a warning rather than a hard faliure

[odlem]

Looks like that code path is covered

cmrfetch/internal/fetch_test.go

Line 23 in 42c889c

func Test_newHash(t *testing.T) {

It also looks like fmt supports a Warnf mode. I'll see about updating the test to handle Warn instead. fmt does not natively support Warnf

I'll also add a hyphen check to sha512 (or could just remove all hyphens from the alg string)

diff --git a/internal/fetch.go b/internal/fetch.go
index f221560..8cd3e3d 100644
--- a/internal/fetch.go
+++ b/internal/fetch.go
@@ -34,12 +34,12 @@ func newHash(alg string) (hash.Hash, error) {
        switch strings.ToLower(alg) {
        case "md5":
                hash = md5.New()
-       case "sha256":
+       case "sha256", "sha-256":
                hash = sha256.New()
        case "sha512":
                hash = sha512.New()
        default:
-               return nil, fmt.Errorf("expected one of md5, sha256, sha512, got %s", alg)
+               return nil, fmt.Warnf("expected one of md5, sha256, sha512, got %s", alg)
        }
        return hash, nil
 }

[odlem]

Looking into this more, I think it needs to stay error, since it has to know which hash object to generate. If none of the cases match, no hash is generated, so the checksum checks in internal/fetch_pool.go would all have to get ignored, since there's no hash to compare to.

[odlem]

After some discussion, we've decided that ignoring the check comparison in the rare event that the primary hash types are not covered is the right thing to do. I will update fetch.go and fetch_pool.go accordingly.

[odlem]

Sample warning result (this warning actually will not print after the change, since we're adding SHA-256 to the list of options)

# ./build/cmrfetch-linux-amd64 granules -c C3518594632-LARC_CLOUD -t 2025-08-21T22:54:00Z,2025-08-21T22:54:00Z --download .
1 results
WARNING: expected one of md5, sha256, sha512, got "SHA-256"
fetched https://data.asdc.earthdata.nasa.gov/asdc-prod-protected/PREFIRE/PREFIRE_SAT1_2B-FLX_R01/2025.08.21/PREFIRE_SAT1_2B-FLX_R01_P00_20250821223824_06708.nc in 3.5s(10.5 Mb/s)

[bmflynn]

Fixed by pull-13