source product version and slicing checks

[johntruckenbrodt]

Currently the processor does not do anything to filter duplicate source products. It often happens that source products (GRD/SLC) are reprocessed and, if the data archive is not optimally curated, multiple versions of the same acquisition may exist. In some cases the reprocessed products are also resliced so that both versions may not easily be compared by the acquisition times.

Consider the following case. A data take of products was first processed with manifest.safe metadata attribute sliceNumber=0, then reprocessed with sliceNumber>0 using the same slicing, and later reprocessed again this time using a different slicing and sliceNumber=0 again. Hence, three versions of the same data take may exist. At the time of writing, ASF only offers the version processed last and CDSE offers all three versions. In case CDSE is used as reference archive, s1ard has no way of determining which are the right products to use.

S1A_IW_GRDH_1SDV	slice_num	pref.	ASF	CDSE
*_20231207T064644_20231207T064709_051545_0638F1_9569	0		x	x
*_20231207T064649_20231207T064714_051545_0638F1_4B74	6	x		x
*_20231207T064649_20231207T064714_051545_0638F1_95E7	0			x
*_20231207T064709_20231207T064734_051545_0638F1_440B	0		x	x
*_20231207T064714_20231207T064739_051545_0638F1_4799	7	x		x
*_20231207T064714_20231207T064739_051545_0638F1_AC9A	0			x
*_20231207T064734_20231207T064759_051545_0638F1_9500	0		x	x
*_20231207T064739_20231207T064804_051545_0638F1_C7A2	8	x		x
*_20231207T064739_20231207T064804_051545_0638F1_FECD	0			x
*_20231207T064759_20231207T064824_051545_0638F1_8191	0		x	x

The reprocessed product with the 0 slice numbers is the one that was processed last.
Also, all products have the s1sarl1:productTimelinessCategory value NRT-3h and thus cannot be differentiated by it:

S1A_IW_GRDH_1SDV	slice_num	proc.time	timeliness
*_20231207T064739_20231207T064804_051545_0638F1_FECD	0	20231207T085207	NRT-3h
*_20231207T064649_20231207T064714_051545_0638F1_95E7	0	20231207T085214	NRT-3h
*_20231207T064714_20231207T064739_051545_0638F1_AC9A	0	20231207T085228	NRT-3h
*_20231207T064649_20231207T064714_051545_0638F1_4B74	6	20231207T155100	NRT-3h
*_20231207T064739_20231207T064804_051545_0638F1_C7A2	8	20231207T155140	NRT-3h
*_20231207T064714_20231207T064739_051545_0638F1_4799	7	20231207T155151	NRT-3h
*_20231207T064644_20231207T064709_051545_0638F1_9569	0	20231207T171137	NRT-3h
*_20231207T064734_20231207T064759_051545_0638F1_9500	0	20231207T171139	NRT-3h
*_20231207T064709_20231207T064734_051545_0638F1_440B	0	20231207T171145	NRT-3h
*_20231207T064759_20231207T064824_051545_0638F1_8191	0	20231207T171149	NRT-3h

So, ASF just keeps the products that were processed last. In this case this might not suffice. I have opened an issue at ASF about this.

[johntruckenbrodt]

@avalentino may I ask for your opinion?

[johntruckenbrodt]

#236 is only a temporary fix for GRD neighbor selection. the scene search functionality still does not differentiate between different versions.

[senmao]

Hi @johntruckenbrodt , by checking all available scenes from datatake 0638F1 (also 407793) at EODC, I have found that we have many duplicate scenes as shown below. There are scenes with sliceNumber=0 and sliceCount=0 while others have correct sliceNumber and sliceCount=26. By filtering out scenes with sliceCount=26, we can find the complete slices for this datatake.

When sliceNumber and sliceCount metadata are 0, I think the scene was not processed with entire datatake (this need to be checked with ESA), therefore sliceNumber and sliceCount can not be assigned. In general, we should use scenes with correct sliceNumber and sliceCount, and avoid scenes with sliceNumber=0 and sliceCount=0. In case there is only scenes with sliceNumber=0 and sliceCount, a consistent way need to be found to handle it.

Regarding timeliness category NRT-3h and Fast-24h, according to this article, the same processing is performed for Sentinel-1 L1/2 products tagged NRT-3h or Fast24h after 23 February 2021.

All scenes from the datatake

(wizsard) scao@scao-P14s:codes$ python wiz_inspect_s1_metadata.py S1A_IW_GRDH_1SDV_20231207T064440_20231207T064509_051545_0638F1_97E6: timeliness=NRT-3h sliceNumber=1 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T064440_20231207T064509_051545_0638F1_C523: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T064509_20231207T064534_051545_0638F1_8574: timeliness=NRT-3h sliceNumber=2 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T064509_20231207T064534_051545_0638F1_B868: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T064534_20231207T064559_051545_0638F1_C8D7: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T064534_20231207T064559_051545_0638F1_D8C7: timeliness=NRT-3h sliceNumber=3 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T064559_20231207T064624_051545_0638F1_411E: timeliness=NRT-3h sliceNumber=4 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T064559_20231207T064624_051545_0638F1_CA18: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T064616_20231207T064644_051545_0638F1_18D2: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T064624_20231207T064649_051545_0638F1_1B17: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T064624_20231207T064649_051545_0638F1_CB5F: timeliness=NRT-3h sliceNumber=5 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T064644_20231207T064709_051545_0638F1_9569: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T064649_20231207T064714_051545_0638F1_4B74: timeliness=NRT-3h sliceNumber=6 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T064649_20231207T064714_051545_0638F1_95E7: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T064709_20231207T064734_051545_0638F1_440B: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T064714_20231207T064739_051545_0638F1_4799: timeliness=NRT-3h sliceNumber=7 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T064714_20231207T064739_051545_0638F1_AC9A: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T064734_20231207T064759_051545_0638F1_9500: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T064739_20231207T064804_051545_0638F1_C7A2: timeliness=NRT-3h sliceNumber=8 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T064739_20231207T064804_051545_0638F1_FECD: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T064759_20231207T064824_051545_0638F1_8191: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T064804_20231207T064829_051545_0638F1_8757: timeliness=NRT-3h sliceNumber=9 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T064804_20231207T064829_051545_0638F1_9D62: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T064824_20231207T064849_051545_0638F1_4A55: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T064829_20231207T064854_051545_0638F1_2E93: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T064829_20231207T064854_051545_0638F1_5285: timeliness=NRT-3h sliceNumber=10 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T064849_20231207T064914_051545_0638F1_E77B: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T064854_20231207T064919_051545_0638F1_0D83: timeliness=NRT-3h sliceNumber=11 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T064854_20231207T064919_051545_0638F1_31AA: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T064914_20231207T064939_051545_0638F1_9391: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T064919_20231207T064944_051545_0638F1_1038: timeliness=NRT-3h sliceNumber=12 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T064919_20231207T064944_051545_0638F1_85B5: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T064939_20231207T065004_051545_0638F1_E2FB: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T064944_20231207T065009_051545_0638F1_1C69: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T064944_20231207T065009_051545_0638F1_A5EA: timeliness=NRT-3h sliceNumber=13 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T065004_20231207T065029_051545_0638F1_2205: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T065009_20231207T065034_051545_0638F1_00B8: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T065009_20231207T065034_051545_0638F1_7474: timeliness=NRT-3h sliceNumber=14 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T065029_20231207T065054_051545_0638F1_8EC0: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T065034_20231207T065059_051545_0638F1_0AAC: timeliness=NRT-3h sliceNumber=15 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T065034_20231207T065059_051545_0638F1_6FCE: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T065054_20231207T065119_051545_0638F1_017B: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T065059_20231207T065124_051545_0638F1_2F1B: timeliness=NRT-3h sliceNumber=16 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T065059_20231207T065124_051545_0638F1_E3C7: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T065119_20231207T065144_051545_0638F1_D3D3: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T065124_20231207T065149_051545_0638F1_6303: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T065124_20231207T065149_051545_0638F1_A811: timeliness=NRT-3h sliceNumber=17 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T065144_20231207T065209_051545_0638F1_B71C: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T065149_20231207T065214_051545_0638F1_30E5: timeliness=NRT-3h sliceNumber=18 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T065149_20231207T065214_051545_0638F1_4755: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T065209_20231207T065234_051545_0638F1_7C67: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T065214_20231207T065239_051545_0638F1_4860: timeliness=NRT-3h sliceNumber=19 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T065214_20231207T065239_051545_0638F1_8FEF: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T065234_20231207T065259_051545_0638F1_29B8: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T065239_20231207T065304_051545_0638F1_365B: timeliness=NRT-3h sliceNumber=20 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T065239_20231207T065304_051545_0638F1_B195: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T065259_20231207T065324_051545_0638F1_0A88: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T065304_20231207T065321_051545_0638F1_1DD6: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T065304_20231207T065329_051545_0638F1_B00E: timeliness=NRT-3h sliceNumber=21 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T065324_20231207T065349_051545_0638F1_7BEE: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T065329_20231207T065354_051545_0638F1_AF28: timeliness=NRT-3h sliceNumber=22 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T065349_20231207T065414_051545_0638F1_399F: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T065354_20231207T065419_051545_0638F1_0A83: timeliness=NRT-3h sliceNumber=23 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T065414_20231207T065439_051545_0638F1_9D5C: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T065419_20231207T065444_051545_0638F1_9D53: timeliness=NRT-3h sliceNumber=24 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T065439_20231207T065504_051545_0638F1_4601: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T065444_20231207T065509_051545_0638F1_C636: timeliness=NRT-3h sliceNumber=25 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T065504_20231207T065527_051545_0638F1_8409: timeliness=NRT-3h sliceNumber=0 sliceCount=0 S1A_IW_GRDH_1SDV_20231207T065509_20231207T065528_051545_0638F1_F224: timeliness=NRT-3h sliceNumber=26 sliceCount=26

scenes which have correct sliceNumber and sliceCount from the datatake

(wizsard) scao@scao-P14s:codes$ python wiz_inspect_s1_metadata.py | grep "sliceCount=26" S1A_IW_GRDH_1SDV_20231207T064440_20231207T064509_051545_0638F1_97E6: timeliness=NRT-3h sliceNumber=1 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T064509_20231207T064534_051545_0638F1_8574: timeliness=NRT-3h sliceNumber=2 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T064534_20231207T064559_051545_0638F1_D8C7: timeliness=NRT-3h sliceNumber=3 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T064559_20231207T064624_051545_0638F1_411E: timeliness=NRT-3h sliceNumber=4 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T064624_20231207T064649_051545_0638F1_CB5F: timeliness=NRT-3h sliceNumber=5 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T064649_20231207T064714_051545_0638F1_4B74: timeliness=NRT-3h sliceNumber=6 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T064714_20231207T064739_051545_0638F1_4799: timeliness=NRT-3h sliceNumber=7 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T064739_20231207T064804_051545_0638F1_C7A2: timeliness=NRT-3h sliceNumber=8 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T064804_20231207T064829_051545_0638F1_8757: timeliness=NRT-3h sliceNumber=9 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T064829_20231207T064854_051545_0638F1_5285: timeliness=NRT-3h sliceNumber=10 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T064854_20231207T064919_051545_0638F1_0D83: timeliness=NRT-3h sliceNumber=11 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T064919_20231207T064944_051545_0638F1_1038: timeliness=NRT-3h sliceNumber=12 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T064944_20231207T065009_051545_0638F1_A5EA: timeliness=NRT-3h sliceNumber=13 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T065009_20231207T065034_051545_0638F1_7474: timeliness=NRT-3h sliceNumber=14 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T065034_20231207T065059_051545_0638F1_0AAC: timeliness=NRT-3h sliceNumber=15 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T065059_20231207T065124_051545_0638F1_2F1B: timeliness=NRT-3h sliceNumber=16 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T065124_20231207T065149_051545_0638F1_A811: timeliness=NRT-3h sliceNumber=17 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T065149_20231207T065214_051545_0638F1_30E5: timeliness=NRT-3h sliceNumber=18 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T065214_20231207T065239_051545_0638F1_4860: timeliness=NRT-3h sliceNumber=19 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T065239_20231207T065304_051545_0638F1_365B: timeliness=NRT-3h sliceNumber=20 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T065304_20231207T065329_051545_0638F1_B00E: timeliness=NRT-3h sliceNumber=21 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T065329_20231207T065354_051545_0638F1_AF28: timeliness=NRT-3h sliceNumber=22 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T065354_20231207T065419_051545_0638F1_0A83: timeliness=NRT-3h sliceNumber=23 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T065419_20231207T065444_051545_0638F1_9D53: timeliness=NRT-3h sliceNumber=24 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T065444_20231207T065509_051545_0638F1_C636: timeliness=NRT-3h sliceNumber=25 sliceCount=26 S1A_IW_GRDH_1SDV_20231207T065509_20231207T065528_051545_0638F1_F224: timeliness=NRT-3h sliceNumber=26 sliceCount=26

[johntruckenbrodt]

Thanks a lot @senmao for looking into this and sharing your insight.

[johntruckenbrodt]

In #250 some slice number filtering was introduced. Scenes are grouped by datatake and if some of the scenes have a 0 slice number and others don't, the ones with 0 are removed.
This filtering does only work if the search result does not return two versions with 0 slice number.
It would actually be better to perform this filtering directly during scene selection so that the returns from the various search methods are identical if one database contains more than another. Of course only if all databases contain the version with non-0 slice numbers.