[Task 4] Dataset error "Test and eval sets have different amount of files [14182 != 0]"

[lcances]

Hi,

I tried to run again the baseline code for the task 4 and after checking missing files and trying to download them I'm hit with the following error :

[I] DCASE2018 / Task4
[I] Download_data                                                                                                                 
[I] Once database is downloaded, do not forget to check your missing_files
[I] check files exist or download data
100%|█████████████████████████████████████████████████████████████████████████████████████████████| 69/69 [00:24<00:00,  2.85it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████| 603/603 [02:48<00:00,  3.57it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████| 13/13 [00:08<00:00,  1.51it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 1003/1003 [04:49<00:00,  3.47it/s]
[I]   DONE       
[I] 
[E] Dataset errors:
FOLD[1]          EVAL SET    Empty set             
FOLD[1]          SETS          Test and eval sets have different amount of files [14182 != 0].
FOLD[1]          SETS          Test and eval sets have different files [0 != 14182 files common].       (datasets.py:1304)
NoneType: None
Dataset errors:
FOLD[1]          EVAL SET    Empty set             
FOLD[1]          SETS          Test and eval sets have different amount of files [14182 != 0].
FOLD[1]          SETS          Test and eval sets have different files [0 != 14182 files common].

I tried to run the baseline again because I wanted to save the output of the second model (which represent the 10 class prediction over the time) for the small test dataset (288 files).
I tried to download all the data again within a fresh repository but, with the current version of repo I can't go further than the download.

Do you know where the problem could come from ?

Or it is possible to download somewhere the output of the second model (on the test dataset) ?

PS : (I dont really mind if there is some missing files)

[turpaultn]

Hi Leo,

Here is the output on the test dataset of the baseline.
res_test.txt

I don't know where the problem is coming from. I'm investigating and try to reproduce the error.

To help me please:
Could you tell me your version of dcase_util please ?
Also could you check how many files you actually have in each dataset folders ?
Did you try to remove the "evaluation_setup" folder and "filelist.python.hash" before running it again?

Then, what surprises me is the fact the error says the test and eval set have 14182 files which is probably all files except unlablel_out_of_domain in your case

[lcances]

Thank for your fast reply.

Could you tell me your version of dcase_util please ?

• dcase_util version : 0.2.5

Also could you check how many files you actually have in each dataset folders ?

• Weak : 1509 files
• Unlabel in domain : 13 809 files
• Unlabel out of domain : 38 996 files
• Test : 275 files

Did you try to remove the "evaluation_setup" folder and "filelist.python.hash" before running it again?

• I just did and nothing changes but the amound of files mentioned :

[I]   DONE       
[I] 
[E] Dataset errors:
FOLD[1]          EVAL SET    Empty set             
FOLD[1]          SETS          Test and eval sets have different amount of files [13809 != 0].
FOLD[1]          SETS          Test and eval sets have different files [0 != 13809 files common]. 	(datasets.py:1304)
NoneType: None
Dataset errors:
FOLD[1]          EVAL SET    Empty set             
FOLD[1]          SETS          Test and eval sets have different amount of files [13809 != 0].
FOLD[1]          SETS          Test and eval sets have different files [0 != 13809 files common].

Thank for the file also, However, I'm currently investigating some segmentation algorithm and I wanted to apply those on the output of the last layer ( TimeDistributed Dense ). I then still need to run the baseline again.

I will be very pleased to help you in any way,

[turpaultn]

Ok I will try to reproduce the error and solve it.

In the meantime, I can give you the model weights, send me an email and I'll share it with you because github doesn't allow me to put it here.

If other people want it, I will put them in the repo.

[turpaultn]

I found the problem, it comes from dcase_util 0.2.5 making some verification about folds.
To make this code compatible with dcase_util I used some tricks which are not possible with the new version of dcase_util. Please, use dcase_util 0.2.4.

[lcances]

Thank you for your work, I'm sorry to answer so late, but I was quite busy.

I will give it a try during the week