FarmReader train is getting blocked for ever

[bappctl]

Environment:
torch: 2.0.1+cu118

Haystack version:
farm-haystack[all-gpu]==1.20.1

Issue:
reader.train(data_dir=<data_dir>, train_filename=<train_filename>, use_gpu=<use_gpu>, n_epochs=10, save_dir=<save_dir>)

getting blocked for ever and not exiting (not sure it was a successful train or not though the model get created in save_dir).

[julian-risch]

Hi @bappctl thank you for reporting this. It looks like the problem occurs at the very end of the training, at the end of the final training epoch. You could try to set the num_processes parameter in the train method to 1. Another idea is about the torch version. In the past there were deadlock issues from time to time depending on torch versions. torch version 2.1 was released just a few days ago. Maybe that resolves the issue.