Training/Output stops without error

[DannyPerspective]

When i do a training on the s3dis dataset, after same epochs the output just stops.

I start the training with:
CUDA_VISIBLE_DEVICES=0 nohup python examples/segmentation/main.py --cfg cfgs/s3dis/pointnext-s.yaml > outputlog 2>&1 &

Sometimes it already stops in the first epoch, sometimes it runs for five-sixes epochs but eventually it stops.

The python process for the training is then still alive (status “sleeping”) and the hardware resources are also still allocated.
Can someone help?

output.log

[Saleem-BIM]

Hello @DannyPerspective,
did you solved your issue or still figureíng out? I am facing similar problem

[DannyPerspective]

Sorry, I could not figure it out. I had to restart training from the latest checkpoint every time it happened.

[Saleem-BIM]

I am getting : ModuleNotFoundError: No module named 'openpoints'. have you faced similar issue in your training stage?

[DannyPerspective]

Seems like the install.sh script hasn't worked properly. Have you executed it?
But that’s is an entirely different issue than this here.

[Saleem-BIM]

no I tried to install.sh agin but the issue still persist

[DannyPerspective]

If there is no folder called “openpoints” in your PointNeXt folder, it didn’t work. Try executing the commands in the install.sh one by one and look if you can find any errors.