GRPO run bugs

[superluckyxhh]

Hello, now I'm encountering a situation like this: when running the finetune_grpo.sh script and loading the model in train_grpo.py, if I don't load anything from bnb_model_from_pretrained_args, the script won't report any error and will directly stop the process. log is "Killing subprocess 4187820". After adding the bnb_model_from_pretrained_args parameter, it runs normally. May I ask what the problem is? I tried many versions of torch but none of them could solve this problem.

[2U1]

The message

Killing subprocess XXXXX

is emitted by the Linux OOM‑killer (or the NVIDIA driver) when the model can’t fit in GPU / CPU memory during from_pretrained.
When you leave bnb_model_from_pretrained_args empty the model loads in full‑precision (FP16/BF16) and exhausts VRAM/RAM, so the kernel kills the subprocess before PyTorch can raise an exception.
Adding bnb_model_from_pretrained_args enables 4‑/8‑bit Bits‑and‑Bytes quantization plus a device_map; the weights are much smaller and spread across devices, so the load completes.

[superluckyxhh]

It doesn't seem to be for this reason. I have 8 A800 Gpus here and I'm running a 3B model. The video memory is definitely sufficient. There are still two situations now:

1. When deepspeed uses the zero3 series, an error will be reported: ValueError: DeepSpeed Zero-3 is not compatible with passing a 'device_map'. The solution is to annotate device_map.
2. Even when using the bnb config, the problem of no error and direct stop occurs when running GRPO. I suspect it's a version issue. The version I'm using is the one provided in e nv.yaml. Here are some logs. I wonder if they are helpful in solving this problem.
   [2025-07-28 17:26:20,544] [INFO] [real_accelerator.py:239:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2025-07-28 17:26:22,568] [INFO] [comm.py:669:init_distributed] cdb=None You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with model.to('cuda'). [2025-07-28 17:26:27,096] [INFO] [launch.py:319:sigkill_handler] Killing subprocess 42444

[2U1]

It runs fine on my env. Could you use the docker image I've uploaded?
Or use the flash-attetion verion
flash-attn == 0.27.4 post1

[superluckyxhh]

It seems there is no version you mentioned
ERROR:Could not find a version that satisfies the requirement flash-attn==0.27.4 (from versions: 0.2.0, 0.2.1, 0.2.2, 0.2.3, 0.2.4, 0.2.5, 0.2.6.post1, 0.2.7, 0.2.8, 1.0.0, 1.0.1, 1.0.2, 1.0.3.post0, 1.0.4, 1.0.5, 1.0.6, 1.0.7, 1.0.8, 1.0.9, 2.0.0.post1, 2.0.1, 2.0.2, 2.0.3, 2.0.4, 2.0.5, 2.0.6, 2.0.6.post2, 2.0.7, 2.0.8, 2.0.9, 2.1.0, 2.1.1, 2.1.2.post3, 2.2.0, 2.2.1, 2.2.2, 2.2.3.post2, 2.2.4, 2.2.4.post1, 2.2.5, 2.3.0, 2.3.1.post1, 2.3.2, 2.3.3, 2.3.4, 2.3.5, 2.3.6, 2.4.0.post1, 2.4.1, 2.4.2, 2.4.3.post1, 2.5.0, 2.5.1.post1, 2.5.2, 2.5.3, 2.5.4, 2.5.5, 2.5.6, 2.5.7, 2.5.8, 2.5.9.post1, 2.6.0.post1, 2.6.1, 2.6.2, 2.6.3, 2.7.0.post2, 2.7.1.post4, 2.7.2.post1, 2.7.3, 2.7.4.post1, 2.8.0.post2, 2.8.1, 2.8.2)

[superluckyxhh]

What puzzles me is that, for instance, when loading the model like this, the program will be automatically killed.
model = Qwen2_5_VLForConditionalGeneration.from_pretrained( model_args.model_name_or_path, torch_dtype=compute_dtype, attn_implementation=model_args.attn_implementation )
Adding the bnb parameter will prevent it from being killed, but the peft model must be used for fine-tuning.
Another approach is to remove deepspeed, but if this is done, training the 3B model will take up all the memory of the 8 A800 cards, resulting in OOM.

[2U1]

That's becuase the when using QLoRa the device_map='auto' is given.
You could add a paraemeter for some temporal fix.