Fix the device error when using ray as vllm-acend backend

Fix the device error when using ray as vllm-acend backend #3672