Vulkan crash on AMD RDNA1 (RX 5500 XT) during buffer initialization

[that-guy-rob]

Summary

whisper.cpp crashes with VK_ERROR_DEVICE_LOST on AMD RX 5500 XT (RDNA1/Navi14) when using Vulkan backend. The crash occurs during KV cache initialization before any inference happens.

Environment

• GPU: AMD Radeon RX 5500 XT (gfx1012, NAVI14, 8GB VRAM)
• Driver: RADV (Mesa 24.2.8-1ubuntu1~24.04.1)
• Vulkan: 1.3.289
• OS: Linux Mint 22.1 (Ubuntu 24.04 based), kernel 6.8.0
• whisper.cpp: Latest master (commit f53dc74)

Steps to Reproduce

git clone https://github.com/ggml-org/whisper.cpp
cd whisper.cpp
cmake -B build -DGGML_VULKAN=1
cmake --build build -j
./models/download-ggml-model.sh base.en
./build/bin/whisper-cli -m models/ggml-base.en.bin -f samples/jfk.wav

Expected Behavior

Whisper transcribes the audio using the Vulkan GPU backend.

Actual Behavior

Crash with error:

radv/amdgpu: The CS has been rejected, see dmesg for more information (-22).
terminate called after throwing an instance of 'vk::DeviceLostError'
  what():  vk::Queue::submit: ErrorDeviceLost

Stack Trace

#6  ggml_vk_submit(std::shared_ptr<vk_context_struct>&, vk::Fence)
#7  ggml_vk_buffer_memset(std::shared_ptr<vk_buffer_struct>&, unsigned long, unsigned int, unsigned long)
#8  whisper_kv_cache_init(whisper_kv_cache&, ggml_backend*, ggml_type, long, long, int)
#9  whisper_init_state()
#10 whisper_init_from_file_with_params()

Root Cause Analysis

The crash occurs in ggml_vk_buffer_memset() at ggml-vulkan.cpp:6588:

subctx->s->buffer.fillBuffer(dst->buffer, offset, size, c);

This function uses the transfer queue (dst->device->transfer_queue.cmd_pool) to execute vkCmdFillBuffer. On RDNA1, the transfer queue (SDMA engine) appears to reject this command with EINVAL (-22).

Key observations:

1. The GPU is correctly detected as RDNA1: ggml_vulkan: 0 = Radeon RX 5500 XT (RADV NAVI14)
2. The crash happens before any compute shaders run
3. Error code -22 (EINVAL) suggests invalid parameters or unsupported operation
4. CPU mode works fine with --no-gpu

Attempted Workarounds (all failed)

• --no-flash-attn - still crashes
• GGML_VK_DISABLE_ASYNC=1 - still crashes
• GGML_VK_PREFER_HOST_MEMORY=1 - still crashes (not UMA, so still uses GPU path)
• VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation - still crashes
• RADV_PERFTEST=transfer_queue=0 - still crashes

Potential Fix

The issue may be that fillBuffer on the dedicated transfer queue doesn't work correctly on RDNA1. Possible solutions:

1. Use compute queue for fillBuffer on RDNA1:
   Modify ggml_vk_buffer_memset() to use compute_queue instead of transfer_queue when device->architecture == vk_device_architecture::AMD_RDNA1

2. Add CPU fallback for non-UMA discrete GPUs:
   The current code only uses CPU memset when eHostVisible && uma. For discrete GPUs, could allocate a staging buffer and use CPU memset + copy.

3. Use vkCmdUpdateBuffer instead:
   For small buffers, vkCmdUpdateBuffer might work better on the transfer queue.

Related Issues

• Similar RADV crashes on RDNA1: https://github.com/felipeagc/amd_repro
• whisper.cpp AMD issues: How to use with a AMD GPU? #2828
• Vulkan broken for AMD in v1.8.0+: Vulkan support broken at v1.8.0 #3455

System Info

$ vulkaninfo --summary | grep -A5 "GPU0"
GPU0:
    apiVersion         = 1.3.289
    driverVersion      = 24.2.8
    vendorID           = 0x1002
    deviceID           = 0x7340
    deviceType         = PHYSICAL_DEVICE_TYPE_DISCRETE_GPU
    deviceName         = Radeon RX 5500 XT (RADV NAVI14)

Note

• This appears to be specific to the SDMA/transfer queue on RDNA1

[that-guy-rob]

---

UPDATE: Solution Found!

After extensive debugging, the issue was resolved with a kernel boot parameter.

Solution

Add iommu=soft to your kernel boot parameters:

sudo nano /etc/default/grub

Change: GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"

To: GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=soft"

sudo update-grub
sudo reboot

Results After Fix

Whisper.cpp now works with Vulkan on RX 5500 XT:
ggml_vulkan: 0 = Radeon RX 5500 XT (RADV NAVI14) (radv)
[00:00:00.000 --> 00:00:11.000] And so my fellow Americans, ask not what your country can do for you...

whisper_print_timings: encode time = 285.56 ms / 1 runs
whisper_print_timings: total time = 2522.02 ms

11 seconds of audio processed in 2.5 seconds (~4.4x realtime) - GPU acceleration working!

What We Tried (Before Finding the Solution)

• Patching ggml_vk_buffer_memset() to use compute queue - didn't help
• Staging buffer with CPU memset + vkCmdCopyBuffer - didn't help
• Forcing single_queue mode - got past init but crashed during inference
• Upgrading Mesa from 24.2.8 to 25.0.7 - didn't help
• Various env vars (GGML_VK_DISABLE_ASYNC, etc.) - didn't help

Root Cause

The -22 (EINVAL) error was caused by Address Translation Services (ATS) issues between the AMDGPU kernel driver and RDNA1. The iommu=soft parameter uses software IOMMU translation, which bypasses this issue.

This may affect other RDNA1 GPUs (RX 5300, 5500, 5600, 5700 series) with the RADV driver.

---