CUDA Detected but using it failed

[wisng]

Been trying to use CUDA, I have 3 versions downloaded 11.8, 12.2 and 12.4.

OS: Windows 10.0.19045 (x64)
Node: 22.11.0 (x64)
TypeScript: 5.4.5
node-llama-cpp: 3.6.0

Error: Binary GPU type mismatch. Expected: cuda, got: false
    at process.onMessage (file://node_modules/node-llama-cpp/dist/bindings/utils/testBindingBinary.js:219:27)
CUDA: CUDA is detected, but using it failed
To resolve errors related to CUDA, see the CUDA guide: https://node-llama-cpp.withcat.ai/guide/CUDA
Vulkan: available

Vulkan device: NVIDIA GeForce RTX 4090
Vulkan used VRAM: 0% (0B/23.58GB)
Vulkan free VRAM: 100% (23.58GB/23.58GB)

CPU model: AMD Ryzen 7 5700G with Radeon Graphics
Math cores: 8
Used RAM: 37.75% (24.12GB/63.89GB)
Free RAM: 62.24% (39.77GB/63.89GB)
Used swap: 50.48% (34.28GB/67.89GB)
Max swap size: 67.89GB
mmap: supported

In the testBindingBinary.js, the gpuType was false.
I also ran npx --no node-llama-cpp source download --gpu cuda, result was the same. Any help would be appreciated.

[wisng]

Theres also an additional message after trying to run the electron project.
[node-llama-cpp] The prebuilt binary for platform "win" "x64" with CUDA support is not compatible with the current system, falling back to using Vulkan

[wisng]

        ntdll.dll => /c/WINDOWS/SYSTEM32/ntdll.dll (0x7ffaec290000)
        KERNEL32.DLL => /c/WINDOWS/System32/KERNEL32.DLL (0x7ffaea950000)
        KERNELBASE.dll => /c/WINDOWS/System32/KERNELBASE.dll (0x7ffae9fa0000)
        ucrtbase.dll => /c/WINDOWS/System32/ucrtbase.dll (0x7ffae9c20000)
        nvcuda.dll => /c/WINDOWS/SYSTEM32/nvcuda.dll (0x7ffa66490000)
        ADVAPI32.dll => /c/WINDOWS/System32/ADVAPI32.dll (0x7ffaea7c0000)
        msvcrt.dll => /c/WINDOWS/System32/msvcrt.dll (0x7ffaea8b0000)
        sechost.dll => /c/WINDOWS/System32/sechost.dll (0x7ffaea2a0000)
        RPCRT4.dll => /c/WINDOWS/System32/RPCRT4.dll (0x7ffaeaeb0000)
        bcrypt.dll => /c/WINDOWS/System32/bcrypt.dll (0x7ffae9b60000)
        MSVCP140.dll => /c/WINDOWS/SYSTEM32/MSVCP140.dll (0x7ffacda50000)
        VCRUNTIME140.dll => /c/WINDOWS/SYSTEM32/VCRUNTIME140.dll (0x7ffacdb90000)
        VCRUNTIME140_1.dll => /c/WINDOWS/SYSTEM32/VCRUNTIME140_1.dll (0x7ffacd190000)
        cudart64_12.dll => /c/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.2/bin/cudart64_12.dll (0x7ffa9b730000)
        CRYPTBASE.DLL => /c/WINDOWS/SYSTEM32/CRYPTBASE.DLL (0x7ffae91d0000)
        ggml-base.dll => /c/git_checkout/llama_cuda/test/node_modules/@node-llama-cpp/win-x64-cuda/bins/win-x64-cuda/ggml-base.dll (0x7ffa70a50000)
        cublas64_12.dll => /c/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.2/bin/cublas64_12.dll (0x7ff9e7c10000)
        cublasLt64_12.dll => /c/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.2/bin/cublasLt64_12.dll (0x7ff9c4950000)

Edit: I noticed the 11.8 folder was still there with the cublas.dll file only. I deleted it, reinstalled node-llama-cpp and these were the updated result.

[giladgd]

The linker resolves seem fine, so that may not be the issue.

It would be helpful if you could do the following:

1. Ensure you have version 3.7.0 installed: npx install node-llama-cpp@latest, npx --no node-llama-cpp --version
2. Open ./node_modules/node-llama-cpp/dist/bindings/utils/testBindingBinary.js
3. On line 195, change LlamaLogLevel.error to LlamaLogLevel.debug
4. Run npx --no node-llama-cpp source clear
5. Run npx --no node-llama-cpp inspect gpu and share its result
6. Change line 14 to return true;
7. Run npx --no node-llama-cpp chat --prompt "Hi there" --debug --gpu cuda hf:mradermacher/Llama-3.2-3B-Instruct-GGUF:Q4_K_M and share its result (please wrap its result with a collapsible since it's very long)

[wisng]

So I restarted my PC, did steps 2-5 (node-llama-cpp was already 3.7.0) and it now says CUDA is available and step 7 is running on CUDA. I haven't done anything in between other than restarting my PC.

OS: Windows 10.0.19045 (x64)
Node: 22.11.0 (x64)
TypeScript: 5.8.2
node-llama-cpp: 3.7.0

ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
load_backend: loaded CUDA backend from C:\git_checkout\llama_cuda\test\node_modules\@node-llama-cpp\win-x64-cuda\bins\win-x64-cuda\ggml-cuda.dll
load_backend: loaded CPU backend from C:\git_checkout\llama_cuda\test\node_modules\@node-llama-cpp\win-x64-cuda\bins\win-x64-cuda\ggml-cpu-haswell.dll
CUDA: available
ggml_vulkan: Found 1 Vulkan devices:
load_backend: loaded Vulkan backend from C:\git_checkout\llama_cuda\test\node_modules\@node-llama-cpp\win-x64-vulkan\bins\win-x64-vulkan\ggml-vulkan.dll
ggml_vulkan: 0 = NVIDIA GeForce RTX 4090 (NVIDIA) | uma: 0 | fp16: 1 | warp size: 32 | shared memory: 49152 | matrix cores: KHR_coopmat
load_backend: loaded CPU backend from C:\git_checkout\llama_cuda\test\node_modules\@node-llama-cpp\win-x64-vulkan\bins\win-x64-vulkan\ggml-cpu-haswell.dll
Vulkan: available

CUDA device: NVIDIA GeForce RTX 4090
CUDA used VRAM: 6.38% (1.53GB/23.99GB)
CUDA free VRAM: 93.61% (22.46GB/23.99GB)

Vulkan device: NVIDIA GeForce RTX 4090
Vulkan used VRAM: 6.38% (1.53GB/23.99GB)
Vulkan free VRAM: 93.61% (22.46GB/23.99GB)

CPU model: AMD Ryzen 7 5700G with Radeon Graphics
Math cores: 8
Used RAM: 48.81% (31.19GB/63.89GB)
Free RAM: 51.18% (32.7GB/63.89GB)
Used swap: 65.83% (44.7GB/67.89GB)
Max swap size: 67.89GB
mmap: supported

[giladgd]

So it seems that CUDA works now with node-llama-cpp on your machine; maybe restarting after uninstalling the older CUDA version changed some of the environment variables that require a restart to come into effect, and that's what solved it.
Anyway, glad you got it working.

I'll document this issue and its solution in the CUDA documentation.

Thanks for helping me resolve this issue :)

[wisng]

Thanks for helping me as well!