[flashinfer][fix] do not check nvcc availability #27990
Conversation
Summary: vllm-project#26443 adds checking of availability of nvcc as a condition to enable flashinfer moe. On devgpus, we may have nvcc so there is no issue. But in tw jobs, there is no nvcc, then flashinfer moe is disabled. Differential Revision: D86104899
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request aims to enable FlashInfer in environments where nvcc is not available by removing the nvcc availability check. While this addresses the issue for environments with pre-compiled kernels, it could introduce runtime crashes for users who lack both nvcc and pre-compiled kernels. I've suggested a safer alternative that makes the nvcc check conditional on the VLLM_HAS_FLASHINFER_CUBIN environment variable. This approach provides the desired flexibility for production environments while preserving the safeguard for other users.
|
@mgoin our internal prod environment uses flashinfer in an AOT fashion, and do not have nvcc. So right now we are seeing flashinfer moe being disabled internally, causing perf regression. |
|
Is there a way we can add unit tests to ensure this doesn't get turned off accidentally again for the model? |
mxz297
changed the title
|
Is nvcc required by the jit compilation of FlashInfer? |
3 participants
Summary: #26443 adds checking of availability of nvcc as a condition to enable flashinfer moe. In our deployment env, there is no nvcc, so flashinfer moe is disabled
Differential Revision: D86104899