Skip to content

vLLM FP8 quantized support for SFT/GRPO #3414

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 28 commits into from
Oct 16, 2025
+495 −7
Merged

vLLM FP8 quantized support for SFT/GRPO #3414

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
6468032
Prefer loading model from pretrained instead of config
Datta0 Oct 1, 2025
e3184a3
Fixup FP8 forward pass and inference
Datta0 Oct 1, 2025
6ef6884
[WIP] Fix lora forwards
Datta0 Oct 1, 2025
51d6626
Infer block size from weight shapes
Datta0 Oct 2, 2025
9888e87
reconstruct weights from fp8 quants for lora matmul
Datta0 Oct 2, 2025
91db140
Return weight transpose and fix dtype
Datta0 Oct 2, 2025
bff4612
Refactor FP8 operations
Datta0 Oct 2, 2025
fb1849c
Fix naming :)
Datta0 Oct 2, 2025
85791f3
Saner compile
Datta0 Oct 3, 2025
4a4f7e2
do not depend on transformers
Datta0 Oct 3, 2025
0b93d94
[WIP] fix training
Datta0 Oct 3, 2025
fb61bf6
Update comment
Datta0 Oct 3, 2025
039fa9d
fixup training
Datta0 Oct 3, 2025
5d6e10b
Merge remote-tracking branch 'origin/main' into vllm_fp8
Datta0 Oct 8, 2025
182e3ce
use dequant kernel from deepseek
Datta0 Oct 9, 2025
c8e7261
Differentiate between fp8 and fbgemmfp8
Datta0 Oct 9, 2025
a3a0a3d
fixup differentiation b/w fp8 and fbgemm_fp8
Datta0 Oct 9, 2025
5603730
make inputs contiguous if required
Datta0 Oct 9, 2025
bfb45b1
Improve dequant
Datta0 Oct 9, 2025
3f277fa
More robust handling
Datta0 Oct 10, 2025
dc4c855
Fixup backward pass for fbgemm_fp8
Datta0 Oct 10, 2025
5b7d755
refactor and use bf16 for dequant
Datta0 Oct 10, 2025
da7d3f9
Use torch fp8 block matmul
Datta0 Oct 11, 2025
5af9f62
Disable torch block matmul for now
Datta0 Oct 12, 2025
5e90163
safer import and cosmetics
Datta0 Oct 14, 2025
80a0449
more cosmectics
Datta0 Oct 15, 2025
dd4bf13
add torchao operations
Datta0 Oct 15, 2025
82c8eef
Spaceeeeeee
Datta0 Oct 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading