vLLM FP8 quantized support for SFT/GRPO #3414
Merged
+495
−7
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
unsloth/kernels/fp8.py
Outdated
| del W_deq | ||
| return grad_X, None, None | ||
|
|
||
| @torch.compile |
There was a problem hiding this comment.
Can you check if torch.compile(fullgraph = True, dynamic = True) works better.
Also try using:
from unsloth_zoo.temporary_patches.common import torch_compile_options, torch_compile
@torch_compile
def ...There was a problem hiding this comment.
I noticed no performance difference between the three when trying out Qwen3-8B between any of the 3
| if weight_fake_quantizer is not None: | ||
| W = weight_fake_quantizer(W) | ||
|
|
||
| W_quant = next((x for x in [getattr(W, "quant_state", None), getattr(base_layer, "weight_scale_inv", None), getattr(base_layer, "weight_scale", None)] if x is not None), None) |
There was a problem hiding this comment.
Tbh best to make an if elif to make it faster
There was a problem hiding this comment.
My only worry is someone mistakenly changing when I add if..else cuz if tensor would fail when tensor exists.
one needs to explicitly do if tensor is not None or something like that
I thought this is a safer way to let people continue this/avoid that
But can change if you feel its better that way
danielhanchen
requested changes
Oct 15, 2025
| if weight_fake_quantizer is not None: | ||
| W = weight_fake_quantizer(W) | ||
|
|
||
| W_quant = next((x for x in [getattr(W, "quant_state", None), getattr(base_layer, "weight_scale_inv", None), getattr(base_layer, "weight_scale", None)] if x is not None), None) |
Datta0
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Depends on unslothai/unsloth-zoo#313