You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Correct the inaccurate docstring for int4_row_quantize()
Summary:
X-link: facebookresearch/FBGEMM#1930
When I was reading the code in `quantize.py`, the docstring in int4_row_quantize() seems inaccurate.
The `int4_row_quantize()` function returns `wq` with shape `[N, K]`, not `[N, K // 2]`:
```
# Line 101: Concatenate chunks back to original K dimension
out = torch.cat(out, dim=-1)
# Line 104: Convert to int8 dtype
out = out.to(dtype=torch.int8)`
```
So `wq` is `[N, K]` stored as `int8` elements, where each `int8` element contains a single int4 value. And after it, the actual packing could be done via:
https://www.internalfb.com/code/fbsource/[c12bfdd174f7897f4615f98b630f2ac612c471fa]/fbcode/deeplearning/fbgemm/fbgemm_gpu/experimental/gen_ai/gen_ai/quantize.py?lines=18
Differential Revision:
D82919480
Privacy Context Container: 151967047006994
0 commit comments