QLORA FSDP doesn't work with 8bit

[ojh31]

System Info

• Accelerate version: 1.10.1
• Platform: Linux-5.15.0-157-generic-x86_64-with-glibc2.39
• accelerate bash location: /opt/uv/venv/bin/accelerate
• Python version: 3.12.11
• Numpy version: 2.2.6
• PyTorch version: 2.9.0.dev20250825+cu128
• PyTorch accelerator: CUDA
• System RAM: 1003.13 GB
• GPU type: NVIDIA H100 80GB HBM3
• Accelerate default config:
  Not found
• peft version: 0.15.2
• transformers version: 4.56.1

Who can help?

@BenjaminBossan @githubnemo

Reproduction

Add --use_8bit_quantization True to peft/examples/sft/run_peft_qlora_fsdp.sh

accelerate launch --config_file "configs/fsdp_config_qlora.yaml"  train.py \
--seed 100 \
--model_name_or_path "meta-llama/Llama-3.1-8B-Instruct" \
--dataset_name "smangrul/ultrachat-10k-chatml" \
--chat_template_format "chatml" \
--add_special_tokens False \
--append_concat_token False \
--splits "train,test" \
--max_seq_len 2048 \
--num_train_epochs 1 \
--logging_steps 5 \
--log_level "info" \
--logging_strategy "steps" \
--eval_strategy "epoch" \
--save_strategy "epoch" \
--bf16 True \
--packing True \
--learning_rate 1e-4 \
--lr_scheduler_type "cosine" \
--weight_decay 1e-4 \
--warmup_ratio 0.0 \
--max_grad_norm 1.0 \
--output_dir "llama-sft-qlora-fsdp" \
--per_device_train_batch_size 2 \
--per_device_eval_batch_size 2 \
--gradient_accumulation_steps 2 \
--gradient_checkpointing True \
--use_reentrant True \
--dataset_text_field "content" \
--use_flash_attn True \
--use_peft_lora True \
--lora_r 8 \
--lora_alpha 16 \
--lora_dropout 0.1 \
--lora_target_modules "all-linear" \
--use_4bit_quantization True \
--use_8bit_quantization True

Expected behavior

Training should complete, instead we get ValueError: Cannot flatten integer dtype tensors

[BenjaminBossan]

Thanks for the report. Indeed, currently the script raises the error you mentioned, both with 4bit and 8bit quantization.

I tried some of my earlier scripts that I know used to work and they fail with the same error. So there at some point, some change in PEFT/transformers/accelerate/trl/torch broke this script. I tried to do a few quick checks with different versions, but as they are often mutually incompatible, it's too much work to figure out what change broke the script.

Still, when I have a bit of time on my hands, I'll check what the reason could be and hopefully provide a fix. In the meantime, if DeepSpeed is an option for you, you could try that instead of FSDP.

NB: There is a bug in the script here (which I swear I fixed it at one point but apparently never pushed the change):

peft/examples/sft/utils.py

Lines 100 to 119 in ec5a1b2

	if args.use_4bit_quantization:
	compute_dtype = getattr(torch, args.bnb_4bit_compute_dtype)
	quant_storage_dtype = getattr(torch, args.bnb_4bit_quant_storage_dtype)
	bnb_config = BitsAndBytesConfig(
	load_in_4bit=args.use_4bit_quantization,
	bnb_4bit_quant_type=args.bnb_4bit_quant_type,
	bnb_4bit_compute_dtype=compute_dtype,
	bnb_4bit_use_double_quant=args.use_nested_quant,
	bnb_4bit_quant_storage=quant_storage_dtype,
	)
	if compute_dtype == torch.float16 and args.use_4bit_quantization:
	major, _ = torch.cuda.get_device_capability()
	if major >= 8:
	print("=" * 80)
	print("Your GPU supports bfloat16, you can accelerate training with the argument --bf16")
	print("=" * 80)
	elif args.use_8bit_quantization:
	bnb_config = BitsAndBytesConfig(load_in_8bit=args.use_8bit_quantization)

The last two lines that check 8bit should be unindented one level, otherwise they won't be reached if a user passes use_4bit_quantization=False and use_4bit_quantization=True. In the example above, both arguments are set to True, so the end result is correct, but it's confusing that both have to be enabled for 8bit to work.

[ojh31]

Thanks @BenjaminBossan for the quick reply!

I agree with all of this:

The last two lines that check 8bit should be unindented one level, otherwise they won't be reached if a user passes use_4bit_quantization=False and use_4bit_quantization=True. In the example above, both arguments are set to True, so the end result is correct, but it's confusing that both have to be enabled for 8bit to work.

To clarify, I do not get the error with 4bit:

currently the script raises the error you mentioned, both with 4bit and 8bit quantization.

[BenjaminBossan]

To clarify, I do not get the error with 4bit:

Interesting, could you please share the command you use for 4bit? Also, what bitsandbytes version are you using?

[ojh31]

To clarify, I do not get the error with 4bit:

Interesting, could you please share the command you use for 4bit? Also, what bitsandbytes version are you using?

Running peft/examples/sft/run_peft_qlora_fsdp.sh unchanged seems to work for me.

bitsandbytes version: 0.47.0

[BenjaminBossan]

I had the opportunity to run some experiments today. I could indeed make FSDP QLoRA work with 4bit bnb but not with 8bit bnb. The int8 parameters obviously cause problems, as FSDP requires the same dtype to shard the weights. It would be possible to ignore all int8 quantized weights, but then there is no point really in using FSDP.

I was hoping to be able to load these modules that are targeted with LoRA in float instead of int8 by using llm_int8_skip_modules. If only a small part of all modules is targeted with LoRA, this would still yield nice memory benefits. Unfortunately, there appears to be a bug in bitsandbytes though when using llm_int8_skip_modules, so that also didn't work. Until that is fixed, the only option might be to use 4bit bnb.

As to why I thought that int8 used to work, I'm not sure. Possibly, it never worked and I only thought it did. There is a bit of a trap in the sft example because these lines are indented one level too far:

peft/examples/sft/utils.py

Lines 118 to 119 in d0fa974

	elif args.use_8bit_quantization:
	bnb_config = BitsAndBytesConfig(load_in_8bit=args.use_8bit_quantization)

It means that if use_4bit_quantization=False and use_8bit_quantization=True is set, there is actually no quantization being applied.

[BenjaminBossan]

@ojh31 I could not get FSDP + 8bit bnb to work, so I might just have misremembered. In fact, I think it never has, as it's a limitation in bitsandbytes (see e.g. this comment). I updated the docs in #2857 to indicate this. There isn't really more that can be done here, so unless you have more comments or questions, I'd suggest to close the issue.