Finetuning with LoRA and video training data.

[tungnt55]

I use the following bash script for finetuning model using LoRA. The loss went down to almost zero, however, when I tried to evaluate the model using the exact data set for training, accuracy is only about 60-70%.
Can someone help me pinpointing the problem with settings below?

#!/bin/bash

# You can use 2B instead of 7B
# MODEL_NAME="Qwen/Qwen2-VL-7B-Instruct"
# MODEL_NAME="Qwen/Qwen2-VL-2B-Instruct"
MODEL_NAME="Qwen/Qwen2.5-VL-3B-Instruct"
# MODEL_NAME="Qwen/Qwen2.5-VL-7B-Instruct"

export PYTHONPATH=src:$PYTHONPATH

# GLOBAL_BATCH_SIZE=128
# BATCH_PER_DEVICE=4
# NUM_DEVICES=8

GLOBAL_BATCH_SIZE=128
BATCH_PER_DEVICE=4
NUM_DEVICES=4

GRAD_ACCUM_STEPS=$((GLOBAL_BATCH_SIZE / (BATCH_PER_DEVICE * NUM_DEVICES)))

# If your dataset is mixed with images and videos, you need to use zero2.
CUDA_VISIBLE_DEVICES=4,5,6,7 deepspeed src/train/train_sft.py \
    --use_liger True \
    --lora_enable True \
    --vision_lora True \
    --freeze_llm True \
    --use_dora False \
    --lora_namespan_exclude "['lm_head', 'embed_tokens']" \
    --lora_rank 64 \
    --lora_alpha 64 \
    --lora_dropout 0.05 \
    --num_lora_modules -1 \
    --deepspeed scripts/zero3_offload.json \
    --model_id $MODEL_NAME \
    --data_path /workspace/data/small.json \
    --image_folder /path/to/your/image/folder \
    --remove_unused_columns False \
    --freeze_vision_tower True \
    --freeze_merger False \
    --bf16 True \
    --fp16 False \
    --disable_flash_attn2 False \
    --output_dir output/finetuned_test_4fps \
    --num_train_epochs 50 \
    --per_device_train_batch_size $BATCH_PER_DEVICE \
    --gradient_accumulation_steps $GRAD_ACCUM_STEPS \
    --video_max_pixels $((1280 * 720)) \
    --fps 4 \
    --learning_rate 1e-5 \
    --merger_lr 1e-5 \
    --vision_lr 2e-6 \
    --weight_decay 0.1 \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --tf32 True \
    --gradient_checkpointing True \
    --report_to tensorboard \
    --lazy_preprocess True \
    --save_strategy "steps" \
    --save_steps 5 \
    --save_total_limit 10 \
    --dataloader_num_workers 4 \

[2U1]

Are you measuring the exact token answer or the perplexity?
Also, the model could have problem like "catastrophic forgetting".

[tungnt55]

The output generated text is formatted as a dictionary, something like this:
"{"answer": "high"}".
I measured the accuracy by extracting the answer text ("high", "low", etc) and compare with the ground truth.
Another thing is that, if I train the model without lora (using a bash script similar to finetune_video.sh), then the accuracy of the trained model is 100%.
I used a very small dataset, only 30 samples.

[2U1]

Did you merge the lora weights?
Or it could be a issue that updates to the lora weight was insufficient. You could increase the rank and the alpha

[tungnt55]

I have tried increase rank and alpha to 96. Results are still strange.
At 100 epoch, the loss is 0.01 and the accuracy is 70%.
At 250 epoch, the loss is 0.0 and accuracy is 66%.
I was using the unmerged weights in these tests.

[2U1]

The fact that accuracy falls while loss → 0 is usually a sign that either

1. The adapter weights are not the ones being evaluated, or

2. Generation‐time prompt / decoding / post-processing doesn’t match training, or

3. The adapter capacity is now huge (>90 % of base) but still subject to heavy weight-decay / dropout, so the effective weights oscillate.

You could set the epoch to about 2 epochs.

[tungnt55]

I have double-checked and confirm that the prompt or text processing is not the issue. Also, under the same setting, training without LoRA works OK, the trained model reach 100% accuracy when their train loss is 0.
I also tried again, this time train with 2 epochs.
Epoch 1, loss: 0.36, accuracy: 33%
Epoch 2, loss: 0.16, accuracy: 30%.

[2U1]

You should freeze the merger too.

Qwen2-VL-Finetune/scripts/finetune_lora_vision.sh

Line 17 in d5ebc1a

# You should freeze the the merger also, becuase the merger is included in the vision_tower.

Also If you were using the unmerged weight then check if you have properly loaded the merger(projection) weight.

[tungnt55]

I have tried again with --freeze_merger=True but it didn't change.
Below is the code that I use for loading the unmerged weights. Could you take a quick look and see if there is any issue with it?

model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    base_model_path,
    low_cpu_mem_usage=True,
    torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
    local_files_only=True
)
model.to(device)

non_lora_path = os.path.join(model_path, 'non_lora_state_dict.bin')
non_lora_trainables = torch.load(non_lora_path, map_location='cpu')


non_lora_trainables = {
    (k[11:] if k.startswith('base_model.') else k): v
    for k, v in non_lora_trainables.items()
}
if any(k.startswith('model.model.') for k in non_lora_trainables):
    non_lora_trainables = {
        (k[6:] if k.startswith('model.') else k): v
        for k, v in non_lora_trainables.items()
    }

model.load_state_dict(non_lora_trainables, strict=False)

model = PeftModel.from_pretrained(model, model_path,is_trainable=False)

model.eval()

[2U1]

res = model.load_state_dict(non_lora_trainables, strict=False)
print(f"[non-lora] missing: {len(res.missing_keys)}, unexpected: {len(res.unexpected_keys)}")

print("example missing (merger):", [k for k in res.missing_keys if "visual.merger" in k][:10])

check if the prefixes are correct.

def strip_prefixes(state, prefixes=("base_model.", "model.", "model.model.")):
    out = {}
    for k, v in state.items():
        kk = k
        for p in prefixes:
            if kk.startswith(p):
                kk = kk[len(p):]
        out[kk] = v
    return out

non_lora_trainables = strip_prefixes(non_lora_trainables)
res = model.load_state_dict(non_lora_trainables, strict=False)
print(f"[non-lora] missing: {len(res.missing_keys)}, unexpected: {len(res.unexpected_keys)}")

This is the safe one.

[tungnt55]

Thank you for sending me the code to check it.
I tried printing out res.missking_keys and res.unexpected_keys here:
lora_load_state_dict.log
I modified the function strip_prefixes() a little bit to fix the issue of missing keys, after fixing, only 'lm_head.weight' remains missing.
However, even after this fix, the accuracy on same dataset remains low when the loss is also very low (close to zero) in training.