Confusion about finetune_cls.sh

[TUDelftHao]

Thank you for this amazing project!

While I am going to conduct the QwenVL classification task, I have some questions about the parameters used in finetune_cls.sh:

1. What is the global batch size used for the default learning rate?
2. Why is the head_lr different from learning_rate? Is this the empirical set？
3. Why are you setting the llm frozen while the vision tower and merger module trainable, which is in contrast to the setup of sft the mllm (usually we only finetune the LLM part).

[2U1]

First of all the setting is not the recommend one. It's a experimental feature so I've just uploaded the settings with the config that I was experimenting.

1. I've used global batch 64 for the training for the experiment.
2. head_lr specifies the the learning rate for the head and, the learning rate are used for all of the parameters that aren't specified. (If you set, merger and vision too, then it will only be applied to the llm part).
3. I was training the data with some specific strategy so, I've freezed the llm.

In my experience. Unfreezing the llm part will impact the performance, so if you have some time and resource, it would be better to tune the llm.
In some cases, after tuning all of the parameter, freezing the llm and vision and tuning only the merger and head could enhance the performacne.

[TUDelftHao]

First of all the setting is not the recommend one. It's a experimental feature so I've just uploaded the settings with the config that I was experimenting.

1. I've used global batch 64 for the training for the experiment.
2. head_lr specifies the the learning rate for the head and, the learning rate are used for all of the parameters that aren't specified. (If you set, merger and vision too, then it will only be applied to the llm part).
3. I was training the data with some specific strategy so, I've freezed the llm.

In my experience. Unfreezing the llm part will impact the performance, so if you have some time and resource, it would be better to tune the llm. In some cases, after tuning all of the parameter, freezing the llm and vision and tuning only the merger and head could enhance the performacne.

Thank you so much for the helpful reply!

[2U1]

@TUDelftHao I've had some research and found that mlp could help when doing classification task using models like Qwen.
I've made some options and unfreezing only some of the layers. So you could try various options for it.

For making a baseline I would recommend unfreezing all the layer and use single layer head. Use cross_entropy for the loss.

    --learning_rate 8e-6 \
    --head_lr 1.6e-5 \
    --vision_lr 1.6e-6 \
    --merger_lr 1.6e-6 \

This could be a good start when using adamw_bnb_8bit optimizer. When you are using adamaw_torch_fused or default adam, then increase the lrs about 20%.