Request for SFT and DPO Hyperparameter Configurations for Qwen2-7B and Other Models

[jianghoucheng]

Hi,

Thanks for your great work!

Would it be possible for you to provide the recommended or default hyperparameter settings for other models like Qwen2-7B, and ideally for both SFT and DPO, on different datasets

[zwhong714]

• For SFT models, the training scripts are the same. You can refer to examples/train_full/qwen2_full_sft_ds3.yaml. The learning rate is set to 2e-6.

• For DPO models, you need to adjust the pref_beta (values can be chosen from {0.001, 0.01, 0.005, 0.05, 0.1, 0.5, 1.0, 2.0, 3.0}) and the learning rate, as we found these two hyperparameters significantly affect the performance of DPO training.

DPO Model	$\beta$	Learning Rate
Qwen2-0.5B-Instruct	0.5	1e-5
Qwen2-1.5B-Instruct	0.1	1e-5
Qwen2-7B-Instruct	2.0	1e-5
Qwen2-1.5B-SFT (base model trained on UltraChat200k)	0.1 (tentative)	1e-5
Qwen2-7B-SFT (base model trained on UltraChat200k)	0.1 (tentative)	1e-5

We provide our trained models as follows:

Model Name	Download Link
🤗 qwen2_1.5B-ultrachat200k-sft	[Link](https://huggingface.co/wh-zhu/qwen2_1.5B-ultrachat200k)
🤗 qwen2_7B-ultrachat200k-sft	[Link](https://huggingface.co/wh-zhu/qwen2_7B-ultrachat200k)
🤗 qwen2_1.5B-ultrachatfeedback-dpo	[Link](https://huggingface.co/wh-zhu/qwen2_1.5B-ultrachatfeedback-dpo)
🤗 qwen2_7B-ultrachatfeedback-WSPO	[Link](https://huggingface.co/wh-zhu/qwen2_7B-ultrachatfeedback-wspo)

Model Name	Download Link
🤗 DeepSeek-R1-Distill-Qwen-1.5B	[Link](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B)
🤗 DeepScaleR-1.5B-Preview	[Link](https://huggingface.co/agentica-org/DeepScaleR-1.5B-Preview)
🤗 DeepSeek-R1-Distill-Qwen-7B	[Link](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B)
🤗 DeepScaleR-7B-WSPO	[Link](https://huggingface.co/wh-zhu/DeepScaleR-7B-WSPO)