Add Humanline #4261
Add Humanline #4261
Conversation
Added humanline logic for log ratio calculation and clipping.
Added a new callback to synchronize the model with a reference model during training.
Added HumanlineSyncRefModelCallback to support humanline synchronization.
added humanline for dpo and kto
|
Thanks @Muennighoff ! Do you have some datasets we can test this on? |
|
Thanks! For GRPO on math, we used just MATH500 (this repo: https://github.com/kawine/open-r1-humanline); Maybe @sijial430 can comment regarding KTO/DPO |
|
For GRPO on instruction following, we use princeton-nlp/llama3-ultrafeedback-armorm mainly (repo: https://github.com/ContextualAI/HALOs/tree/research). Thanks! |
|
To provide some more context, all our math experiments were done in HF's open-r1 repo and were done by subclassing GRPOTrainer. All the major changes are in this file, save for adding some new arguments to configs.py (prefixed with 'humanline') and adding some recipes to the recipes/ folder. The results, as discussed in the paper, are: 
Our instruction-following experiments were done in the research branch of the HALOs repo. 
|
| kl = (policy_KL_logps - reference_KL_logps).mean().detach() | ||
| if self.humanline: | ||
| policy_KL_logps.clamp_(min=self.humanline_log_eps_P, max=self.humanline_log_eps_R) | ||
| reference_KL_logps.clamp_(min=self.humanline_log_eps_P, max=self.humanline_log_eps_R) |
There was a problem hiding this comment.
@sijial430 This is incorrect. The logp ratios should be clamped, not the raw logps.
|
|
||
| @staticmethod | ||
| def get_batch_logps( | ||
| self, |
There was a problem hiding this comment.
@sijial430 get_batch_logps should be kept a static method. humanline should be passed as an argument.
|
@Muennighoff will you open an another PR or just fix this one? |
|
@kashif Thanks! We will reopen a new PR soon |
5 participants
Adds Humanline from https://arxiv.org/abs/2509.24207
cc @kawine @sijial430