Performance discrepancies between Full Fine-Tuning and LoRA Fine-Tuning #463
Comments
|
Adding an experiment from our slack thread for completeness. Could not reproduce the problem with the below setup, which could mean RCA should involve accelerate_launch.py usage and knowledge of checkpoint creation time for each of the methods. ResultsTorchrun
accelerate launch
Above numbers align with theoretical understanding of throughputs (lora should be slightly higher than full finetuning) and what was reported in lora paper. Torchrun command for loraTorchrun command for full finetuningAccelerate for loraAccelerate for full finetuningSetup instructionsfms-hf-tuning version used v2.2.1. Use a clean conda env. The InfraGPUs - 8 X A100 80 GB Platform |
|
I want to add a few more bits of information that I experienced in the past and I observed recently too. When launching with Models other than @kmehant @anhuong I would highly appreciate if you could provide me any feedback about this. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
For the past few months I have been testing and assessing the performance of fms-hf-tuning by fine-tuning different large models (Full Fine-Tuning, LoRA Fine-Tuning and QLoRA Fine-Tuning). When launching Full Fine-Tuning and LoRA Fine-Tuning using the standalone accelerate_launch.py script, Full Fine-Tuning is faster than LoRA Fine-Tuning (exact same settings) and the memory savings are not as high as one would expect.
However, launching the training using
torchrunrather thanaccelerate(which is what the launch script uses under the hood) seems to report better performance in both runtime and GPU memory usage. Although with fairly simple settings, just a handful of models are able to finish Full Fine-Tuning without getting OOMs.I have been fine-tuning a wide variety of models:
However, I have focused in
ibm-granite/granite-3b-code-instructsince it was the one presenting the largest difference in training runtime.Platform
All the experiments have been executed in a Red Hat OpenShift cluster (4.16) with the RHOAI operator enabled (2.16). The fms-hf-tuning image version was v2.2.1.
Settings
The general settings for Full Fine-Tuning are:
with a few additions for LoRA Fine-Tuning:
When running the experiments using the accelerate_launch.py script, my
acceleratesettings were:And during the testing, I have added target modules to target other attention layers as well as modifying batch sizes (all of this was reported via internal Slack discussions, although it did not alter the initial observations).
When running the experiments using
torchrundirectly, I used the following command:python -m torch.distributed.run \ --nproc_per_node=8 \ --nnodes=1 \ --node_rank=0 \ --master_addr=localhost \ --master_port=29500 \ --module tuning.sft_trainer \ --fsdp_auto_wrap_policy=TRANSFORMER_BASED_WRAP \ --fsdp_backward_prefetch=BACKWARD_POST \ --fsdp_forward_prefetch=False \ --fsdp_offload_params=False \ --fsdp_state_dict_type=FULL_STATE_DICT \ --fsdp_sync_module_states=True \ --fsdp_use_orig_params=FalseExpected behavior
The expected behaviour would be that for the same model and exact same settings, LoRA Fine-Tuning should have a lower training runtime and a lower GPU memory usage.
Observed behavior
When launching Full Fine-Tuning and LoRA Fine-Tuning using the standalone accelerate_launch.py script, Full Fine-Tuning is faster than LoRA Fine-Tuning (exact same settings) and the memory savings are not as high as one would expect.
This are the results for the
ibm-granite/granite-3b-code-instructmodel using accelerate_launch.py:And this are the results using
torchrunfor the same model:Additional context
We have been discussing this internally via Slack for a few weeks so feel free to reach out for more context or data.
The text was updated successfully, but these errors were encountered: