Improve LoRA Compatibility by Renaming `value_proj` and `query_proj` in `PaliGemmaVitAttention`

[b05505027]

Hi
I was working on my project to finetune PaliGemma model and noticed that the attention layers inside paligemma.backbone.vit_encoder are not being LoRA enabled when calling:

paligemma.backbone.enable_lora(rank=LORA_RANK)

(where LORA_RANK is an integer value representing the LoRA rank).

I found that the PaliGemmaVitAttention class uses names like value_proj and query_proj for its query and value layers. However, the enable_lora function defined in the Backbone class only enables layers named as value_dense, query_dense, query, or value.

To ensure consistency and enable LoRA properly for these layers, I propose renaming:

• value_proj → value_dense
• query_proj → query_dense

in the PaliGemmaVitAttention's query and value layers. This ensures that the enable_lora function correctly applies LoRA modifications to the ViT encoder layers without requiring additional modifications to the backbone class.

If there are alternative naming preferences that better fit the project’s conventions, I’m happy to adjust accordingly. Thanks!

[Gopi-Uppari]

Hi @b05505027,

Thank you for your detailed analysis, if your proposal aligns well with the existing enable_lora function, please go ahead and raise a pull request for the changes. We'll review and test them to ensure compatibility.

Thank you.

[abheesht17]

@b05505027 - thanks for bringing this up! If we rename layers, we will have to re-upload checkpoints on Kaggle. I think the better thing to do is override the enable_lora method here: https://github.com/keras-team/keras-hub/blob/v0.18.1/keras_hub/src/models/backbone.py#L194 in the PaliGemmaBackbone class, and change target_names to ["query_dense", "value_dense", "query", "value", "query_proj", "value_proj"].

Alternatively, we can add target_names as an arg to enable_lora, which might make things easier for other models in case they also have naming discrepancies.