Converting Safetensors Format Weights from Llama Model with New Tokens to LitGPT Format

[GuocunWang]

Hello! I would like to convert the weights of a Llama model that has new tokens added into the LitGPT format, but I only have the file in the Safetensors format. Could you provide a solution for it?

[bhimrazy]

Hi @GuocunWang, this guide might be helpful for your use case:
👉 https://github.com/Lightning-AI/litgpt/blob/main/tutorials/convert_hf_checkpoint.md

Even if new tokens have been added, they might have be reflected in the tokenizer_config.json file while saving the model.
Let us know how it goes. Thanks!