Llama 3.1 RoPE not implemented -- Finetuning will lead to spelling/grammar mistakes

[calvintwr]

Bug description

I'm re-posting this as the original issue created with a question tag is not receiving attention.

Per #1699

It seems that Llama 3.1 RoPE is not implemented. The Llama 3.1 RoPE is non-standard, according to https://github.com/huggingface/transformers/blob/5c1027bf09717f664b579e01cbb8ec3ef5aeb140/src/transformers/modeling_rope_utils.py#L329-L347

Finetuning will lead to spelling/grammar mistakes.

From @zzhhjjj:

Dear LitGPT Maintainer,

Thank you for your great work. I encountered an issue while trying to fine-tune LLaMA 3.1 and came here for reference. I was looking for the LLaMA 3.1 RoPE function change, but I couldn't find it in your repository. Based on this PR, it seems like it hasn't been added yet.

https://github.com/Lightning-AI/litgpt/pull/1619/files#diff-3b8a58a4d021803b3171b886bb9162fd659e671131f3f61036f9210cb5d0bc7c

Reference: https://github.com/huggingface/transformers/blob/5c1027bf09717f664b579e01cbb8ec3ef5aeb140/src/transformers/modeling_rope_utils.py#L329-L347

Thanks for your help.

What operating system are you using?

Linux

LitGPT Version

Version: 0.4.11

[rasbt]

Thanks, I was out for 2 weeks and am just reading this. I may not have the bandwidth to address this immediately due to other issues on my list, but I hope to get to it soon.

[rasbt]

Hey @calvintwr , thanks again for bringing this up. It should be fixed now as of #1745

[calvintwr]

Hey @rasbt thanks for this!