OnElast thing

pytorch · Oct 29, 2024 · 0448ed7 · 0448ed7
1 parent b208c94
commit 0448ed7
Showing 1 changed file with 3 additions and 5 deletions.
diff --git a/docs/source/tutorials/memory_optimizations.rst b/docs/source/tutorials/memory_optimizations.rst
@@ -301,8 +301,8 @@ As above, these parameters are also specified under the ``model`` flag or config
   tune run lora_finetune_single_device --config llama3/8B_lora_single_device  \
   model.apply_lora_to_mlp=True \
   model.lora_attn_modules=["q_proj","k_proj","v_proj"] \
-  model.lora_rank=128 \
-  model.lora_rank=256
+  model.lora_rank=32 \
+  model.lora_rank=64
 
 .. code-block:: yaml
 
@@ -311,7 +311,7 @@ As above, these parameters are also specified under the ``model`` flag or config
     apply_lora_to_mlp: True
     lora_attn_modules: ["q_proj", "k_proj", "v_proj"]
     lora_rank: 32
-    lora_rank: 64
+    lora_alpha: 64
 
 .. note::
 
@@ -369,8 +369,6 @@ or, by modifying a config:
     lora_rank: 32
     lora_alpha: 64
 
-
-
 .. _glossary_dora:
 
 Weight-Decomposed Low-Rank Adaptation (DoRA)