LoRA support in model builder (#955)

This PR adds LoRA MatMul changes in model builder. It includes changes made by Kunal and few changes done to make it work with Olive. It covers the scenario where base_layer of float and adapters are float. This PR will be followed with support for quant models scenario --------- Co-authored-by: Kunal Vaishnavi <[email protected]>
microsoft · Oct 21, 2024 · 4253ecc · 4253ecc
1 parent b932cbd
commit 4253ecc
Show file tree

Hide file tree

Showing 2 changed files with 226 additions and 51 deletions.
diff --git a/src/python/py/models/README.md b/src/python/py/models/README.md
@@ -20,6 +20,7 @@ This folder contains the model builder for quickly creating optimized and quanti
     - [Use 8 Bits Quantization in QMoE](#use-8-bits-quantization-in-qmoe)
     - [Hugging Face Authentication](#hugging-face-authentication)
     - [Use QDQ Pattern for Quantization](#use-qdq-pattern-for-quantization)
+    - [LoRA Models](#lora-models)
   - [Unit Testing Models](#unit-testing-models)
     - [Option 1: Use the model builder directly](#option-1-use-the-model-builder-directly)
     - [Option 2: Edit the config.json file](#option-2-edit-the-configjson-file-on-disk-and-then-run-the-model-builder)
@@ -217,6 +218,20 @@ python3 -m onnxruntime_genai.models.builder -i path_to_local_folder_on_disk -o p
 python3 builder.py -i path_to_local_folder_on_disk -o path_to_output_folder -p precision -e execution_provider -c cache_dir_to_store_temp_files --extra_options use_qdq=1
 ```
 
+#### LoRA Models
+
+This scenario is where you have a finetuned model with LoRA adapters and your model can be loaded in the Hugging Face style via [PEFT](https://github.com/huggingface/peft).
+
+- path_to_local_folder_on_disk = location where base_model's weights are present
+
+```
+# From wheel:
+python3 -m onnxruntime_genai.models.builder -i path_to_local_folder_on_disk -o path_to_output_folder -p fp16 -e execution_provider -c cache_dir_to_store_temp_files --extra_options adapter_path=path_to_adapter_files
+
+# From source:
+python3 builder.py -i path_to_local_folder_on_disk -o path_to_output_folder -p fp16 -e execution_provider -c cache_dir_to_store_temp_files --extra_options adapter_path=path_to_adapter_files
+```
+
 ### Unit Testing Models
 
 This scenario is where your PyTorch model is already downloaded locally (either in the default Hugging Face cache directory or in a local folder on disk). If it is not already downloaded locally, here is an example of how you can download it.