Finetuning on MacOS

[adrvs]

Hi, any idea if finetune.py should run on MacOS ?

I tried the readme command

python finetune.py \
    --model_name_or_path name/or/path/to/hf/llama/7b/model \
    --train_file "data/vigogne_data_cleaned.json" \
    --output_dir "outputs/llama-7b-ft-vigogne-lora" \
    --run_name "llama-7b-ft-vigogne-lora" \
    --overwrite_output_dir \
    --model_max_length_percentile 95 \
    --preprocessing_num_workers 4 \
    --dataloader_num_workers 1 \
    --lora_r 8 \
    --lora_alpha 16 \
    --lora_dropout 0.05 \
    --target_modules "q_proj" "v_proj" \
    --per_device_train_batch_size 16 \
    --per_device_eval_batch_size 8 \
    --gradient_accumulation_steps 8 \
    --num_train_epochs 3 \
    --learning_rate 3e-4 \
    --warmup_steps 100 \
    --logging_steps 25 \
    --save_strategy "steps" \
    --save_steps 200 \
    --save_total_limit 3 \
    --report_to "tensorboard" "wandb"

and got

ValueError: FP16 Mixed precision training with AMP or APEX (`--fp16`) and FP16 half precision evaluation (`--fp16_full_eval`) can only be used on CUDA devices.

So I tried adding --use_mps_device and --fp16 0 but got

File "/Users/usr/PycharmProjects/vigogne/finetune.py", line 200, in train
    model = AutoModelForCausalLM.from_pretrained(
  File "/Users/usr/PycharmProjects/vigogne/.venv/lib/python3.9/site-packages/transformers/models/auto/auto_factory.py", line 468, in from_pretrained
    return model_class.from_pretrained(
  File "/Users/usr/PycharmProjects/vigogne/.venv/lib/python3.9/site-packages/transformers/modeling_utils.py", line 2730, in from_pretrained
    raise ValueError(
ValueError: 
                        Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit
                        the quantized model. If you want to dispatch the model on the CPU or the disk while keeping
                        these modules in 32-bit, you need to set `load_in_8bit_fp32_cpu_offload=True` and pass a custom
                        `device_map` to `from_pretrained`. Check
                        https://huggingface.co/docs/transformers/main/en/main_classes/quantization#offload-between-cpu-and-gpu
                        for more details.

[bofenghuang]

Hi @adrvs,

I have never run the training on M1.

But according to the last error message, you don't have enough memory to fit the model. You could try dispatching the model between CPU and GPU. See here