Ludwig New Version Issues of Repeating output

[ahsantfw]

Hi there,

I have been using Ludwig for fine-tuning LLMs and it was all going until few weeks ago, this issue started coming up and I had moved on to write code from scratch.

I am having issues with predictions:
I am following Ludwig's Github Colab link for Mistral-7B, Even by using the exact same configuration, My model predictions give a repeated answer with a separation of 'y' or space, Can Anyone let me know what mistake I would be making in this?

Config:
import yaml
from ludwig.api import LudwigModel
import logging

A configuration that automates the whole finetuning pipeline using Ludwig

qlora_fine_tuning_config = yaml.safe_load(
"""
model_type: llm
base_model: mistralai/Mistral-7B-v0.1
input_features:

• name: Input
  type: text
  output_features:
• name: output
  type: text
  prompt:
  template: >-

  Context: You are an expert who converts...... long context of around 400 words....

  Input: {Input}

  output:

generation:
temperature: 0.1
max_new_tokens: 2048
adapter:
type: lora
quantization:
bits: 4
preprocessing:
global_max_sequence_length: 2048
split:
type: random
probabilities:
- 0.90
- 0.05
- 0.05
trainer:
type: finetune
epochs: 7
batch_size: 1
eval_batch_size: 2
gradient_accumulation_steps: 16
learning_rate: 0.0004
learning_rate_scheduler:
warmup_fraction: 0.03
"""
)
Now, it gives the correct output but repeats it for like more than 20 times. Also, while looking into logs, it turns the template into some kind of gibberish language as well.
Predictions Output:
"Actual output" y " Actual Output" " Actual Output"

Solution: For now, downgrading the Ludwig to 0.8.6, resolves this issue and everything works fine.

I request the team to look into the new update and settle this issue

[arnavgarg1]

Hi @ahsantfw! We fixed a recent regression in LLM fine-tuning quality in Ludwig 0.9.3. More info here. Are you able to retry with the latest stable version when you get a chance?