Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

(Flux) Dreambooth trained model generates noise #2939

Open
antey3064 opened this issue Oct 30, 2024 · 2 comments
Open

(Flux) Dreambooth trained model generates noise #2939

antey3064 opened this issue Oct 30, 2024 · 2 comments

Comments

@antey3064
Copy link

What could be the reason?

Guidance Scale:1

last7_e000001_00_20241031021302_54

02:07:45-358115 INFO Start training Dreambooth...
02:07:45-359117 INFO Validating lr scheduler arguments...
02:07:45-360117 INFO Validating optimizer arguments...
02:07:45-362120 INFO Validating D:/SD/MyImages/testPink\log existence and writability... SUCCESS
02:07:45-363120 INFO Validating D:/SD/MyImages/testPink\model existence and writability... SUCCESS
02:07:45-363120 INFO Validating D:/SD/webui_forge_cu121_torch231/webui/models/Stable-diffusion/flux1-dev.safetensors
existence... SUCCESS
02:07:45-365122 INFO Validating D:/SD/MyImages/testPink\img existence... SUCCESS
02:07:45-366122 INFO Folder 1_xyzpinkdress dress: 1 repeats found
02:07:45-367123 INFO Folder 1_xyzpinkdress dress: 8 images found
02:07:45-368124 INFO Folder 1_xyzpinkdress dress: 8 * 1 = 8 steps
02:07:45-369125 INFO Regularization factor: 1
02:07:45-370126 INFO Total steps: 8
02:07:45-370126 INFO Train batch size: 1
02:07:45-373129 INFO Gradient accumulation steps: 1
02:07:45-375797 INFO Epoch: 18
02:07:45-377055 INFO max_train_steps (8 / 1 / 1 * 18 * 1) = 144
02:07:45-378067 INFO lr_warmup_steps = 0
02:07:45-382736 INFO Saving training config to D:/SD/MyImages/testPink\model\last7_20241031-020745.json...
02:07:45-384965 INFO Executing command: D:\SD\kohya_ss\venv\Scripts\accelerate.EXE launch --dynamo_backend no
--dynamo_mode default --gpu_ids 0 --mixed_precision bf16 --num_processes 1 --num_machines 1
--num_cpu_threads_per_process 2 D:/SD/kohya_ss/sd-scripts/flux_train.py --config_file
D:/SD/MyImages/testPink\model/config_dreambooth-20241031-020745.toml
D:\SD\kohya_ss\venv\lib\site-packages\diffusers\utils\outputs.py:63: FutureWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree._register_pytree_node(
D:\SD\kohya_ss\venv\lib\site-packages\diffusers\utils\outputs.py:63: FutureWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree._register_pytree_node(
2024-10-31 02:07:56 INFO Loading settings from train_util.py:4435
D:/SD/MyImages/testPink\model/config_dreambooth-20241031-020745.toml...
INFO D:/SD/MyImages/testPink\model/config_dreambooth-20241031-020745 train_util.py:4454
2024-10-31 02:07:56 INFO Using DreamBooth method. flux_train.py:107
INFO prepare images. train_util.py:1956
INFO get image size from name of cache files train_util.py:1873
100%|████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:00<?, ?it/s]
INFO set image size from cache files: 8/8 train_util.py:1901
INFO found directory D:\SD\MyImages\testPink\img\1_xyzpinkdress dress train_util.py:1903
contains 8 image files
read caption: 100%|██████████████████████████████████████████████████████████████████████████████| 8/8 [00:00<?, ?it/s]
INFO 8 train images with repeating. train_util.py:1997
INFO 0 reg images. train_util.py:2000
WARNING no regularization images / 正則化画像が見つかりませんでした train_util.py:2005
INFO [Dataset 0] config_util.py:567
batch_size: 1
resolution: (1024, 1024)
enable_bucket: True
network_multiplier: 1.0
min_bucket_reso: 256
max_bucket_reso: 2048
bucket_reso_steps: 64
bucket_no_upscale: True

                           [Subset 0 of Dataset 0]
                             image_dir: "D:\SD\MyImages\testPink\img\1_xyzpinkdress dress"
                             image_count: 8
                             num_repeats: 1
                             shuffle_caption: False
                             keep_tokens: 0
                             keep_tokens_separator:
                             caption_separator: ,
                             secondary_separator: None
                             enable_wildcard: False
                             caption_dropout_rate: 0
                             caption_dropout_every_n_epoches: 0
                             caption_tag_dropout_rate: 0.0
                             caption_prefix: None
                             caption_suffix: None
                             color_aug: False
                             flip_aug: False
                             face_crop_aug_range: None
                             random_crop: False
                             token_warmup_min: 1
                             token_warmup_step: 0
                             alpha_mask: False
                             custom_attributes: {}
                             is_reg: False
                             class_tokens: xyzpinkdress dress
                             caption_extension: .txt


                INFO     [Dataset 0]                                                              config_util.py:573
                INFO     loading image sizes.                                                      train_util.py:923

100%|████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:00<?, ?it/s]
INFO make buckets train_util.py:946
WARNING min_bucket_reso and max_bucket_reso are ignored if bucket_no_upscale is train_util.py:963
set, because bucket reso is defined by image size automatically /
bucket_no_upscaleが指定された場合は、bucketの解像度は画像サイズから自動計
算されるため、min_bucket_resoとmax_bucket_resoは無視されます
INFO number of images (including repeats) / train_util.py:992
各bucketの画像枚数（繰り返し回数を含む）
INFO bucket 0: resolution (896, 1088), count: 6 train_util.py:997
INFO bucket 1: resolution (1024, 1024), count: 2 train_util.py:997
INFO mean ar error (without repeats): 0.01759017994531699 train_util.py:1002
INFO Checking the state dict: Diffusers or BFL, dev or schnell flux_utils.py:62
INFO prepare accelerator flux_train.py:177
accelerator device: cuda
INFO Building AutoEncoder flux_utils.py:152
INFO Loading state dict from flux_utils.py:157
D:/SD/webui_forge_cu121_torch231/webui/models/VAE/ae.safetensors
INFO Loaded AE: flux_utils.py:160
2024-10-31 02:07:57 INFO [Dataset 0] train_util.py:2480
INFO caching latents with caching strategy. train_util.py:1048
INFO caching latents... train_util.py:1093
100%|██████████████████████████████████████████████████████████████████████████████████| 8/8 [00:00<00:00, 1998.24it/s]
D:\SD\kohya_ss\venv\lib\site-packages\transformers\tokenization_utils_base.py:1601: FutureWarning: clean_up_tokenization_spaces was not set. It will be set to True by default. This behavior will be depracted in transformers v4.45, and will be then set to False by default. For more details check this issue: huggingface/transformers#31884
warnings.warn(
You are using the default legacy behaviour of the <class 'transformers.models.t5.tokenization_t5.T5Tokenizer'>. This is expected, and simply means that the legacy (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set legacy=False. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in huggingface/transformers#24565
2024-10-31 02:07:58 INFO Building CLIP flux_utils.py:165
INFO Loading state dict from flux_utils.py:258
D:/SD/webui_forge_cu121_torch231/webui/models/text_encoder/clip_l.safeten
sors
INFO Loaded CLIP: flux_utils.py:261
INFO Loading state dict from flux_utils.py:306
D:/SD/webui_forge_cu121_torch231/webui/models/text_encoder/t5xxl_fp16.saf
etensors
INFO Loaded T5xxl: flux_utils.py:309
2024-10-31 02:08:07 INFO [Dataset 0] train_util.py:2502
INFO caching Text Encoder outputs with caching strategy. train_util.py:1227
INFO checking cache validity... train_util.py:1238
100%|███████████████████████████████████████████████████████████████████████████████████| 8/8 [00:00<00:00, 584.69it/s]
INFO no Text Encoder outputs to cache train_util.py:1265
INFO cache Text Encoder outputs for sample prompt: flux_train.py:240
D:/SD/MyImages/testPink\model\sample/prompt.txt
INFO cache Text Encoder outputs for prompt: woman standing, dress, flux_train.py:250
xyzpinkdress, full body
2024-10-31 02:08:08 INFO cache Text Encoder outputs for prompt: flux_train.py:250
2024-10-31 02:08:09 INFO Checking the state dict: Diffusers or BFL, dev or schnell flux_utils.py:62
INFO Building Flux model dev from BFL checkpoint flux_utils.py:116
INFO Loading state dict from flux_utils.py:133
D:/SD/webui_forge_cu121_torch231/webui/models/Stable-diffusion/flux1-dev.
safetensors
INFO Loaded Flux: flux_utils.py:145
FLUX: Gradient checkpointing enabled. CPU offload: False
INFO enable block swap: blocks_to_swap=10 flux_train.py:295
number of trainable parameters: 11901408320
prepare optimizer, data loader etc.
INFO use Adafactor optimizer | {'relative_step': True} train_util.py:4748
INFO relative_step is true / relative_stepがtrueです train_util.py:4751
WARNING learning rate is used as initial_lr / 指定したlearning train_util.py:4753
rateはinitial_lrとして使用されます
WARNING unet_lr and text_encoder_lr are ignored / train_util.py:4765
unet_lrとtext_encoder_lrは無視されます
INFO use adafactor_scheduler / スケジューラにadafactor_schedulerを使用します train_util.py:4770
running training / 学習開始
num examples / サンプル数: 8
num batches per epoch / 1epochのバッチ数: 8
num epochs / epoch数: 18
batch size per device / バッチサイズ: 1
gradient accumulation steps / 勾配を合計するステップ数 = 1
total optimization steps / 学習ステップ数: 144
steps: 0%| | 0/144 [00:00<?, ?it/s]
epoch 1/18
2024-10-31 02:08:22 INFO epoch is incremented. current_epoch: 0, epoch: 1 train_util.py:715
D:\SD\kohya_ss\venv\lib\site-packages\torch\autograd\graph.py:825: UserWarning: cuDNN SDPA backward got grad_output.strides() != output.strides(), attempting to materialize a grad_output with matching strides... (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cudnn\MHA.cpp:676.)
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
steps: 6%|███▍ | 8/144 [02:15<38:23, 16.94s/it, avr_loss=1.8]2024-10-31 02:10:25 INFO flux_train_utils.py:59
INFO generating sample images at step / サンプル画像生成 ステップ: 8 flux_train_utils.py:60
INFO prompt: woman standing, dress, xyzpinkdress, full body flux_train_utils.py:176
INFO height: 1024 flux_train_utils.py:178
INFO width: 1024 flux_train_utils.py:179
INFO sample_steps: 20 flux_train_utils.py:180
INFO scale: 1.0 flux_train_utils.py:181
INFO seed: 54 flux_train_utils.py:184
Using cached text encoder outputs for prompt: woman standing, dress, xyzpinkdress, full body
Encoding prompt: woman standing, dress, xyzpinkdress, full body
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [02:31<00:00, 7.56s/it]
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [02:31<00:00, 7.74s/it]
epoch 2/18
@antey3064
Copy link
Author

problem solved. The problem was in the incorrectly configured Dreambooth

@kubakubkub
Copy link

problem solved. The problem was in the incorrectly configured Dreambooth

You might sharing your config ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@antey3064 @kubakubkub