You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
My LORA training has been interrupted twice now, given the same error. I think I may know the reason it is caused, but not sure. Both times it was interrupted, was after I updated the promp.txt file and it started generating the next sample images. I was looking through the generation and 2 of my prompts didn't have the correct formatting, (Not having a space between a period and the option --w) causing those two prompt to generate with undesirable widths. (512) It happened twice because I never updated the prompt in the GUI, so when I loaded the weights of my last saved LORA and started training again, the GUI overwrote my changes to the prompt.txt. I edited the prompt.txt file again this morning and the same error occurred.
Example my prompt line ended like:
...with palm trees and the ocean in the background.--w 768 --h 1024 --d 777 --s 30
and I changed it to:
...with palm trees and the ocean in the background. --w 768 --h 1024 --d 777 --s 30
The crashes happened when the exact prompt that was changed was set to generate, witch was the 4th prompt out of 5 total. The first time it crashed was during the 2nd set of sample generations, and the second crash was during the 6th set.
Here is my first traceback, I can post the second if needed:
Using cached text encoder outputs for prompt: The image is a close-up portrait of Ptyn1_0 sitting in the back seat of acar. She is smiling and looking directly at the camera. She is wearing a white tank. The car interior is visible through the window, and there are other cars visible in the background. Encoding prompt: The image is a close-up portrait of Ptyn1_0 sitting in the back seat of a car. She is smiling and looking directly at the camera. She is wearing a white tank. The car interior is visible through the window, and there are other cars visible in the background. 100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [03:13<00:00, 6.44s/it] 2024-11-03 01:13:46 INFO prompt: Full body portrait photograph of Ptyn1_0 wearing a pastel flux_train_utils.py:175 bikini, she has blonde hair and blue eyes, she is barefoot, standing on an empty beach, with palm trees and the ocean in the background. INFO height: 1024 flux_train_utils.py:177 INFO width: 768 flux_train_utils.py:178 INFO sample_steps: 30 flux_train_utils.py:179 INFO scale: 3.5 flux_train_utils.py:180 INFO seed: 777 flux_train_utils.py:183 Encoding prompt: Full body portrait photograph of Ptyn1_0 wearing a pastel bikini, she has blonde hair and blue eyes, she is barefoot, standing on an empty beach, with palm trees and the ocean in the background. Traceback (most recent call last): File "E:\_python_projects\Kohya_GUI_Flux_Installer_v42\kohya_ss\sd-scripts\flux_train_network.py", line 564, in <module> trainer.train(args) File "E:\_python_projects\Kohya_GUI_Flux_Installer_v42\kohya_ss\sd-scripts\train_network.py", line 1302, in train self.sample_images(accelerator, args, epoch + 1, global_step, accelerator.device, vae, tokenizers, text_encoder, unet) File "E:\_python_projects\Kohya_GUI_Flux_Installer_v42\kohya_ss\sd-scripts\flux_train_network.py", line 315, in sample_images flux_train_utils.sample_images( File "E:\_python_projects\Kohya_GUI_Flux_Installer_v42\kohya_ss\sd-scripts\library\flux_train_utils.py", line 89, in sample_images sample_image_inference( File "E:\_python_projects\Kohya_GUI_Flux_Installer_v42\kohya_ss\sd-scripts\library\flux_train_utils.py", line 224, insample_image_inference t5_attn_mask = t5_attn_mask.to(accelerator.device) if args.apply_t5_attn_mask else None AttributeError: 'NoneType' object has no attribute 'to' steps: 13%|██████▊ | 910/6825 [3:23:06<22:00:14, 13.39s/it, avr_loss=0.393] Traceback (most recent call last): File "C:\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "E:\_python_projects\Kohya_GUI_Flux_Installer_v42\kohya_ss\venv\Scripts\accelerate.EXE\__main__.py", line 7, in <module> sys.exit(main()) File "E:\_python_projects\Kohya_GUI_Flux_Installer_v42\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 48, in main args.func(args) File "E:\_python_projects\Kohya_GUI_Flux_Installer_v42\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1106, in launch_command simple_launcher(args) File "E:\_python_projects\Kohya_GUI_Flux_Installer_v42\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 704, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['E:\\_python_projects\\Kohya_GUI_Flux_Installer_v42\\kohya_ss\\venv\\Scripts\\python.exe', 'E:/_python_projects/Kohya_GUI_Flux_Installer_v42/kohya_ss/sd-scripts/flux_train_network.py', '--config_file', 'E:\\_Dropbox\\Dropbox\\AI Source material\\SD\\Created Characters\\Paityn_Hill\\Train_Setup\\model/config_lora-20241102-224749.toml']' returned non-zero exit status 1. 01:13:52-387054 INFO Training has ended.
If this is a problem with the actual script files please let me know.
The text was updated successfully, but these errors were encountered:
Greetings,
My LORA training has been interrupted twice now, given the same error. I think I may know the reason it is caused, but not sure. Both times it was interrupted, was after I updated the promp.txt file and it started generating the next sample images. I was looking through the generation and 2 of my prompts didn't have the correct formatting, (Not having a space between a period and the option --w) causing those two prompt to generate with undesirable widths. (512) It happened twice because I never updated the prompt in the GUI, so when I loaded the weights of my last saved LORA and started training again, the GUI overwrote my changes to the prompt.txt. I edited the prompt.txt file again this morning and the same error occurred.
Example my prompt line ended like:
...with palm trees and the ocean in the background.--w 768 --h 1024 --d 777 --s 30
and I changed it to:
...with palm trees and the ocean in the background. --w 768 --h 1024 --d 777 --s 30
The crashes happened when the exact prompt that was changed was set to generate, witch was the 4th prompt out of 5 total. The first time it crashed was during the 2nd set of sample generations, and the second crash was during the 6th set.
Here is my first traceback, I can post the second if needed:
Using cached text encoder outputs for prompt: The image is a close-up portrait of Ptyn1_0 sitting in the back seat of acar. She is smiling and looking directly at the camera. She is wearing a white tank. The car interior is visible through the window, and there are other cars visible in the background. Encoding prompt: The image is a close-up portrait of Ptyn1_0 sitting in the back seat of a car. She is smiling and looking directly at the camera. She is wearing a white tank. The car interior is visible through the window, and there are other cars visible in the background. 100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [03:13<00:00, 6.44s/it] 2024-11-03 01:13:46 INFO prompt: Full body portrait photograph of Ptyn1_0 wearing a pastel flux_train_utils.py:175 bikini, she has blonde hair and blue eyes, she is barefoot, standing on an empty beach, with palm trees and the ocean in the background. INFO height: 1024 flux_train_utils.py:177 INFO width: 768 flux_train_utils.py:178 INFO sample_steps: 30 flux_train_utils.py:179 INFO scale: 3.5 flux_train_utils.py:180 INFO seed: 777 flux_train_utils.py:183 Encoding prompt: Full body portrait photograph of Ptyn1_0 wearing a pastel bikini, she has blonde hair and blue eyes, she is barefoot, standing on an empty beach, with palm trees and the ocean in the background. Traceback (most recent call last): File "E:\_python_projects\Kohya_GUI_Flux_Installer_v42\kohya_ss\sd-scripts\flux_train_network.py", line 564, in <module> trainer.train(args) File "E:\_python_projects\Kohya_GUI_Flux_Installer_v42\kohya_ss\sd-scripts\train_network.py", line 1302, in train self.sample_images(accelerator, args, epoch + 1, global_step, accelerator.device, vae, tokenizers, text_encoder, unet) File "E:\_python_projects\Kohya_GUI_Flux_Installer_v42\kohya_ss\sd-scripts\flux_train_network.py", line 315, in sample_images flux_train_utils.sample_images( File "E:\_python_projects\Kohya_GUI_Flux_Installer_v42\kohya_ss\sd-scripts\library\flux_train_utils.py", line 89, in sample_images sample_image_inference( File "E:\_python_projects\Kohya_GUI_Flux_Installer_v42\kohya_ss\sd-scripts\library\flux_train_utils.py", line 224, insample_image_inference t5_attn_mask = t5_attn_mask.to(accelerator.device) if args.apply_t5_attn_mask else None AttributeError: 'NoneType' object has no attribute 'to' steps: 13%|██████▊ | 910/6825 [3:23:06<22:00:14, 13.39s/it, avr_loss=0.393] Traceback (most recent call last): File "C:\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "E:\_python_projects\Kohya_GUI_Flux_Installer_v42\kohya_ss\venv\Scripts\accelerate.EXE\__main__.py", line 7, in <module> sys.exit(main()) File "E:\_python_projects\Kohya_GUI_Flux_Installer_v42\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 48, in main args.func(args) File "E:\_python_projects\Kohya_GUI_Flux_Installer_v42\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1106, in launch_command simple_launcher(args) File "E:\_python_projects\Kohya_GUI_Flux_Installer_v42\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 704, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['E:\\_python_projects\\Kohya_GUI_Flux_Installer_v42\\kohya_ss\\venv\\Scripts\\python.exe', 'E:/_python_projects/Kohya_GUI_Flux_Installer_v42/kohya_ss/sd-scripts/flux_train_network.py', '--config_file', 'E:\\_Dropbox\\Dropbox\\AI Source material\\SD\\Created Characters\\Paityn_Hill\\Train_Setup\\model/config_lora-20241102-224749.toml']' returned non-zero exit status 1. 01:13:52-387054 INFO Training has ended.
If this is a problem with the actual script files please let me know.
The text was updated successfully, but these errors were encountered: