Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Support for Flex.1 Alpha #1891

Open
stepfunction83 opened this issue Jan 22, 2025 · 11 comments
Open

Add Support for Flex.1 Alpha #1891

stepfunction83 opened this issue Jan 22, 2025 · 11 comments

Comments

@stepfunction83
Copy link

stepfunction83 commented Jan 22, 2025 •
edited
Loading

Creating an Issue from the related discussion: #1882

Ostris has created a new Flux model that is both distilled to a lower parameter count and has a custom trained guidance module which allows the model to be much more easily and flexibly finetuned.

https://huggingface.co/ostris/Flex.1-alpha

From my experience so far creating a few LoRAs, the hype is 100% real and this is trainable in a way that base Flux Dev is simply not.

To train this, the newly created guidance module needs to be bypassed during training and then restored afterwards. This is currently implemented in ai-toolkit and would need to be implemented in sd-scripts to be able to support training of the model.

This seems to be the key snippet of code to bypass the guidance block in ai-toolkit:

def guidance_embed_bypass_forward(self, timestep, guidance, pooled_projection):
    timesteps_proj = self.time_proj(timestep)
    timesteps_emb = self.timestep_embedder(
        timesteps_proj.to(dtype=pooled_projection.dtype))  # (N, D)
    pooled_projections = self.text_embedder(pooled_projection)
    conditioning = timesteps_emb + pooled_projections
    return conditioning

# bypass the forward function

def bypass_flux_guidance(transformer):
    if hasattr(transformer.time_text_embed, '_bfg_orig_forward'):
        return
    # dont bypass if it doesnt have the guidance embedding
    if not hasattr(transformer.time_text_embed, 'guidance_embedder'):
        return
    transformer.time_text_embed._bfg_orig_forward = transformer.time_text_embed.forward
    transformer.time_text_embed.forward = partial(
        guidance_embed_bypass_forward, transformer.time_text_embed
    )

# restore the forward function

def restore_flux_guidance(transformer):
    if not hasattr(transformer.time_text_embed, '_bfg_orig_forward'):
        return
    transformer.time_text_embed.forward = transformer.time_text_embed._bfg_orig_forward
    del transformer.time_text_embed._bfg_orig_forward

bypass_flux_guidance is applied before noise prediction, while restore_flux_guidance is applied after noise prediction and also before sampling.
@stepfunction83
Copy link
Author

stepfunction83 commented Jan 22, 2025 •
edited
Loading

This was also implemented into diffusion-pipe through this commit: tdrussell/diffusion-pipe@90ab3f9

@stepfunction83
Copy link
Author

stepfunction83 commented Jan 23, 2025 •
edited
Loading

I've created a minimal working example here:

https://github.com/stepfunction83/sd-scripts/tree/sd3

I was able to quickly train a 1000 step finetune of flex and was able to test it in Comfy to validate that the training does take and the guidance module is not destroyed in the process.

Additionally, the sampling was corrected as well and now works as expected.

Just add the --bypass_flux_guidance command line argument

@stepfunction83 stepfunction83 mentioned this issue Jan 23, 2025
Open
@CodeAlexx
Copy link

i use the GUI and it says bypass_flux_guidance unknown

@CodeAlexx
Copy link

flux_train.py: error: unrecognized arguments: -bypass_flux_guidance
Traceback (most recent call last):
File "/home/alex/kohya_ss/venv/bin/accelerate", line 8, in
sys.exit(main())

@stepfunction83
Copy link
Author

stepfunction83 commented Jan 23, 2025 •
edited
Loading

Looks like you're missing a "-". It should be --bypass_flux_guidance.

@stepfunction83
Copy link
Author

stepfunction83 commented Jan 23, 2025 •
edited
Loading

If it still doesn't work, in your kohya_ss folder, delete the sd-scripts folder and then navigate to the kohya_ss in a terminal and run this:

git clone https://github.com/stepfunction83/sd-scripts.git -b sd3

It will replace the sd-scripts folder with the version with the update.

@CodeAlexx
Copy link 

       dud as you said

[--cpu_offload_checkpointing]
flux_train.py: error: unrecognized arguments: --bypass_flux_guidance
Traceback (most recent call last):
File "/home/alex/kohya_ss/venv/bin/accelerate", line 8, in
sys.exit(main())
File "/h

@stepfunction83
Copy link
Author

Can you provide the full error please?

@CodeAlexx
Copy link 

                     ...

19:21:11-815746 INFO Executing command:
/home/alex/kohya_ss/venv/bin/accelerate launch
--dynamo_backend no --dynamo_mode default
--mixed_precision bf16 --num_processes 1 --num_machines
1 --num_cpu_threads_per_process 4
/home/alex/kohya_ss/sd-scripts/flux_train.py
--config_file
/home/alex/FlexTest/model/config_dreambooth-20250122-19
2111.toml --bypass_flux_guidance
/home/alex/kohya_ss/venv/lib/python3.10/site-packages/diffusers/utils/outputs.py:63: FutureWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree._register_pytree_node(
2025-01-22 19:21:16.060291: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-01-22 19:21:16.060314: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-01-22 19:21:16.060852: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-01-22 19:21:16.064371: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-01-22 19:21:16.601468: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
/home/alex/kohya_ss/venv/lib/python3.10/site-packages/diffusers/utils/outputs.py:63: FutureWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree._register_pytree_node(
usage: flux_train.py [-h]
[--console_log_level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
[--console_log_file CONSOLE_LOG_FILE]
[--console_log_simple] [--v2] [--v_parameterization]
[--pretrained_model_name_or_path PRETRAINED_MODEL_NAME_OR_PATH]
[--tokenizer_cache_dir TOKENIZER_CACHE_DIR]
[--train_data_dir TRAIN_DATA_DIR] [--cache_info]
[--shuffle_caption]
[--caption_separator CAPTION_SEPARATOR]
[--caption_extension CAPTION_EXTENSION]
[--caption_extention CAPTION_EXTENTION]
[--keep_tokens KEEP_TOKENS]
[--keep_tokens_separator KEEP_TOKENS_SEPARATOR]
[--secondary_separator SECONDARY_SEPARATOR]
[--enable_wildcard] [--caption_prefix CAPTION_PREFIX]
[--caption_suffix CAPTION_SUFFIX] [--color_aug]
[--flip_aug] [--face_crop_aug_range FACE_CROP_AUG_RANGE]
[--random_crop] [--debug_dataset]
[--resolution RESOLUTION] [--cache_latents]
[--vae_batch_size VAE_BATCH_SIZE]
[--cache_latents_to_disk] [--skip_cache_check]
[--enable_bucket] [--min_bucket_reso MIN_BUCKET_RESO]
[--max_bucket_reso MAX_BUCKET_RESO]
[--bucket_reso_steps BUCKET_RESO_STEPS]
[--bucket_no_upscale]
[--token_warmup_min TOKEN_WARMUP_MIN]
[--token_warmup_step TOKEN_WARMUP_STEP] [--alpha_mask]
[--dataset_class DATASET_CLASS]
[--caption_dropout_rate CAPTION_DROPOUT_RATE]
[--caption_dropout_every_n_epochs CAPTION_DROPOUT_EVERY_N_EPOCHS]
[--caption_tag_dropout_rate CAPTION_TAG_DROPOUT_RATE]
[--reg_data_dir REG_DATA_DIR] [--in_json IN_JSON]
[--dataset_repeats DATASET_REPEATS]
[--output_dir OUTPUT_DIR] [--output_name OUTPUT_NAME]
[--huggingface_repo_id HUGGINGFACE_REPO_ID]
[--huggingface_repo_type HUGGINGFACE_REPO_TYPE]
[--huggingface_path_in_repo HUGGINGFACE_PATH_IN_REPO]
[--huggingface_token HUGGINGFACE_TOKEN]
[--huggingface_repo_visibility HUGGINGFACE_REPO_VISIBILITY]
[--save_state_to_huggingface] [--resume_from_huggingface]
[--async_upload]
[--save_precision {None,float,fp16,bf16}]
[--save_every_n_epochs SAVE_EVERY_N_EPOCHS]
[--save_every_n_steps SAVE_EVERY_N_STEPS]
[--save_n_epoch_ratio SAVE_N_EPOCH_RATIO]
[--save_last_n_epochs SAVE_LAST_N_EPOCHS]
[--save_last_n_epochs_state SAVE_LAST_N_EPOCHS_STATE]
[--save_last_n_steps SAVE_LAST_N_STEPS]
[--save_last_n_steps_state SAVE_LAST_N_STEPS_STATE]
[--save_state] [--save_state_on_train_end]
[--resume RESUME] [--train_batch_size TRAIN_BATCH_SIZE]
[--max_token_length {None,150,225}] [--mem_eff_attn]
[--torch_compile]
[--dynamo_backend {eager,aot_eager,inductor,aot_ts_nvfuser,nvprims_nvfuser,cudagraphs,ofi,fx2trt,onnxrt,tensort,ipex,tvm}]
[--xformers] [--sdpa] [--vae VAE]
[--max_train_steps MAX_TRAIN_STEPS]
[--max_train_epochs MAX_TRAIN_EPOCHS]
[--max_data_loader_n_workers MAX_DATA_LOADER_N_WORKERS]
[--persistent_data_loader_workers] [--seed SEED]
[--gradient_checkpointing]
[--gradient_accumulation_steps GRADIENT_ACCUMULATION_STEPS]
[--mixed_precision {no,fp16,bf16}] [--full_fp16]
[--full_bf16] [--fp8_base] [--ddp_timeout DDP_TIMEOUT]
[--ddp_gradient_as_bucket_view] [--ddp_static_graph]
[--clip_skip CLIP_SKIP] [--logging_dir LOGGING_DIR]
[--log_with {tensorboard,wandb,all}]
[--log_prefix LOG_PREFIX]
[--log_tracker_name LOG_TRACKER_NAME]
[--wandb_run_name WANDB_RUN_NAME]
[--log_tracker_config LOG_TRACKER_CONFIG]
[--wandb_api_key WANDB_API_KEY] [--log_config]
[--noise_offset NOISE_OFFSET]
[--noise_offset_random_strength]
[--multires_noise_iterations MULTIRES_NOISE_ITERATIONS]
[--ip_noise_gamma IP_NOISE_GAMMA]
[--ip_noise_gamma_random_strength]
[--multires_noise_discount MULTIRES_NOISE_DISCOUNT]
[--adaptive_noise_scale ADAPTIVE_NOISE_SCALE]
[--zero_terminal_snr] [--min_timestep MIN_TIMESTEP]
[--max_timestep MAX_TIMESTEP]
[--loss_type {l1,l2,huber,smooth_l1}]
[--huber_schedule {constant,exponential,snr}]
[--huber_c HUBER_C] [--huber_scale HUBER_SCALE]
[--lowram] [--highvram]
[--sample_every_n_steps SAMPLE_EVERY_N_STEPS]
[--sample_at_first]
[--sample_every_n_epochs SAMPLE_EVERY_N_EPOCHS]
[--sample_prompts SAMPLE_PROMPTS]
[--sample_sampler {ddim,pndm,lms,euler,euler_a,heun,dpm_2,dpm_2_a,dpmsolver,dpmsolver++,dpmsingle,k_lms,k_euler,k_euler_a,k_dpm_2,k_dpm_2_a}]
[--config_file CONFIG_FILE] [--output_config]
[--metadata_title METADATA_TITLE]
[--metadata_author METADATA_AUTHOR]
[--metadata_description METADATA_DESCRIPTION]
[--metadata_license METADATA_LICENSE]
[--metadata_tags METADATA_TAGS]
[--conditioning_data_dir CONDITIONING_DATA_DIR]
[--masked_loss] [--deepspeed] [--zero_stage {0,1,2,3}]
[--offload_optimizer_device {None,cpu,nvme}]
[--offload_optimizer_nvme_path OFFLOAD_OPTIMIZER_NVME_PATH]
[--offload_param_device {None,cpu,nvme}]
[--offload_param_nvme_path OFFLOAD_PARAM_NVME_PATH]
[--zero3_init_flag] [--zero3_save_16bit_model]
[--fp16_master_weights_and_gradients]
[--save_model_as {None,ckpt,safetensors,diffusers,diffusers_safetensors}]
[--use_safetensors] [--optimizer_type OPTIMIZER_TYPE]
[--use_8bit_adam] [--use_lion_optimizer]
[--learning_rate LEARNING_RATE]
[--max_grad_norm MAX_GRAD_NORM]
[--optimizer_args [OPTIMIZER_ARGS ...]]
[--lr_scheduler_type LR_SCHEDULER_TYPE]
[--lr_scheduler_args [LR_SCHEDULER_ARGS ...]]
[--lr_scheduler LR_SCHEDULER]
[--lr_warmup_steps LR_WARMUP_STEPS]
[--lr_decay_steps LR_DECAY_STEPS]
[--lr_scheduler_num_cycles LR_SCHEDULER_NUM_CYCLES]
[--lr_scheduler_power LR_SCHEDULER_POWER]
[--fused_backward_pass]
[--lr_scheduler_timescale LR_SCHEDULER_TIMESCALE]
[--lr_scheduler_min_lr_ratio LR_SCHEDULER_MIN_LR_RATIO]
[--dataset_config DATASET_CONFIG]
[--min_snr_gamma MIN_SNR_GAMMA]
[--scale_v_pred_loss_like_noise_pred]
[--v_pred_like_loss V_PRED_LIKE_LOSS]
[--debiased_estimation_loss] [--weighted_captions]
[--cache_text_encoder_outputs]
[--cache_text_encoder_outputs_to_disk]
[--text_encoder_batch_size TEXT_ENCODER_BATCH_SIZE]
[--disable_mmap_load_safetensors]
[--weighting_scheme {sigma_sqrt,logit_normal,mode,cosmap,none,uniform}]
[--logit_mean LOGIT_MEAN] [--logit_std LOGIT_STD]
[--mode_scale MODE_SCALE]
[--blocks_to_swap BLOCKS_TO_SWAP] [--clip_l CLIP_L]
[--t5xxl T5XXL] [--ae AE]
[--controlnet_model_name_or_path CONTROLNET_MODEL_NAME_OR_PATH]
[--t5xxl_max_token_length T5XXL_MAX_TOKEN_LENGTH]
[--apply_t5_attn_mask] [--guidance_scale GUIDANCE_SCALE]
[--timestep_sampling {sigma,uniform,sigmoid,shift,flux_shift}]
[--sigmoid_scale SIGMOID_SCALE]
[--model_prediction_type {raw,additive,sigma_scaled}]
[--discrete_flow_shift DISCRETE_FLOW_SHIFT]
[--mem_eff_save]
[--fused_optimizer_groups FUSED_OPTIMIZER_GROUPS]
[--blockwise_fused_optimizers]
[--skip_latents_validity_check]
[--double_blocks_to_swap DOUBLE_BLOCKS_TO_SWAP]
[--single_blocks_to_swap SINGLE_BLOCKS_TO_SWAP]
[--cpu_offload_checkpointing]
flux_train.py: error: unrecognized arguments: --bypass_flux_guidance
Traceback (most recent call last):
File "/home/alex/kohya_ss/venv/bin/accelerate", line 8, in
sys.exit(main())
File "/home/alex/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
args.func(args)
File "/home/alex/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1106, in launch_command
simple_launcher(args)
File "/home/alex/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 704, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/home/alex/kohya_ss/venv/bin/python3.10', '/home/alex/kohya_ss/sd-scripts/flux_train.py', '--config_file', '/home/alex/FlexTest/model/config_dreambooth-20250122-192111.toml', '--bypass_flux_guidance']' returned non-zero exit status 2.
19:21:19-051305 INFO Training has ended.

@CodeAlexx
Copy link

arning: LD_LIBRARY_PATH environment variable is not set.
Certain functionalities may not work correctly.
Please ensure that the required libraries are properly configured.

If you use WSL2 you may want to: export LD_LIBRARY_PATH=/usr/lib/wsl/lib/

2025-01-22 19:19:53.640904: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-01-22 19:19:53.640945: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-01-22 19:19:53.716917: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-01-22 19:19:53.867606: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-01-22 19:19:54.814822: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
19:19:59-176499 INFO Kohya_ss GUI version: v24.2.0

19:19:59-238859 INFO Submodule initialized and updated.
19:19:59-239670 INFO nVidia toolkit detected
19:20:00-051452 INFO Torch 2.5.0+cu124
19:20:00-082369 INFO Torch backend: nVidia CUDA 12.4 cuDNN 90100
19:20:00-113689 INFO Torch detected GPU: NVIDIA GeForce RTX 3090 Ti VRAM
24241MB Arch 8.6 Cores 84
19:20:00-114581 INFO Python version is 3.10.12 (main, Jan 17 2025, 14:35:34)
[GCC 11.4.0]
19:20:00-115334 INFO Installing/Validating requirements from
/home/alex/kohya_ss/requirements_linux.txt...
19:20:00-517228 INFO Looking in indexes: https://pypi.org/simple,
https://download.pytorch.org/whl/cu124
19:20:00-518006 INFO Obtaining file:///home/alex/kohya_ss/sd-scripts (from
-r /home/alex/kohya_ss/requirements.txt (line 37))
19:20:00-518822 INFO Preparing metadata (setup.py): started
19:20:00-640948 INFO Preparing metadata (setup.py): finished with status
'done'
19:20:02-405829 INFO Installing collected packages: library
19:20:02-406571 INFO Attempting uninstall: library
19:20:02-407074 INFO Found existing installation: library 0.0.0
19:20:02-407966 INFO Uninstalling library-0.0.0:
19:20:03-895019 INFO Successfully uninstalled library-0.0.0
19:20:03-895749 INFO Running setup.py develop for library
19:20:04-127580 INFO Successfully installed library-0.0.0
19:20:04-633854 INFO headless: False
19:20:04-637441 INFO Using shell=True when running external commands...

To create a public link, set share=True in launch().
19:20:35-478530 INFO Loading config...
19:21:11-740917 INFO Start training Dreambooth...
19:21:11-741670 INFO Validating lr scheduler arguments...
19:21:11-742370 INFO Validating optimizer arguments...
19:21:11-742940 INFO Validating /home/alex/FlexTest/logs existence and
writability... SUCCESS
19:21:11-743540 INFO Validating /home/alex/FlexTest/model existence and
writability... SUCCESS
19:21:11-744200 INFO Validating /home/alex/kohya_ss/Flex.1-alpha.safetensors
existence... SUCCESS
19:21:11-744764 INFO Validating /home/alex/FlexTest/images existence...
SUCCESS
19:21:11-745402 INFO Validating /home/alex/kohya_ss/ae.safetensors
existence... SUCCESS
19:21:11-746186 INFO Folder 5_1024photo: 5 repeats found
19:21:11-772477 INFO Folder 5_1024photo: 2997 images found
19:21:11-773280 INFO Folder 5_1024photo: 2997 * 5 = 14985 steps
19:21:11-774057 INFO Folder 5_hiresphotos: 5 repeats found
19:21:11-776724 INFO Folder 5_hiresphotos: 170 images found
19:21:11-777917 INFO Folder 5_hiresphotos: 170 * 5 = 850 steps
19:21:11-778958 INFO Folder 5_512photo: 5 repeats found
19:21:11-806888 INFO Folder 5_512photo: 2997 images found
19:21:11-807939 INFO Folder 5_512photo: 2997 * 5 = 14985 steps
19:21:11-808516 INFO Regularization factor: 1
19:21:11-809015 INFO Total steps: 30820
19:21:11-809505 WARNING Both lr_warmup and lr_warmup_steps are set.
lr_warmup_steps will be used.
19:21:11-810046 INFO Train batch size: 1
19:21:11-811001 INFO Gradient accumulation steps: 1
19:21:11-811565 INFO Epoch: 2
19:21:11-812066 INFO max_train_steps (30820 / 1 / 1 * 2 * 1) = 61640
19:21:11-812704 INFO lr_warmup_steps = 4240
19:21:11-814878 INFO Saving training config to
/home/alex/FlexTest/model/flextest_20250122-192111.json
...
19:21:11-815746 INFO Executing command:
/home/alex/kohya_ss/venv/bin/accelerate launch
--dynamo_backend no --dynamo_mode default
--mixed_precision bf16 --num_processes 1 --num_machines
1 --num_cpu_threads_per_process 4
/home/alex/kohya_ss/sd-scripts/flux_train.py
--config_file
/home/alex/FlexTest/model/config_dreambooth-20250122-19
2111.toml --bypass_flux_guidance
/home/alex/kohya_ss/venv/lib/python3.10/site-packages/diffusers/utils/outputs.py:63: FutureWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree._register_pytree_node(
2025-01-22 19:21:16.060291: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-01-22 19:21:16.060314: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-01-22 19:21:16.060852: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-01-22 19:21:16.064371: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-01-22 19:21:16.601468: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
/home/alex/kohya_ss/venv/lib/python3.10/site-packages/diffusers/utils/outputs.py:63: FutureWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree._register_pytree_node(
usage: flux_train.py [-h]
[--console_log_level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
[--console_log_file CONSOLE_LOG_FILE]
[--console_log_simple] [--v2] [--v_parameterization]
[--pretrained_model_name_or_path PRETRAINED_MODEL_NAME_OR_PATH]
[--tokenizer_cache_dir TOKENIZER_CACHE_DIR]
[--train_data_dir TRAIN_DATA_DIR] [--cache_info]
[--shuffle_caption]
[--caption_separator CAPTION_SEPARATOR]
[--caption_extension CAPTION_EXTENSION]
[--caption_extention CAPTION_EXTENTION]
[--keep_tokens KEEP_TOKENS]
[--keep_tokens_separator KEEP_TOKENS_SEPARATOR]
[--secondary_separator SECONDARY_SEPARATOR]
[--enable_wildcard] [--caption_prefix CAPTION_PREFIX]
[--caption_suffix CAPTION_SUFFIX] [--color_aug]
[--flip_aug] [--face_crop_aug_range FACE_CROP_AUG_RANGE]
[--random_crop] [--debug_dataset]
[--resolution RESOLUTION] [--cache_latents]
[--vae_batch_size VAE_BATCH_SIZE]
[--cache_latents_to_disk] [--skip_cache_check]
[--enable_bucket] [--min_bucket_reso MIN_BUCKET_RESO]
[--max_bucket_reso MAX_BUCKET_RESO]
[--bucket_reso_steps BUCKET_RESO_STEPS]
[--bucket_no_upscale]
[--token_warmup_min TOKEN_WARMUP_MIN]
[--token_warmup_step TOKEN_WARMUP_STEP] [--alpha_mask]
[--dataset_class DATASET_CLASS]
[--caption_dropout_rate CAPTION_DROPOUT_RATE]
[--caption_dropout_every_n_epochs CAPTION_DROPOUT_EVERY_N_EPOCHS]
[--caption_tag_dropout_rate CAPTION_TAG_DROPOUT_RATE]
[--reg_data_dir REG_DATA_DIR] [--in_json IN_JSON]
[--dataset_repeats DATASET_REPEATS]
[--output_dir OUTPUT_DIR] [--output_name OUTPUT_NAME]
[--huggingface_repo_id HUGGINGFACE_REPO_ID]
[--huggingface_repo_type HUGGINGFACE_REPO_TYPE]
[--huggingface_path_in_repo HUGGINGFACE_PATH_IN_REPO]
[--huggingface_token HUGGINGFACE_TOKEN]
[--huggingface_repo_visibility HUGGINGFACE_REPO_VISIBILITY]
[--save_state_to_huggingface] [--resume_from_huggingface]
[--async_upload]
[--save_precision {None,float,fp16,bf16}]
[--save_every_n_epochs SAVE_EVERY_N_EPOCHS]
[--save_every_n_steps SAVE_EVERY_N_STEPS]
[--save_n_epoch_ratio SAVE_N_EPOCH_RATIO]
[--save_last_n_epochs SAVE_LAST_N_EPOCHS]
[--save_last_n_epochs_state SAVE_LAST_N_EPOCHS_STATE]
[--save_last_n_steps SAVE_LAST_N_STEPS]
[--save_last_n_steps_state SAVE_LAST_N_STEPS_STATE]
[--save_state] [--save_state_on_train_end]
[--resume RESUME] [--train_batch_size TRAIN_BATCH_SIZE]
[--max_token_length {None,150,225}] [--mem_eff_attn]
[--torch_compile]
[--dynamo_backend {eager,aot_eager,inductor,aot_ts_nvfuser,nvprims_nvfuser,cudagraphs,ofi,fx2trt,onnxrt,tensort,ipex,tvm}]
[--xformers] [--sdpa] [--vae VAE]
[--max_train_steps MAX_TRAIN_STEPS]
[--max_train_epochs MAX_TRAIN_EPOCHS]
[--max_data_loader_n_workers MAX_DATA_LOADER_N_WORKERS]
[--persistent_data_loader_workers] [--seed SEED]
[--gradient_checkpointing]
[--gradient_accumulation_steps GRADIENT_ACCUMULATION_STEPS]
[--mixed_precision {no,fp16,bf16}] [--full_fp16]
[--full_bf16] [--fp8_base] [--ddp_timeout DDP_TIMEOUT]
[--ddp_gradient_as_bucket_view] [--ddp_static_graph]
[--clip_skip CLIP_SKIP] [--logging_dir LOGGING_DIR]
[--log_with {tensorboard,wandb,all}]
[--log_prefix LOG_PREFIX]
[--log_tracker_name LOG_TRACKER_NAME]
[--wandb_run_name WANDB_RUN_NAME]
[--log_tracker_config LOG_TRACKER_CONFIG]
[--wandb_api_key WANDB_API_KEY] [--log_config]
[--noise_offset NOISE_OFFSET]
[--noise_offset_random_strength]
[--multires_noise_iterations MULTIRES_NOISE_ITERATIONS]
[--ip_noise_gamma IP_NOISE_GAMMA]
[--ip_noise_gamma_random_strength]
[--multires_noise_discount MULTIRES_NOISE_DISCOUNT]
[--adaptive_noise_scale ADAPTIVE_NOISE_SCALE]
[--zero_terminal_snr] [--min_timestep MIN_TIMESTEP]
[--max_timestep MAX_TIMESTEP]
[--loss_type {l1,l2,huber,smooth_l1}]
[--huber_schedule {constant,exponential,snr}]
[--huber_c HUBER_C] [--huber_scale HUBER_SCALE]
[--lowram] [--highvram]
[--sample_every_n_steps SAMPLE_EVERY_N_STEPS]
[--sample_at_first]
[--sample_every_n_epochs SAMPLE_EVERY_N_EPOCHS]
[--sample_prompts SAMPLE_PROMPTS]
[--sample_sampler {ddim,pndm,lms,euler,euler_a,heun,dpm_2,dpm_2_a,dpmsolver,dpmsolver++,dpmsingle,k_lms,k_euler,k_euler_a,k_dpm_2,k_dpm_2_a}]
[--config_file CONFIG_FILE] [--output_config]
[--metadata_title METADATA_TITLE]
[--metadata_author METADATA_AUTHOR]
[--metadata_description METADATA_DESCRIPTION]
[--metadata_license METADATA_LICENSE]
[--metadata_tags METADATA_TAGS]
[--conditioning_data_dir CONDITIONING_DATA_DIR]
[--masked_loss] [--deepspeed] [--zero_stage {0,1,2,3}]
[--offload_optimizer_device {None,cpu,nvme}]
[--offload_optimizer_nvme_path OFFLOAD_OPTIMIZER_NVME_PATH]
[--offload_param_device {None,cpu,nvme}]
[--offload_param_nvme_path OFFLOAD_PARAM_NVME_PATH]
[--zero3_init_flag] [--zero3_save_16bit_model]
[--fp16_master_weights_and_gradients]
[--save_model_as {None,ckpt,safetensors,diffusers,diffusers_safetensors}]
[--use_safetensors] [--optimizer_type OPTIMIZER_TYPE]
[--use_8bit_adam] [--use_lion_optimizer]
[--learning_rate LEARNING_RATE]
[--max_grad_norm MAX_GRAD_NORM]
[--optimizer_args [OPTIMIZER_ARGS ...]]
[--lr_scheduler_type LR_SCHEDULER_TYPE]
[--lr_scheduler_args [LR_SCHEDULER_ARGS ...]]
[--lr_scheduler LR_SCHEDULER]
[--lr_warmup_steps LR_WARMUP_STEPS]
[--lr_decay_steps LR_DECAY_STEPS]
[--lr_scheduler_num_cycles LR_SCHEDULER_NUM_CYCLES]
[--lr_scheduler_power LR_SCHEDULER_POWER]
[--fused_backward_pass]
[--lr_scheduler_timescale LR_SCHEDULER_TIMESCALE]
[--lr_scheduler_min_lr_ratio LR_SCHEDULER_MIN_LR_RATIO]
[--dataset_config DATASET_CONFIG]
[--min_snr_gamma MIN_SNR_GAMMA]
[--scale_v_pred_loss_like_noise_pred]
[--v_pred_like_loss V_PRED_LIKE_LOSS]
[--debiased_estimation_loss] [--weighted_captions]
[--cache_text_encoder_outputs]
[--cache_text_encoder_outputs_to_disk]
[--text_encoder_batch_size TEXT_ENCODER_BATCH_SIZE]
[--disable_mmap_load_safetensors]
[--weighting_scheme {sigma_sqrt,logit_normal,mode,cosmap,none,uniform}]
[--logit_mean LOGIT_MEAN] [--logit_std LOGIT_STD]
[--mode_scale MODE_SCALE]
[--blocks_to_swap BLOCKS_TO_SWAP] [--clip_l CLIP_L]
[--t5xxl T5XXL] [--ae AE]
[--controlnet_model_name_or_path CONTROLNET_MODEL_NAME_OR_PATH]
[--t5xxl_max_token_length T5XXL_MAX_TOKEN_LENGTH]
[--apply_t5_attn_mask] [--guidance_scale GUIDANCE_SCALE]
[--timestep_sampling {sigma,uniform,sigmoid,shift,flux_shift}]
[--sigmoid_scale SIGMOID_SCALE]
[--model_prediction_type {raw,additive,sigma_scaled}]
[--discrete_flow_shift DISCRETE_FLOW_SHIFT]
[--mem_eff_save]
[--fused_optimizer_groups FUSED_OPTIMIZER_GROUPS]
[--blockwise_fused_optimizers]
[--skip_latents_validity_check]
[--double_blocks_to_swap DOUBLE_BLOCKS_TO_SWAP]
[--single_blocks_to_swap SINGLE_BLOCKS_TO_SWAP]
[--cpu_offload_checkpointing]
flux_train.py: error: unrecognized arguments: --bypass_flux_guidance
Traceback (most recent call last):
File "/home/alex/kohya_ss/venv/bin/accelerate", line 8, in
sys.exit(main())
File "/home/alex/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
args.func(args)
File "/home/alex/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1106, in launch_command
simple_launcher(args)
File "/home/alex/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 704, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/home/alex/kohya_ss/venv/bin/python3.10', '/home/alex/kohya_ss/sd-scripts/flux_train.py', '--config_file', '/home/alex/FlexTest/model/config_dreambooth-20250122-192111.toml', '--bypass_flux_guidance']' returned non-zero exit status 2.
19:21:19-051305 INFO Training has ended.

@stepfunction83
Copy link
Author

Let me try it with a fresh install...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@stepfunction83 @CodeAlexx