[Bug]: Failed to obtain Lora after training, please check the training process.没有发现具体的错误 #413
Open
2 tasks done
Comments
longglecc
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is there an existing issue for this?
Is EasyPhoto the latest version?
What happened?
Loading weights [59ffe2243a] from /home/gaol/codes/temp/stable-diffusion-webui/models/Stable-diffusion/Chilloutmix-Ni-pruned-fp16-fix.safetensors
2024-03-16 15:22:02,890 - modelscope - INFO - PyTorch version 2.0.1+cu118 Found.
2024-03-16 15:22:02,891 - modelscope - INFO - TensorFlow version 2.13.0 Found.
2024-03-16 15:22:02,891 - modelscope - INFO - Loading ast index from /home/gaol/.cache/modelscope/ast_indexer
2024-03-16 15:22:02,947 - modelscope - INFO - Loading done! Current index file version is 1.9.3, with md5 985d60ab3829178ada728d5649a2ffda and a total number of 943 components indexed
2024-03-16 15:22:03,980 - modelscope - INFO - Use user-specified model revision: v1.0.3
2024-03-16 15:22:04,303 - modelscope - WARNING - ('PIPELINES', 'face_recognition', 'face_recognition') not found in ast index file
2024-03-16 15:22:04,303 - modelscope - INFO - initiate model from /home/gaol/.cache/modelscope/hub/bubbliiiing/cv_retinafce_recognition
2024-03-16 15:22:04,303 - modelscope - INFO - initiate model from location /home/gaol/.cache/modelscope/hub/bubbliiiing/cv_retinafce_recognition.
2024-03-16 15:22:04,304 - modelscope - INFO - initialize model from /home/gaol/.cache/modelscope/hub/bubbliiiing/cv_retinafce_recognition
2024-03-16 15:22:04,306 - modelscope - WARNING - ('MODELS', 'face_recognition', 'face_recognition') not found in ast index file
2024-03-16 15:22:04,850 - modelscope - INFO - Model revision not specified, use revision: v2.0.2
2024-03-16 15:22:06,525 - modelscope - INFO - initiate model from /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface
2024-03-16 15:22:06,525 - modelscope - INFO - initiate model from location /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface.
2024-03-16 15:22:06,526 - modelscope - WARNING - No preprocessor field found in cfg.
2024-03-16 15:22:06,526 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file.
2024-03-16 15:22:06,526 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'model_dir': '/home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface'}. trying to build by task and model information.
2024-03-16 15:22:06,526 - modelscope - WARNING - Find task: face-detection, model type: None. Insufficient information to build preprocessor, skip building preprocessor
2024-03-16 15:22:06,526 - modelscope - INFO - loading model from /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface/pytorch_model.pt
/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
warnings.warn(
/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or
Nonefor 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passingweights=None.warnings.warn(msg)
2024-03-16 15:22:06,914 - modelscope - INFO - load model done
2024-03-16 15:22:07,571 - modelscope - INFO - load facefusion models done
2024-03-16 15:22:07,571 - modelscope - INFO - init done
2024-03-16 15:22:07,572 - modelscope - WARNING - No preprocessor field found in cfg.
2024-03-16 15:22:07,572 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file.
2024-03-16 15:22:07,572 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'model_dir': '/home/gaol/.cache/modelscope/hub/bubbliiiing/cv_retinafce_recognition'}. trying to build by task and model information.
2024-03-16 15:22:07,572 - modelscope - WARNING - No preprocessor key ('face_recognition', 'face_recognition') found in PREPROCESSOR_MAP, skip building preprocessor.
2024-03-16 15:22:07,572 - modelscope - INFO - image face recognition model init done
2024-03-16 15:22:08,121 - modelscope - INFO - Use user-specified model revision: v2.0.2
2024-03-16 15:22:09,744 - modelscope - INFO - initiate model from /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface
2024-03-16 15:22:09,744 - modelscope - INFO - initiate model from location /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface.
2024-03-16 15:22:09,746 - modelscope - WARNING - No preprocessor field found in cfg.
2024-03-16 15:22:09,746 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file.
2024-03-16 15:22:09,746 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'model_dir': '/home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface'}. trying to build by task and model information.
2024-03-16 15:22:09,746 - modelscope - WARNING - Find task: face-detection, model type: None. Insufficient information to build preprocessor, skip building preprocessor
2024-03-16 15:22:09,746 - modelscope - INFO - loading model from /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface/pytorch_model.pt
2024-03-16 15:22:10,014 - modelscope - INFO - load model done
2024-03-16 15:22:10,645 - modelscope - INFO - Use user-specified model revision: v1.0.0
2024-03-16 15:22:10,832 - modelscope - INFO - initiate model from /home/gaol/.cache/modelscope/hub/damo/cv_u2net_salient-detection
2024-03-16 15:22:10,832 - modelscope - INFO - initiate model from location /home/gaol/.cache/modelscope/hub/damo/cv_u2net_salient-detection.
2024-03-16 15:22:10,834 - modelscope - INFO - initialize model from /home/gaol/.cache/modelscope/hub/damo/cv_u2net_salient-detection
2024-03-16 15:22:11,091 - modelscope - WARNING - No preprocessor field found in cfg.
2024-03-16 15:22:11,091 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file.
2024-03-16 15:22:11,091 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'model_dir': '/home/gaol/.cache/modelscope/hub/damo/cv_u2net_salient-detection'}. trying to build by task and model information.
2024-03-16 15:22:11,091 - modelscope - WARNING - No preprocessor key ('detection', 'semantic-segmentation') found in PREPROCESSOR_MAP, skip building preprocessor.
2024-03-16 15:22:11,691 - modelscope - INFO - Use user-specified model revision: v1.0.2
2024-03-16 15:22:11,943 - modelscope - WARNING - ('PIPELINES', 'skin-retouching-torch', 'skin-retouching-torch') not found in ast index file
2024-03-16 15:22:11,943 - modelscope - INFO - initiate model from /home/gaol/.cache/modelscope/hub/damo/cv_unet_skin_retouching_torch
2024-03-16 15:22:11,943 - modelscope - INFO - initiate model from location /home/gaol/.cache/modelscope/hub/damo/cv_unet_skin_retouching_torch.
2024-03-16 15:22:11,944 - modelscope - WARNING - No preprocessor field found in cfg.
2024-03-16 15:22:11,944 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file.
2024-03-16 15:22:11,944 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'model_dir': '/home/gaol/.cache/modelscope/hub/damo/cv_unet_skin_retouching_torch'}. trying to build by task and model information.
2024-03-16 15:22:11,944 - modelscope - WARNING - Find task: skin-retouching-torch, model type: None. Insufficient information to build preprocessor, skip building preprocessor
2024-03-16 15:22:12,637 - modelscope - INFO - Model revision not specified, use revision: v2.0.2
2024-03-16 15:22:14,524 - modelscope - INFO - initiate model from /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface
2024-03-16 15:22:14,524 - modelscope - INFO - initiate model from location /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface.
2024-03-16 15:22:14,526 - modelscope - WARNING - No preprocessor field found in cfg.
2024-03-16 15:22:14,526 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file.
2024-03-16 15:22:14,526 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'model_dir': '/home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface'}. trying to build by task and model information.
2024-03-16 15:22:14,526 - modelscope - WARNING - Find task: face-detection, model type: None. Insufficient information to build preprocessor, skip building preprocessor
2024-03-16 15:22:14,526 - modelscope - INFO - loading model from /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface/pytorch_model.pt
2024-03-16 15:22:14,797 - modelscope - INFO - load model done
2024-03-16 15:22:15.412807243 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'Sub__1664:0'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.412822062 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'Shape__1662:0'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.412831040 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_4__1660'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525554751 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt__2649'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525574428 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt__2657'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525577167 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt__2644'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525579922 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt__2594'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525583236 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt__2596'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525586891 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt__2653'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525589904 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt__2624'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525592868 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt__2652'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525596530 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt__2645'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525599328 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt__2643'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525602181 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt__2648'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525604868 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt__2647'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525607502 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt__2641'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525611251 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt__2633'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525614155 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt__2632'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525617785 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt__2614'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525620526 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt__2613'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525623573 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt__2658'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525626884 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt__2606'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:15.525630091 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'const_fold_opt__2598'. It is not used by any node and should be removed from the model.
2024-03-16 15:22:16,225 - modelscope - INFO - Use user-specified model revision: v1.0.0
2024-03-16 15:22:16,454 - modelscope - INFO - initiate model from /home/gaol/.cache/modelscope/hub/damo/cv_gpen_image-portrait-enhancement
2024-03-16 15:22:16,455 - modelscope - INFO - initiate model from location /home/gaol/.cache/modelscope/hub/damo/cv_gpen_image-portrait-enhancement.
2024-03-16 15:22:16,456 - modelscope - INFO - initialize model from /home/gaol/.cache/modelscope/hub/damo/cv_gpen_image-portrait-enhancement
Loading ResNet ArcFace
2024-03-16 15:22:18,087 - modelscope - INFO - load face enhancer model done
2024-03-16 15:22:18,337 - modelscope - INFO - load face detector model done
2024-03-16 15:22:18,583 - modelscope - INFO - load sr model done
2024-03-16 15:22:19,307 - modelscope - INFO - load fqa model done
0%| | 0/15 [00:00<?, ?it/s]2024-03-16 15:22:19,761 - modelscope - WARNING - task skin-retouching-torch input definition is missing
2024-03-16 15:22:21,407 - modelscope - WARNING - task skin-retouching-torch output keys are missing
2024-03-16 15:22:21,411 - modelscope - WARNING - task face_recognition input definition is missing
2024-03-16 15:22:21,555 - modelscope - INFO - model inference done
2024-03-16 15:22:21,555 - modelscope - WARNING - task face_recognition output keys are missing
7%|?????????????? | 1/15 [00:02<00:31, 2.22s/it]2024-03-16 15:22:22,273 - modelscope - INFO - model inference done
13%|???????????????????????????? | 2/15 [00:02<00:17, 1.34s/it]2024-03-16 15:22:22,961 - modelscope - INFO - model inference done
20%|????????????????????????????????????????? | 3/15 [00:03<00:12, 1.04s/it]2024-03-16 15:22:23,690 - modelscope - INFO - model inference done
27%|??????????????????????????????????????????????????????? | 4/15 [00:04<00:10, 1.09it/s]2024-03-16 15:22:24,654 - modelscope - INFO - model inference done
33%|????????????????????????????????????????????????????????????????????? | 5/15 [00:05<00:09, 1.07it/s]2024-03-16 15:22:25,290 - modelscope - INFO - model inference done
40%|?????????????????????????????????????????????????????????????????????????????????? | 6/15 [00:05<00:07, 1.20it/s]2024-03-16 15:22:26,010 - modelscope - INFO - model inference done
47%|???????????????????????????????????????????????????????????????????????????????????????????????? | 7/15 [00:06<00:06, 1.26it/s]2024-03-16 15:22:27,181 - modelscope - INFO - model inference done
53%|?????????????????????????????????????????????????????????????????????????????????????????????????????????????? | 8/15 [00:07<00:06, 1.09it/s]2024-03-16 15:22:28,236 - modelscope - INFO - model inference done
60%|??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? | 9/15 [00:08<00:05, 1.04it/s]2024-03-16 15:22:29,217 - modelscope - INFO - model inference done
67%|???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? | 10/15 [00:09<00:04, 1.04it/s]2024-03-16 15:22:30,171 - modelscope - INFO - model inference done
73%|?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? | 11/15 [00:10<00:03, 1.04it/s]2024-03-16 15:22:31,393 - modelscope - INFO - model inference done
80%|???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? | 12/15 [00:12<00:03, 1.04s/it]2024-03-16 15:22:32,414 - modelscope - INFO - model inference done
87%|????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? | 13/15 [00:13<00:02, 1.03s/it]2024-03-16 15:22:33,103 - modelscope - INFO - model inference done
93%|??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? | 14/15 [00:13<00:00, 1.07it/s]2024-03-16 15:22:33,818 - modelscope - INFO - model inference done
100%|????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 15/15 [00:14<00:00, 1.04it/s]
selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/0.jpg total scores: 0.5047546609308565 face angles 0.9206686066500163
selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/12.jpg total scores: 0.49227943258374113 face angles 0.9208677161732743
selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/6.jpg total scores: 0.4752780163014266 face angles 0.9655220357517296
selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/2.jpg total scores: 0.4714630612945019 face angles 0.9158910610739464
selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/13.jpg total scores: 0.46400164259168414 face angles 0.9542378038928844
selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/4.jpg total scores: 0.43687105673798315 face angles 0.8650432225418873
selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/7.jpg total scores: 0.4353354856725917 face angles 0.9829790242269123
selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/11.jpg total scores: 0.4166387907586268 face angles 0.7875557185843958
selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/8.jpg total scores: 0.4151525819786572 face angles 0.9470527983614994
selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/5.jpg total scores: 0.3955914799303482 face angles 0.9771394596698413
selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/14.jpg total scores: 0.38413634652262346 face angles 0.9993536825291083
selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/10.jpg total scores: 0.34892923676821336 face angles 0.7398419459302334
selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/9.jpg total scores: 0.3327070567244012 face angles 0.8863536608311919
selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/1.jpg total scores: 0.3145806877875612 face angles 0.6169503507383685
selected paths: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/original_backup/3.jpg total scores: 0.3134642199143675 face angles 0.8505665005567431
jpg: 0.jpg face_id_scores 0.5047546609308565
jpg: 12.jpg face_id_scores 0.49227943258374113
jpg: 11.jpg face_id_scores 0.4166387907586268
jpg: 2.jpg face_id_scores 0.4714630612945019
jpg: 1.jpg face_id_scores 0.3145806877875612
jpg: 4.jpg face_id_scores 0.43687105673798315
jpg: 6.jpg face_id_scores 0.4752780163014266
jpg: 13.jpg face_id_scores 0.46400164259168414
jpg: 10.jpg face_id_scores 0.34892923676821336
jpg: 7.jpg face_id_scores 0.4353354856725917
jpg: 8.jpg face_id_scores 0.4151525819786572
jpg: 5.jpg face_id_scores 0.3955914799303482
jpg: 14.jpg face_id_scores 0.38413634652262346
jpg: 9.jpg face_id_scores 0.3327070567244012
jpg: 3.jpg face_id_scores 0.3134642199143675
15it [00:13, 1.12it/s]
save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/0.jpg
save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/1.jpg
save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/2.jpg
save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/3.jpg
save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/4.jpg
save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/5.jpg
save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/6.jpg
save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/7.jpg
save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/8.jpg
save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/9.jpg
save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/10.jpg
save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/11.jpg
save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/12.jpg
save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/13.jpg
save processed image to /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images/train/14.jpg
2024-03-16 15:22:47,750 - EasyPhoto - train_file_path : /home/gaol/codes/temp/stable-diffusion-webui/extensions/sd-webui-EasyPhoto/scripts/train_kohya/train_lora.py
2024-03-16 15:22:47,750 - EasyPhoto - cache_log_file_path: /home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-tmp/train_kohya_log.txt
2024-03-16 15:22:49.493300: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-03-16 15:22:52.402091: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-03-16 15:22:53,191 - modelscope - INFO - PyTorch version 2.0.1+cu118 Found.
2024-03-16 15:22:53,192 - modelscope - INFO - TensorFlow version 2.13.0 Found.
2024-03-16 15:22:53,192 - modelscope - INFO - Loading ast index from /home/gaol/.cache/modelscope/ast_indexer
2024-03-16 15:22:53,209 - modelscope - INFO - Loading done! Current index file version is 1.9.3, with md5 985d60ab3829178ada728d5649a2ffda and a total number of 943 components indexed
03/16/2024 15:22:53 - INFO - main - Distributed environment: MULTI_GPU Backend: nccl
Num processes: 1
Process index: 0
Local process index: 0
Device: cuda:0
Mixed precision type: fp16
{'sample_max_value', 'variance_type', 'thresholding', 'clip_sample_range', 'dynamic_thresholding_ratio', 'timestep_spacing', 'prediction_type'} was not found in config. Values will be initialized to default values.
UNet2DConditionModel: 64, 8, 768, False, False
loading u-net:
loading vae:
loading text encoder:
create LoRA network. base dim (rank): 128, alpha: 64
neuron dropout: p=None, rank dropout: p=None, module dropout: p=None
create LoRA for Text Encoder:
create LoRA for Text Encoder: 72 modules.
create LoRA for U-Net: 192 modules.
enable LoRA for text encoder
enable LoRA for U-Net
Resolving data files: 100%|??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 31/31 [00:00<00:00, 436320.21it/s]
Downloading and preparing dataset imagefolder/default to /home/gaol/.cache/huggingface/datasets/imagefolder/default-cd3ed4aa3d015dcd/0.0.0/37fbb85cc714a338bea574ac6c7d0b5be5aff46c1862c1989b20e0771199e93f...
Downloading data files: 100%|????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 16/16 [00:00<00:00, 106017.16it/s]
Downloading data files: 100%|????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 15/15 [00:00<00:00, 116293.09it/s]
Extracting data files: 100%|???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 15/15 [00:00<00:00, 6948.05it/s]
Dataset imagefolder downloaded and prepared to /home/gaol/.cache/huggingface/datasets/imagefolder/default-cd3ed4aa3d015dcd/0.0.0/37fbb85cc714a338bea574ac6c7d0b5be5aff46c1862c1989b20e0771199e93f. Subsequent calls will reuse this data.
100%|????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 1/1 [00:00<00:00, 1410.32it/s]
03/16/2024 15:23:01 - INFO - main - ***** Running training *****
03/16/2024 15:23:01 - INFO - main - Num examples = 15
03/16/2024 15:23:01 - INFO - main - Num Epochs = 3000
03/16/2024 15:23:01 - INFO - main - Instantaneous batch size per device = 8
03/16/2024 15:23:01 - INFO - main - Total train batch size (w. parallel, distributed & accumulation) = 64
03/16/2024 15:23:01 - INFO - main - Gradient Accumulation steps = 8
03/16/2024 15:23:01 - INFO - main - Total optimization steps = 3000
Steps: 0%| | 0/3000 [00:00<?, ?it/s]2024-03-16 15:23:02,628 - modelscope - INFO - Model revision not specified, use revision: v2.0.2
2024-03-16 15:23:04,325 - modelscope - INFO - initiate model from /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface
2024-03-16 15:23:04,325 - modelscope - INFO - initiate model from location /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface.
2024-03-16 15:23:04,326 - modelscope - WARNING - No preprocessor field found in cfg.
2024-03-16 15:23:04,326 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file.
2024-03-16 15:23:04,326 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'model_dir': '/home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface'}. trying to build by task and model information.
2024-03-16 15:23:04,326 - modelscope - WARNING - Find task: face-detection, model type: None. Insufficient information to build preprocessor, skip building preprocessor
2024-03-16 15:23:04,326 - modelscope - INFO - loading model from /home/gaol/.cache/modelscope/hub/damo/cv_resnet50_face-detection_retinaface/pytorch_model.pt
/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
warnings.warn(
/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or
Nonefor 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passingweights=None.warnings.warn(msg)
2024-03-16 15:23:04,575 - modelscope - INFO - load model done
/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/modules/conv.py:459: UserWarning: Applied workaround for CuDNN issue, install nvrtc.so (Triggered internally at ../aten/src/ATen/native/cudnn/Conv_v8.cpp:80.)
return F.conv2d(input, weight, bias, self.stride,
[2024-03-16 15:23:09,607] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
[2024-03-16 15:23:09,809] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing call
[2024-03-16 15:23:09,812] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
[2024-03-16 15:23:11,013] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2024-03-16 15:23:11,029] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function debug_wrapper
Traceback (most recent call last):
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/dynamo/output_graph.py", line 670, in call_user_compiler
compiled_fn = compiler_fn(gm, self.fake_example_inputs())
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/dynamo/debug_utils.py", line 1055, in debug_wrapper
compiled_gm = compiler_fn(gm, example_inputs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/init.py", line 1390, in call
return compile_fx(model, inputs, config_patches=self.config)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 455, in compile_fx
return aot_autograd(
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 48, in compiler_fn
cg = aot_module_simplified(gm, example_inputs, **kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2822, in aot_module_simplified
compiled_fn = create_aot_dispatcher_function(
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper
r = func(*args, **kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2515, in create_aot_dispatcher_function
compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1676, in aot_wrapper_dedupe
fw_metadata, _out = run_functionalized_fw_and_collect_metadata(
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 607, in inner
flat_f_outs = f(*flat_f_args)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2793, in functional_call
out = Interpreter(mod).run(*args[params_len:], **kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/fx/interpreter.py", line 136, in run
self.env[node] = self.run_node(node)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/fx/interpreter.py", line 177, in run_node
return getattr(self, n.op)(n.target, args, kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/fx/interpreter.py", line 294, in call_module
return submod(*args, **kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/extensions/sd-webui-EasyPhoto/scripts/train_kohya/utils/lora_utils.py", line 140, in forward
lx = self.lora_down(x)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_inductor/overrides.py", line 38, in torch_function
return func(*args, **kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/utils/_stats.py", line 20, in wrapper
return fn(args, **kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 987, in torch_dispatch
return self.dispatch(func, types, args, kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1066, in dispatch
args, kwargs = self.validate_and_convert_non_fake_tensors(
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1220, in validate_and_convert_non_fake_tensors
return tree_map_only(
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/utils/_pytree.py", line 266, in tree_map_only
return tree_map(map_only(ty)(fn), pytree)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/utils/_pytree.py", line 196, in tree_map
return tree_unflatten([fn(i) for i in flat_args], spec)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/utils/_pytree.py", line 196, in
return tree_unflatten([fn(i) for i in flat_args], spec)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/utils/_pytree.py", line 247, in inner
return f(x)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1212, in validate
raise Exception(
Exception: Please convert all Tensors to FakeTensors first or instantiate FakeTensorMode with 'allow_non_fake_inputs'. Found in aten._to_copy.default((Parameter containing:
tensor([[-0.0176, -0.0284, 0.0112, ..., 0.0230, 0.0277, 0.0246],
[ 0.0259, 0.0047, 0.0272, ..., 0.0275, 0.0020, 0.0336],
[-0.0033, -0.0350, 0.0030, ..., -0.0004, 0.0218, 0.0237],
...,
[ 0.0084, -0.0230, -0.0020, ..., 0.0119, 0.0042, 0.0045],
[-0.0068, -0.0207, -0.0300, ..., -0.0241, 0.0173, 0.0291],
[-0.0163, -0.0165, 0.0014, ..., -0.0115, 0.0128, 0.0221]],
device='cuda:0', requires_grad=True),), **{'dtype': torch.float16})
While executing %self_text_model_encoder_layers_0_self_attn_q_proj : [#users=1] = call_module[target=self_text_model_encoder_layers_0_self_attn_q_proj](args = (%self_text_model_encoder_layers_0_layer_norm1,), kwargs = {})
Original traceback:
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 272, in forward
query_states = self.q_proj(hidden_states) * self.scale
| File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 383, in forward
hidden_states, attn_weights = self.self_attn(
| File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 654, in forward
layer_outputs = encoder_layer(
| File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 740, in forward
encoder_outputs = self.encoder(
| File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 822, in forward
return self.text_model(
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/gaol/codes/temp/stable-diffusion-webui/extensions/sd-webui-EasyPhoto/scripts/train_kohya/train_lora.py", line 1397, in
main()
File "/home/gaol/codes/temp/stable-diffusion-webui/extensions/sd-webui-EasyPhoto/scripts/train_kohya/utils/gpu_info.py", line 190, in wrapper
result = func(*args, **kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/extensions/sd-webui-EasyPhoto/scripts/train_kohya/train_lora.py", line 1132, in main
encoder_hidden_states = text_encoder(batch["input_ids"])[0]
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 82, in forward
return self.dynamo_ctx(self._orig_mod.forward)(*args, **kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 209, in _fn
return fn(*args, **kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/utils/operations.py", line 581, in forward
return model_forward(*args, **kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/utils/operations.py", line 569, in call
return convert_to_fp32(self.model_forward(*args, **kwargs))
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
return func(*args, **kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 337, in catch_errors
return callback(frame, cache_size, hooks)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 404, in _convert_frame
result = inner_convert(frame, cache_size, hooks)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 104, in _fn
return fn(*args, **kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 262, in _convert_frame_assert
return _compile(
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper
r = func(*args, **kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 324, in _compile
out_code = transform_code_object(code, transform)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 445, in transform_code_object
transformations(instructions, code_options)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 311, in transform
tracer.run()
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1726, in run
super().run()
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 576, in run
and self.step()
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 540, in step
getattr(self, inst.opname)(inst)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1792, in RETURN_VALUE
self.output.compile_subgraph(
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 541, in compile_subgraph
self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 588, in compile_and_call_fx_graph
compiled_fn = self.call_user_compiler(gm)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper
r = func(args, **kwargs)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 675, in call_user_compiler
raise BackendCompilerFailed(self.compiler_fn, e) from e
torch._dynamo.exc.BackendCompilerFailed: debug_wrapper raised Exception: Please convert all Tensors to FakeTensors first or instantiate FakeTensorMode with 'allow_non_fake_inputs'. Found in aten._to_copy.default((Parameter containing:
tensor([[-0.0176, -0.0284, 0.0112, ..., 0.0230, 0.0277, 0.0246],
[ 0.0259, 0.0047, 0.0272, ..., 0.0275, 0.0020, 0.0336],
[-0.0033, -0.0350, 0.0030, ..., -0.0004, 0.0218, 0.0237],
...,
[ 0.0084, -0.0230, -0.0020, ..., 0.0119, 0.0042, 0.0045],
[-0.0068, -0.0207, -0.0300, ..., -0.0241, 0.0173, 0.0291],
[-0.0163, -0.0165, 0.0014, ..., -0.0115, 0.0128, 0.0221]],
device='cuda:0', requires_grad=True),), **{'dtype': torch.float16})
While executing %self_text_model_encoder_layers_0_self_attn_q_proj : [#users=1] = call_module[target=self_text_model_encoder_layers_0_self_attn_q_proj](args = (%self_text_model_encoder_layers_0_layer_norm1,), kwargs = {})
Original traceback:
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 272, in forward
query_states = self.q_proj(hidden_states) * self.scale
| File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 383, in forward
hidden_states, attn_weights = self.self_attn(
| File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 654, in forward
layer_outputs = encoder_layer(
| File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 740, in forward
encoder_outputs = self.encoder(
| File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 822, in forward
return self.text_model(
Set torch._dynamo.config.verbose=True for more information
You can suppress this exception and fall back to eager by setting:
torch._dynamo.config.suppress_errors = True
Steps: 0%| | 0/3000 [00:11<?, ?it/s]
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 2387) of binary: /home/gaol/codes/temp/stable-diffusion-webui/venv/bin/python3
Traceback (most recent call last):
File "/home/gaol/miniforge3/envs/stable/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/gaol/miniforge3/envs/stable/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 989, in
main()
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 985, in main
launch_command(args)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 970, in launch_command
multi_gpu_launcher(args)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 646, in multi_gpu_launcher
distrib_run.run(args)
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/gaol/codes/temp/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
/home/gaol/codes/temp/stable-diffusion-webui/extensions/sd-webui-EasyPhoto/scripts/train_kohya/train_lora.py FAILED
Failures:
<NO_OTHER_FAILURES>
Root Cause (first observed failure):
[0]:
time : 2024-03-16_15:23:15
host : rtx3060
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 2387)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
2024-03-16 15:23:16,093 - EasyPhoto - Error executing the command: Command '['/home/gaol/codes/temp/stable-diffusion-webui/venv/bin/python3', '-m', 'accelerate.commands.launch', '--mixed_precision=fp16', '--main_process_port=3456', '/home/gaol/codes/temp/stable-diffusion-webui/extensions/sd-webui-EasyPhoto/scripts/train_kohya/train_lora.py', '--pretrained_model_name_or_path=/home/gaol/codes/temp/stable-diffusion-webui/extensions/sd-webui-EasyPhoto/models/stable-diffusion-v1-5', '--pretrained_model_ckpt=/home/gaol/codes/temp/stable-diffusion-webui/models/Stable-diffusion/Chilloutmix-Ni-pruned-fp16-fix.safetensors', '--train_data_dir=/home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/processed_images', '--caption_column=text', '--resolution=512', '--random_flip', '--train_batch_size=8', '--gradient_accumulation_steps=8', '--dataloader_num_workers=12', '--max_train_steps=3000', '--checkpointing_steps=100', '--learning_rate=0.0001', '--lr_scheduler=constant', '--lr_warmup_steps=0', '--train_text_encoder', '--seed=124102', '--rank=128', '--network_alpha=64', '--validation_prompt=easyphoto_face, easyphoto, 1person', '--validation_steps=100', '--output_dir=/home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/user_weights', '--logging_dir=/home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-user-id-infos/li/user_weights', '--enable_xformers_memory_efficient_attention', '--mixed_precision=fp16', '--template_dir=/home/gaol/codes/temp/stable-diffusion-webui/extensions/sd-webui-EasyPhoto/models/training_templates', '--template_mask', '--merge_best_lora_based_face_id', '--merge_best_lora_name=li', '--cache_log_file=/home/gaol/codes/temp/stable-diffusion-webui/outputs/easyphoto-tmp/train_kohya_log.txt']' returned non-zero exit status 1.
Applying attention optimization: Doggettx... done.
Steps to reproduce the problem
What should have happened?
RTX3060
Commit where the problem happens
webui:
EastPhoto:
What browsers do you use to access the UI ?
No response
Command Line Arguments
List of enabled extensions
Console logs
Additional information
No response
The text was updated successfully, but these errors were encountered: