[Bug]: not saving small model

[BornSaint]

Project Version

3.2.6

Platform and OS Version

Linux x64

Affected Devices

PC

Existing Issues

No response

What happened?

After playing with parameters, trainer stopped to save small models

Starting training...
Loaded pretrained (G) '/media/r/transition/Audio_AI/napplio/Applio/pretraineds/f0Ov2Super40kG.pth'
Loaded pretrained (D) '/media/r/transition/Audio_AI/napplio/Applio/pretraineds/f0Ov2Super40kD.pth'
Doomfist | epoch=1 | step=13 | time=19:10:26 | training_speed=0:00:10                                                                                              
Doomfist | epoch=2 | step=26 | time=19:10:33 | training_speed=0:00:07 | lowest_value=37.253 (epoch 2 and step 21)                                                  
Doomfist | epoch=3 | step=39 | time=19:10:40 | training_speed=0:00:06 | lowest_value=27.753 (epoch 3 and step 29)                                                  
Doomfist | epoch=4 | step=52 | time=19:10:48 | training_speed=0:00:07 | lowest_value=27.753 (epoch 3 and step 29)                                                  
Saved model '/media/r/transition/Audio_AI/napplio/Applio/logs/Doomfist/G_65.pth' (epoch 5)                                                                         
Saved model '/media/r/transition/Audio_AI/napplio/Applio/logs/Doomfist/D_65.pth' (epoch 5)
Doomfist | epoch=5 | step=65 | time=19:11:49 | training_speed=0:01:01 | lowest_value=27.753 (epoch 3 and step 29)
Doomfist | epoch=6 | step=78 | time=19:11:57 | training_speed=0:00:07 | lowest_value=24.576 (epoch 6 and step 71)                                                  
Doomfist | epoch=7 | step=91 | time=19:12:04 | training_speed=0:00:06 | lowest_value=24.576 (epoch 6 and step 71)                                                  
Doomfist | epoch=8 | step=104 | time=19:12:11 | training_speed=0:00:06 | lowest_value=24.576 (epoch 6 and step 71)                                                 
Doomfist | epoch=9 | step=117 | time=19:12:18 | training_speed=0:00:06 | lowest_value=24.576 (epoch 6 and step 71)                                                 
Saved model '/media/r/transition/Audio_AI/napplio/Applio/logs/Doomfist/G_130.pth' (epoch 10)                                                                       
Saved model '/media/r/transition/Audio_AI/napplio/Applio/logs/Doomfist/D_130.pth' (epoch 10)

import os
import time
import warnings
import sys
from shutil import rmtree
from subprocess import Popen

warnings.filterwarnings("ignore")
root = '/media/r/transition/Python/dataset_downloader/overwatch_tanks_dataset_2024'
characters = os.listdir(root)

sample_rate = '40000'
batch_size = '6'
total_epochs = '60'
save_every_epoch = '5'
save_only_latest = False
save_every_weights = False
custom_pretrained = True
overtraining_detector=False
overtraining_threshold=20
d_path = '/media/r/transition/Audio_AI/napplio/Applio/pretraineds/f0Ov2Super40kD.pth'
g_path = '/media/r/transition/Audio_AI/napplio/Applio/pretraineds/f0Ov2Super40kG.pth'

command = '''\
.venv/bin/python core.py preprocess \\
--model_name {voice_name} \\
--dataset_path {dataset_path} \\
--sample_rate {sample_rate} \\
--cpu_cores 6 \\
--cut_preprocess False \\
--process_effects True \\
--noise_reduction False && \\

.venv/bin/python core.py extract \\
--model_name {voice_name} \\
--rvc_version "v2" --f0_method "rmvpe" \\
--pitch_guidance "True" --cpu_cores 6 \\
--gpu 0 --sample_rate {sample_rate} \\
--embedder_model "contentvec" && \\

.venv/bin/python core.py index \\
--model_name {voice_name} \\
--rvc_version "v2" \\
--index_algorithm "Faiss" \
'''
train = '''\
.venv/bin/python core.py train \\
--model_name {voice_name} \\
--rvc_version "v2" \\
--save_every_epoch {save_every_epoch} \\
--save_only_latest "{save_only_latest}" \\
--save_every_weights "{save_every_weights}" \\
--sample_rate {sample_rate} \\
--total_epoch {total_epochs} \\
--batch_size {batch_size} \\
--cache_data_in_gpu "False" \\
--overtraining_detector {overtraining_detector} \\
--overtraining_threshold {overtraining_threshold} \\
--custom_pretrained {custom_pretrained} \\
--d_pretrained_path {d_path} \\
--g_pretrained_path {g_path}
'''
characters.sort()

try:
    for char in characters[1:]:
        # call('pwd')
        print('\n\ncurrent char:', char, f'{characters.index(char)}/{len(characters)}\n\n')


        if char != "Ramattra" and not os.path.exists(os.path.join('logs', char)):
            dataset_path = os.path.join(root, char)

            command_string = ' && \\\n'.join([command,train]).format(voice_name=char, 
                                    dataset_path=dataset_path, 
                                    sample_rate=sample_rate, 
                                    save_every_epoch=save_every_epoch, 
                                    total_epochs=total_epochs,
                                    batch_size=batch_size,
                                    custom_pretrained=custom_pretrained,
                                    save_every_weights=save_every_weights,
                                    save_only_latest=save_only_latest,
                                    overtraining_detector=overtraining_detector,
                                    overtraining_threshold=overtraining_threshold,
                                    d_path=d_path,
                                    g_path=g_path)
        else:
            command_string = train.format(voice_name=char, 
                                    sample_rate=sample_rate, 
                                    save_every_epoch=save_every_epoch, 
                                    total_epochs=total_epochs,
                                    batch_size=batch_size,
                                    custom_pretrained=custom_pretrained,
                                    save_every_weights=save_every_weights,
                                    save_only_latest=save_only_latest,
                                    overtraining_detector=overtraining_detector,
                                    overtraining_threshold=overtraining_threshold,
                                    d_path=d_path, 
                                    g_path=g_path)
        with open('temp_script.sh', 'w', encoding='utf-8') as f:
            f.write(command_string)
        a = Popen('./temp_script.sh', shell=True)
        a.wait()
except KeyboardInterrupt:
    ...
finally:
    os.remove('./temp_script.sh')

Steps to reproduce

1. run the python script

Expected behavior

save only small models

Attachments

No response

Screenshots or Videos

No response

Additional Information

No response

[AznamirWoW]

what do you think save_every_weights = False does

[BornSaint]

what do you think save_every_weights = False does

a few moments ago it was working like this:
when True it was saving the D, G and the small model
when False it was saving only small model