Skip to content

XTTS Model Finetuning Guide (Simple Version)

Jump to bottom
erew123 edited this page Nov 10, 2024 · 24 revisions

NOTE For those interested in highly detailed topics such as advanced training parameters, memory management, optimizing audio data quality, and understanding model performance metrics, Understanding Model Training Results, please refer to the XTTS Model Finetuning Guide (Advanced Version). Additionally, please refer to Coqui's documentation & forums for anything not covered in these guides https://docs.coqui.ai/en/latest/index.html

Table of Contents

  1. Quick Start Guide
  2. Core Concepts
  3. Technical Deep Dives
  4. Troubleshooting & Optimization
  5. Deploying Your Model

1. Quick Start Guide

Requirements

  • Hardware: Nvidia GPU with at least 12GB VRAM (Windows) or 16GB VRAM (Linux)
  • Software: CUDA 11.8 toolkit (requirements should be installed by the atsetup setup process)
  • Data: Minimum 2 minutes of clean voice samples (MP3, WAV, or FLAC)
  • Storage: 18GB of free disk space

Step-by-Step Process

  1. Prepare Environment:

    • Install and launch AllTalk.
    • Ensure CUDA toolkit is correctly installed (requirements should be installed by the atsetup setup process).
    • Close any GPU-intensive applications.

  2. Prepare Audio:

    • Place audio files in /finetune/put-voice-samples-in-here/.
    • Ensure audio is clear, free of background noise, and contains a variety of phrases.
    • Note: During training, temporary files are stored in finetune/tmp-trn, and Gradio (if used) manages temporary files in finetune/gradio_temp.

  3. Run Finetuning:

    • Launch the finetuning script: 
      # For standalone AllTalk:
cd alltalk_tts
./start_finetune.sh  # (Linux)
start_finetune.bat   # (Windows)
    • Follow the interface steps:
      • Step 1: Generate dataset
        • Use Dataset Validation Tab (if needed)
      • Step 2: Train model
      • Step 3: Test results
      • Step 4: Export model

  4. After Training:

    • Save your model.
    • Optionally delete training data to free up space.
    • Select the wav files to using for voice cloning Details here.

Common Issues & Quick Solutions

  • Out of Memory: Reduce batch size or use gradient accumulation.
  • Poor Quality Output: Verify audio quality and consider increasing training epochs.
  • Training Errors: Ensure CUDA is installed and GPU is available.
  • Slow Training: Adjust batch size and worker threads for efficiency.
  • Model Detection Issues: Ensure all required files (config.json, model.pth, mel_stats.pth, speakers_xtts.pth, vocab.json, dvae.pth) are present in the models/xtts/{base-model} folder.

2. Core Concepts

Understanding Finetuning

Finetuning is the process of adjusting a pre-trained model using new, specific voice data to better capture unique vocal characteristics. This is useful for creating personalized voice models that retain the base model's general abilities while improving on the nuances of your specific dataset.

When to Finetune:

  • Base model doesn’t capture desired vocal qualities.
  • Adaptation needed for accents, unique voices, or specific speaking styles.
  • Custom voices for particular applications.

Preparing Audio Data

  • Duration: Minimum of 2 minutes, 5–10 minutes recommended.
  • Content: Clear, noise-free, and consistent volume. Avoid background sounds or music.
  • Format: MP3, WAV, or FLAC. Any sample rate works, and stereo or mono is supported.

Tips:

  • Use varied sentence lengths, tones, and speaking speeds.
  • Pre-process to remove background noise, split long files if needed, and ensure content quality.

Training Process Overview

  1. Dataset Generation:

    • Audio is segmented and transcribed using Whisper.
    • Dataset split into training and evaluation sets (metadata_train.csv and metadata_eval.csv).
    • Use the Dataset Validation tab to look for inconsistencies in the transcriptions & correct as necessary.

  2. Model Training:

    • Set training parameters and monitor progress. Model learns to emulate the target voice.
    • Validation Paths: Paths for validation data and Whisper model transcription are used to validate and monitor quality. Customize these paths if you want to specify a different dataset for validation.

  3. Testing:

    • Run tests with different text inputs to evaluate quality, pronunciation, and emotional range.

  4. Model Export:

    • Compact model, save essential files, and clean up training artifacts.
    • After the model is compacted and moved, select the wav files from the models folder for using with voice cloning Details here.

3. Technical Deep Dives

Training Parameters

Epochs

An epoch is a full pass through the dataset. Choose the number of epochs based on the desired outcome:

  • Standard voices: 10–20 epochs
  • Highly unique voices or accents: 40+ epochs
  • Complex voices or new languages: May require 100+ epochs.

Tip: Monitor loss values to avoid overfitting. See Overtraining.

Batch Size

  • Small Batch Size (4–8): Lower memory use, more frequent updates.
  • Large Batch Size (32–64): Faster processing, requires more memory.

Gradient Accumulation: If VRAM is limited, simulate larger batch sizes with gradient accumulation:

batch_size = 8
gradient_accumulation_steps = 4  # Effective batch size = 32

Learning Rate

Controls how quickly the model learns:

  • 1e-6 to 5e-6: Stable, slow learning.
  • 1e-5: Balanced, suitable for most cases.
  • 1e-3 and higher: Fast but can be unstable.

Schedulers: Use learning rate scheduling (e.g., cosine annealing or exponential decay) for better results over long training runs.

Memory Management

  • Windows: Can use system RAM as extended VRAM.
  • Linux: Limited to physical VRAM, requiring higher VRAM for reliable operation.

Optimization Strategies:

  1. Lower batch size or increase gradient accumulation if memory is limited.
  2. Adjust worker threads and limit audio length for efficiency.
  3. Signal Handling: Use the GUI’s stop option or standard interrupts to safely halt training without corrupting model state.

Advanced Features

BPE Tokenization

The script uses Byte-Pair Encoding (BPE) tokenization, which helps the model handle complex text, diverse languages, accents, and dialects more effectively. This feature allows the model to better manage unique vocabularies and speech patterns.

4. Troubleshooting & Optimization

Improving Model Quality

You can read advanced sections on this topic here

  1. Signs of Good Training Progression:

    • Loss values consistently decrease over epochs. Example: 
      Text Loss: 0.02 -> 0.015 -> 0.012
Mel Loss: 4.5 -> 3.8 -> 3.2
Average Loss: 4.0 -> 3.5 -> 3.0
    • Model saves as "BEST MODEL" at new performance milestones.

  2. Recognizing Overtraining:

    • Loss values plateau or start increasing, e.g.,: 
      Epoch 5: Text Loss = 0.009, Mel Loss = 2.9
Epoch 7: Text Loss = 0.010, Mel Loss = 3.1  # Performance worsening
    • Solutions: Implement early stopping, reduce training epochs, or lower learning rate.

Performance Optimization

  1. Hardware:

    • Adjust batch size based on GPU capability.
    • Optimize worker count and monitor VRAM.

  2. Configuration:

    • Use efficient settings, e.g., float16 precision if supported.
    • Regularly monitor memory and processing efficiency.
    • Adjust gradient_accumulation_steps for limited VRAM.

5. Deploying Your Model

Model Export and Storage

After finetuning, export and organize the model files:

  • Essential Files:

    • model.pth: Main model
    • config.json: Configuration settings
    • vocab.json: Tokenizer vocabulary
    • speakers_xtts.pth: Speaker embeddings
    • dvae.pth: Discrete Variational Autoencoder (DVAE) model file
    • mel_stats.pth: Mel spectrogram statistics

  • Storage Requirements:

    • Model size: ~1.5GB
    • Reference audio varies based on content.

In the wavs folder, under the models folder, you will find an audio_report.txt along with a few folders. The audio_report.txt will explain which wav files are suitable to use for voice cloning with the XTTS model you have finetuned, along with details of what you can do with the other files if you so need. The files you want to use for voice cloning should be copied into your alltalk_tts\voices\ folder.

Integration and Usage

Folder Structure:

models/
└── xtts/
    └── your_model_name/
        ├── wavs/ 
        │    ├── audio_report.txt (READ THIS FILE)
        │    ├── suitable/ 
        │    ├── too_long/ 
        │    └── too_short/ 
        ├── model.pth
        ├── config.json
        ├── vocab.json
        ├── dvae.pth
        ├── mel_stats.pth
        └── speakers_xtts.pth

AllTalk Version 2 Index

Installation

System Requirements

Features
3rd Party Integrations
XTTS Finetuning Guides
API Documentation
Support & Help
Clone this wiki locally