VQ-VAE is a type of variational autoencoder that uses vector quantisation to obtain a discrete latent representation. It differs from VAEs in two key ways:
- Encoder network outputs discrete, rather than continuous, codes.
- A prior is learnt rather than static.
In order to learn a discrete latent representation, ideas from vector quantisation (VQ) are incorporated. Using the VQ method allows the model to circumvent issues of posterior collapse - where the latents are ignored when they are paired with a powerful autoregressive decoder - typically observed in the VAE framework. Pairing these representations with an autoregressive prior, the model can generate high quality images, videos, and speech as well as doing high quality speaker conversion and unsupervised learning of phonemes.
- Initially VQVAE is trained to learn discreate features from the images through a Image recontruction task.
- Later we collect all the Discreate latent codes and train a prior ontop of these latent codes.
- Here we choose gpt as our prior model which will predict the next tokens based on the previously predicted tokens.
