WAIVE-Plugins

AI x Archive music tools

A plugin suite that combines music, sound and MIDI generation with European cultural archives. Aims to be an offline, modular version of WAIVE-Studio that can be integrated into your DAW. Built with DISTRHO Plugin Framework and ONNX Runtime.

• WAIVE-Sequencer: a rhythmic pattern generator
• WAIVE-Sampler: a sample player, sample library and sample generator all in one

Developed by Arran Lyon for Thunderboom Records. Contributions and pull-requests welcome, especially regarding stability and performance improvements.

Download and Install

WAIVE-Sampler and WAIVE-Sequencer are available in VST, VST3, CLAP and Audio Unit (macOS only) plugin formats.

1. Download the installer from the Releases page for your platform. You can find the download links under the Assets heading. For newer Apple computers with M series processors, use macOS_arm64, whereas older machines with Intel chips should use macOS_x64.
2. Install:
   • MacOS: open the .dmg file, open the .pkg and follow the instructions.
   • Windows: run the installer .exe and follow the instructions.
   • Linux: extract the zip and move the plugins to you prefered plugin folder (e.g. /usr/lib/vst3)
3. In your DAW, rescan plugins if it does not do so automatically.

Ableton Live usage notes

• If you want to use the VST3 plugin, make sure that "Use VST3 System Plug-in Folders" is On in Settings > Plug-ins before rescanning.
• To use WAIVE-Sequencer, add this plugin to an empty MIDI track. Then, on a separate MIDI track add your instrument (e.g. WAIVE-Sampler) then set MIDI From to the name of the first track and make sure to set it to use Post-FX.

Build from source (for developers)

If you want to build WAIVE-Plugins for your platform from source, follow the separate instructions here.

Troubleshooting

WAIVE-Sampler not loading sources list

• Make sure you are connected to the internet
• Click "View Folder" button to open up the location the database is saved in you file browser. Delete the file WAIVE.db, close and remove the plugin from the track, and re-add it and re-open.
  • On macOS, this is located at /Users/[your username]/Library/Application Support/WAIVE
  • On Windows, this is located at C:\Users\[your username]\App Data\Local\WAIVE

About

WAIVE-Plugins are built on top of several bespoke music and audio analysis models in order to create a unique and custom musical toolbox. These models will continue to be fine-tuned and updated on new data.

WAIVE-Sequencer utilises a flexible approach of modelling rhythmic data first outlined by Jon Gillick and collaborators. This works by encoding a percussion loop into it's Score and Groove components, then a model (a Variational AutoEncoder, VAE) is trained to combine them into a final output sequence. By adjusting the groove information, the same pattern (score) can be played in a different style. For the plugin, we also trained 2 smaller models (VAEs) that can encode the training data and can generate new Grooves and Scores in a variety of styles.

We use a combination of MIDI data and raw audio to collect rhythm information to train the models. In the case of raw audio, first the percussion instruments are isolated using demucs, then a method (Non-Negative Matrix Factorisation, demoed here) to further separate the individual percussion instruments is used to extract the score and groove information.

Rhythmic datasets include:

• Groove MIDI Dataset from Magenta
• Folkloras, Baltic folk music archive.
• Nederlandse Liederenbank, Dutch traditional music
• Free Music Archive selected electronic dance tracks

WAIVE-Sampler uses common techniques from machine listening to make measurements (called "features") of the audio, giving each sound a unique fingerprint and description of the audio content which can then be interpreted by other algorithms. At the same time, another method attempts to identify when percussive moments (called "onsets") occur using Complex Spectral Difference. The features are used to automatically identify appropriate moments in the Source audio that could be used for different drum hits, e.g. onsets that contain a broadband of frequency content can be shaped to make a snare sound.

For the Sample Map, the (log) Mel Spectrogram of the sample is measured, which computes a detailed representation of the frequency content of the audio. A simple, pretrained fully connected neural-net reads the measured features and outputs a 2D coordinate on where the audio sample should be placed on the map, e.g. such that kick drum sounds a grouped together, away from the hi-hat samples.

For the Tag cloud, we use an off-the-shelf word embedding model (GloVe) to produce a 2D representation (using TSNE dimensionality reduction) of all the tags in the database, such that semantically similar tags a clustered together to enable easier exploration of the available sources.

References

• Jon Gillick, Adam Roberts, Jesse Engel, Douglas Eck, and David Bamman (2019) 'Learning to Groove with Inverse Sequence Transformations.', International Conference on Machine Learning 2019 (ICML).
• Gillick, J., Yang, J., Cella, C.-E. and Bamman, D. (2021) ‘Drumroll Please: Modeling Multi-Scale Rhythmic Gestures with Flexible Grids’, Transactions of the International Society for Music Information Retrieval, 4(1), p. 156–166

Licenses

• DPF ISC license
• ONNX Runtime MIT
• libsndfile LGPL-2.1
• VG5000 font SIL Open Font License, Version 1.1
• Poppins Light font SIL Open Font License, Version 1.1
• kissfft BSD-3-Clause
• Gist GPL-3.0 license
• libsamplerate BSD-3-Clause
• nlohmann/json MIT
• TinyOSC ISC license
• tinyfiledialogs zlib/libpng license
• POCO Boost Software License