AI x Archive music tools
A plugin suite that combines music, sound and MIDI generation with European cultural archives. Aims to be an offline, modular version of WAIVE-Studio that can be integrated into your DAW. Built with DISTRHO Plugin Framework and ONNX Runtime.
- WAIVE-Midi: a rhythmic pattern generator
- WAIVE-Sampler: a sample player, sample library and sample generator in one
Developed by Arran Lyon for Thunderboom Records. Contributions and pull-requests welcome, especially regarding stability improvements and Windows release.
Currently, there is no installer provided, so you must install the plugins manually:
If you are reinstalled the plugin from an older, develpment version, you may need to delete the old database file as described in the Troubleshooting section.
- Download and extract the latest archive from the Releases page for your platform (currently only available for macOS Apple Silicon and Linux). You can find the download links under the Assets heading.
- Choose which plugin format you prefer and place the corresponding bundle (e.g.
WAIVE_Sampler.vst3
) in your plugins path of your DAW:-
Common (system-wide) plugin paths:
VST2 VST3 CLAP AudioUnit macOS Library/Audio/Plug-Ins/VST
Library/Audio/Plug-Ins/VST3
Library/Audio/Plug-Ins/CLAP
Library/Audio/Plug-Ins/Components
Linux /usr/lib/vst
/usr/lib/vst3
/usr/lib/clap
-- Windows C:\Program Files\Common Files\VST2
C:\Program Files\Common Files\VST3
C:\Program Files\Common Files\CLAP
-- -
To access these folders in macOS, open Finder, press
Shift+Command+G
, and enter the path you wish to access.
-
- In your DAW, rescan plugins if it does not do so automatically.
- If you installed the plug-ins in the above directories, then make sure that "Use VST3 System Plug-in Folders" is On (same for VST2) in Settings > Plug-ins before rescanning.
- To use WAIVE-Midi, add this plugin to an empty MIDI track. Then, on a separate MIDI track add your instrument (e.g. WAIVE-Sampler) then set MIDI From to the name of the first track and make sure to set it to use Post-FX.
- The VST3 and AudioUnit version of WAIVE-Midi currently are not loading in Live, so use the VST2 version.
To build WAIVE-Plugins from source.
Requires statically built onnxruntime for your platform. You can download pre-built libraries from csukuangfj/onnxruntime-libs, or build them yourself (such as with ort-builder). Tested with 1.17.1.
Requires cmake
and ninja
:
- on Mac, with homebrew:
$ brew install cmake ninja
- on Linux: use your distributions package manager
- Windows: coming soon
Requires vcpkg
:
- on Mac:
(you may wish to add the last line to your .bashrc or .zshrc to make it permanent)
$ git clone https://github.com/microsoft/vcpkg "$HOME/vcpkg" $ export VCPKG_ROOT="$HOME/vcpkg"
- on Windows/Linux: vcpkg installation instructions
$ git clone --recursive https://github.com/ThunderboomRecords/WAIVE.git
$ cd WAIVE/
Copy the lib/
and include/
folders from the static built onnxruntime you downloaded in the prerequisite step into a new folder
WAIVE/external/onnxruntime/
, then from project root:
$ mkdir build
$ cmake --preset=default
$ cmake --build ./build -j8 --config Release
The plugins are found in build/bin
folder. Move your prefered format binary to your plugins folder (see instructions above).
Coming soon.
- Make sure you are connected to the internet
- Click "View Folder" button to open up the location the database is saved in you file browser. Delete the file WAIVE.db, close and remove the plugin from the track, and re-add it and re-open.
- On macOS, this is located at
/Users/[your username]/Library/Application Support/WAIVE
- On macOS, this is located at
WAIVE-Plugins are built on top of several bespoke music and audio analysis models in order to create a unique and custom musical toolbox. These models will continue to be fine-tuned and updated on new data.
WAIVE-Midi utilises a flexible approach of modelling rhythmic data first outlined by Jon Gillick and collaborators. This works by encoding a percussion loop into it's Score and Groove components, then a model (a Variational AutoEncoder, VAE) is trained to combine them into a final output sequence. By adjusting the groove information, the same pattern (score) can be played in a different style. For the plugin, we also trained 2 smaller models (VAEs) that can encode the training data and can generate new Grooves and Scores in a variety of styles.
We use a combination of MIDI data and raw audio to collect rhythm information to train the models. In the case of raw audio, first the percussion instruments are isolated using demucs, then a method (Non-Negative Matrix Factorisation, demoed here) to further separate the individual percussion instruments is used to extract the score and groove information.
Rhythmic datasets include:
- Groove MIDI Dataset from Magenta
- Folkloras, Baltic folk music archive.
- Free Music Archive selected electronic dance tracks
WAIVE-Sampler uses common techniques from machine listening to make measurements (called "features") of the audio, giving each sound a unique fingerprint and description of the audio content which can then be interpreted by other algorithms. At the same time, another method attempts to identify when percussive moments (called "onsets") occur using Complex Spectral Difference. The features are used to automatically identify appropriate moments in the Source audio that could be used for different drum hits, e.g. onsets that contain a broadband of frequency content can be shaped to make a snare sound.
For the Sample Map, the (log) Mel Spectrogram of the sample is measured, which computes a detailed representation of the frequency content of the audio. A simple, pretrained fully connected neural-net reads the measured features and outputs a 2D coordinate on where the audio sample should be placed on the map, e.g. such that kick drum sounds a grouped together, away from the hi-hat samples.
For the Tag cloud, we use an off-the-shelf word embedding model (GloVe) to produce a 2D representation (using TSNE dimensionality reduction) of all the tags in the database, such that semantically similar tags a clustered together to enable easier exploration of the available sources.
- Jon Gillick, Adam Roberts, Jesse Engel, Douglas Eck, and David Bamman (2019) 'Learning to Groove with Inverse Sequence Transformations.', International Conference on Machine Learning 2019 (ICML).
- Gillick, J., Yang, J., Cella, C.-E. and Bamman, D. (2021) ‘Drumroll Please: Modeling Multi-Scale Rhythmic Gestures with Flexible Grids’, Transactions of the International Society for Music Information Retrieval, 4(1), p. 156–166
- DPF ISC license
- ONNX Runtime MIT
- libsndfile LGPL-2.1
- VG5000 font SIL Open Font License, Version 1.1
- Poppins Light font SIL Open Font License, Version 1.1
- kissfft BSD-3-Clause
- Gist GPL-3.0 license
- libsamplerate BSD-3-Clause
- nlohmann/json MIT
- TinyOSC ISC license
- tinyfiledialogs zlib/libpng license
- POCO Boost Software License