External mappings for MIDI instruments

[Bubobubobubobubo]

This draft PR proposes the addition of custom external instrument MIDI mappings exposing new parameters to be patterned. A mapping currently
takes this form:

let mapping = {
  port: "MIDI Bus 1",
  dummy: {cc: 80, channel: 1}, // channel is optional
  controller: true // whether or not to emit a note
}

note("C E G C").dummy(0.2).midi(mapping)

It could be much better with the following additions:

• custom range mapping
• automatically declaring new params (if they do not exist)
• documentation

[nkymut]

Hi! I've added Program Change and SYSEX support to MIDI mapping.

https://github.com/nkymut/strudel/tree/add-program-change

[Bubobubobubobubo]

@felixroos : anything we can do to push this further towards a merge?

[felixroos]

@felixroos : anything we can do to push this further towards a merge?

i can test this soon

[felixroos]

how about making the syntax more like a registry, so you wouldn't need a variable? something like:

midimap({
  port: "MIDI Bus 1",
  dummy: {cc: 80, channel: 1}, // channel is optional
  controller: true // whether or not to emit a note
})

note("C E G C").dummy(0.2).midi('MIDI Bus 1')

the midi function could check if there is a mapping for the given output and apply it if it exists.
another possibility would be to register new keys, so you could add multiple mappings for a single port:

midimap('mything', {
  port: "MIDI Bus 1",
  dummy: {cc: 80, channel: 1}, // channel is optional
  controller: true // whether or not to emit a note
})

note("C E G C").dummy(0.2).midi('mything')

the upsides of using a registry are

1. you could move the setup stuff (midimap call) somewhere else, no variable needs to be in scope
2. you can pattern it. thats currently not possible with the midi function, but who knows what the future brings

what do you think?

[yaxu]

Some ideas building on the registry idea:

// Generalised idea of simple way of defining a 'target' for patterns
const {dummy} = target('mything', new Midi({
  port: "MIDI Bus 1",
  dummy: {cc: 80, channel: 1}, // channel is optional
  controller: true // whether or not to emit a note
});

// So we can e.g. make an OSC, MQTT, Serial target as well:
target('myotherthing', new OSC({
  address: "127.0.0.1",
  port: 6010,
  schedule: "bundlestamp",
  shape: ['s', 'speed', 'pan'],
  handshake: false
}));

// .midi('mything') could just be .mything()
$: dummy("0.2*2 0.3(3,8)").note("<C E G C>").mything();

// It should be possible to send to more than one target
$: dummy("0.2*2 0.3(3,8)").note("<C E G C>").mything().myotherthing();

// I guess if you specify a target it doesn't go to the default unless you specify explicitly
$: dummy("0.2*2 0.3(3,8)").note("<C E G C>").mything().myotherthing().superdough();

These targets could be definable in strudel settings as a preamble to add to all patterns

The above contributes to #623 but I guess don't need to solve all at once.. My idea would be to have an interface for MIDI which is general enough to be extended to these other cases.

[yaxu]

Maybe this would be better:

$: dummy("0.2*2 0.3(3,8)").note("<C E G C>").target("mything, myotherthing, superdough")

Allowing this sort of thing:

$: dummy("0.2*2 0.3(3,8)").note("<C E G C>").target("mything").off(0.125, x => x.target("myotherthing"))

[felixroos]

my hot take would be to integrate targets into "s" at some point... but doesn't have to be in this PR

[Bubobubobubobubo]

my hot take would be to integrate targets into "s" at some point... but doesn't have to be in this PR

This would be the logical conclusion to making MIDI/OSC a first class citizen of the system.

I don't have much to add to your suggestions except that IMO this is peak Tidal/Strudel. Being able to pattern anything while "hiding" the subtleties of the backend used for producing pattern side effects in the real world.

@felixroos, @yaxu: How do you suggest implementing the midimap registry?

[felixroos]

my hot take would be to integrate targets into "s" at some point... but doesn't have to be in this PR

This would be the logical conclusion to making MIDI/OSC a first class citizen of the system.

I don't have much to add to your suggestions except that IMO this is peak Tidal/Strudel. Being able to pattern anything while "hiding" the subtleties of the backend used for producing pattern side effects in the real world.

@felixroos, @yaxu: How do you suggest implementing the midimap registry?

i'd maybe try to focus on midi in this PR so it doesn't get too hairy.
the first question would be if we want to be able to have multiple midi configs for one device name, which is probably handy ( see #1212 (comment) ).

then we could use a Map to map the names to the configs. the midi function would then check if the given name is either a config name or a device name and act accordingly.
ideally, the midi config would reuse existing control names:

midimap('mything', {
  port: "MIDI Bus 1",
  lpf: {cc: 80, channel: 1, range: [0,20000], exp: 2},
  controller: true // whether or not to emit a note
})

note("C E G C").lpf(400).midi('mything')

with the range mapping in place, the lpf function can be used normally. the config would then remap 400 to a value within the midi cc value range. This would mean you could play the same pattern with midi or superdough in the same way.
But maybe even the ranges are too much for this PR, so we concentrate on the registry and values are mapped as is?

@yaxu we could do a similar thing with osc but also hide it behind a function like oscmap, then see how we generalize it internally?

[yaxu]

@felixroos Yep ok let focus on midi

I feel like there are two things here - mapping strudel controls to CC values, and mapping midi messages to midi ports. Maybe these should be separated?

Can the midimap function support registering multiple control->cc mappings at once?

I guess if there are CCs and notes in one hap, then the CCs should be sent first and slightly ahead of time so as not to delay the note.

[Bubobubobubobubo]

the first question would be if we want to be able to have multiple midi configs for one device name

Most definitely. It's extremely common to split instruments by channel so we should support it. Your current plan seems good. The only thing missing is that map/memory system, and it shouldn't be too hard to implement.

I feel like there are two things here - mapping strudel controls to CC values, and mapping midi messages to midi ports. Maybe these should be separated?

Indeed, but they are separated in the current implementation. The only thing that declaring a mapping does is giving a new parameter name to a CCN, with CCV as parameter. If you omit the parameter name, no CC message is sent.

then the CCs should be sent first and slightly ahead of time so as not to delay the note

Yes, and there is a condition to it: sending a note on the same MIDI Port and channel. Otherwise, we don't really care. It can sometimes happen, currently, that the CC change happens after the note. Depending on the algorithm/synth used, it can sound messy.

I don't have time to tackle this in the next few days but will definitely have it at some point. I don't know if you have already planned for a 1.2 release schedule but I would really love to see this getting merged. It is quite essential to my current performance method. Let me know 📆

[felixroos]

yaxu I feel like there are two things here - mapping strudel controls to CC values, and mapping midi messages to midi ports. Maybe these should be separated?

good point, when they are separated (or at least separatable), then we can have a config for a common midi controller or plugin that doesnt contain the name of the audio device the person was using when mapping the device or plugin:

addMidimap('arturia-dx7', {
  lpf: {cc: 80, channel: 1, range: [0,20000], exp: 2},
  lpq: {cc: 81: .... },
  controller: true // whether or not to emit a note
})

note("C E G C").lpf(400).midimap('arturia-dx7').midi('MIDI Bus 1')

now, addMidimap does the registering and Pattern.midimap does the remapping on the pattern, turning controls like lpf into ccn / ccv pairs. the device / portname appears as usual only in the midi function.

Bubobubobubobubo Indeed, but they are separated in the current implementation. The only thing that declaring a mapping does is giving a new parameter name to a CCN, with CCV as parameter. If you omit the parameter name, no CC message is sent.

seems to be a misunderstanding here? or maybe i'm the one who misunderstood when answering above..

yaxu Can the midimap function support registering multiple control->cc mappings at once?

I'd say thats important yes. Then we would have a single config that maps from strudel controls to cc messages for a given device / plugin.

Bubobubobubobubo Yes, and there is a condition to it: sending a note on the same MIDI Port and channel. Otherwise, we don't really care. It can sometimes happen, currently, that the CC change happens after the note. Depending on the algorithm/synth used, it can sound messy.

should be doable to detect that

edit: another question would be if it should be possible to have multiple midimaps on a single pattern.. i'd say yes? each mapping can potentially handle different controls for different channels. overlaps would need to be handled (e.g. last mapping wins)

edit 2: alternative syntax, inspired by samples function:

midimaps({
  'arturia-dx7': {
    lpf: {cc: 80, channel: 1, range: [0,20000], exp: 2},
    lpq: {cc: 81: .... },
    controller: true // whether or not to emit a note
  }
})

edit 3: loading midi map(s) from a url:

midimaps('github:felixroos/midimaps')

[yaxu]

good point, when they are separated (or at least separatable), then we can have a config for a common midi controller or plugin that doesnt contain the name of the audio device the person was using when mapping the device or plugin:

Great! I think the 'midichan' should be separated/separable too then though?

Also is the control mapping side MIDI-specific? It seems it could also be used for things like adding alternative curves for superdough/osc parameters (most obviously gain). So this could just be called addMapping or similar, with the parameters just mapping to control names. Where a pattern value is used, an object describes the mapping.

addMapping({
  'arturia-dx7': {
    lpf: {ccn: 80, ccv: {range: [0,20000], exp: 2}, midichan: 10},
    controller: true // whether or not to emit a note
  }
})

(I'm unsure what the controller param is for)

With a specification like this the params can be overridden in the pattern, e.g. if a midichannel is changed:

note("c a f e").mapping("arturia-dx7").midichan("7").midi("MIDI Bus 1")

Again I think adding a mapping as a method would be nice:

note("c a f e").arturia_dx7().midichan("7").midi("MIDI Bus 1")

A way to add mappings to all patterns would be nice:

addMapping({
  'ALL': {
    overdrive: {distort: {range: [0,100]}, gain: {range: [0, 100]}},
  }
})

This also allows one-to-many mappings as per the above.. Similar to @daslyfe's idea to add generic timbre parameters that adjust more than one synthesis parameter to superdirt.

The value mappings could even be in the form of sanitised method calls, similar to the YAML strudel implementation, if that isn't too much of a security problem.. Then it would be really flexible mapping beyond range etc.

[Bubobubobubobubo]

I'm unsure what the controller param is for

It is not useful anymore if we carry on with the changes you are suggesting so don't worry about it 😄

seems to be a misunderstanding here? or maybe i'm the one who misunderstood when answering above..

Indeed, I get it now that you have both developed your ideas some more.

These changes will ask for a total rewrite of this current PR. I'm not sure if this branch is useful anymore. Note that it will also break #1244.

[felixroos]

if we open it up to a generic remapping, the json format would be syntax sugar for an fmap operation like this:

note("g1")
  .lpf("<400 2000 8000>")
  .fmap(
  value=>({
      ...value, // keep lpf?
      ccn: 80,
      ccv: value.lpf/20000 // in reality this would be value.cutoff ... (just in case you try to run this)
  })
).log()

the corresponding json would looks something like:

{ lpf: { ccv: { range:[0,20000] }, ccn: 80 } }

this type of remapping would certainly be useful in other scenarios (e.g. pitch tracked filter etc..).
i once sketched a function called "remap" that did this in a compact way but i can't find it anymore :-/
i faintly remember it looked something like:

note("g1").lpf("<400 2000 8000>").remap("lpf:0:20000:ccv:0:1")

or, using the json:

note("g1").lpf("<400 2000 8000>").remap({ lpf: { ccv: { range:[0,20000] }, ccn: 80 }})

there are some considerations to make:

• will the mapped control be replaced? in the example, should lpf be kept? if not, there could be something like keep: true.
• how to handle non numeric values? what about notes?
• .. ?

i fear this generalization might lead to this PR getting harder to handle..

[felixroos]

A way to add mappings to all patterns would be nice

could be done with all(x=>x.remap(...))

[felixroos]

one addition: the midi map has the specialty that the output range (ccv) is always [0, 1]. a more generic mapping would require specifying input and output range, altho the output range could default to [0, 1]. in some advanced future strudel, each control could have a range config...........

[yaxu]

then the CCs should be sent first and slightly ahead of time so as not to delay the note

Yes, and there is a condition to it: sending a note on the same MIDI Port and channel. Otherwise, we don't really care. It can sometimes happen, currently, that the CC change happens after the note. Depending on the algorithm/synth used, it can sound messy.

The problem is that the hardware MIDI protocol is pretty slow and serial. So it's not just about order but throughput. If you send a load of CC values to get the instrument ready for a note, the note will happen noticeably late, unless you send the CC ahead of time.

Even worse, the CC notes are separate from notes, because the MIDI committee had pianos and dead white guy music on the brain where musical events are only about pitch and duration and everything else are just properties of an instrument i.e. piano pedals.

This means we can't easily map timbral dimensions to notes at pattern time.

note("c e f g").lpf(300).distort(3).midi(...)

If lpf maps to straight to CC params in the pattern it would end up as note("c e f g").stack(cc("30:30,34:30")), which would actually double up each note.

So with this in mind I think the mapping can't be done in a nice generic way because general MIDI is not actually general enough. I think the mapping should after all be passed to the .midi() method so that the mapping can happen at the end of the chain where it's translated to CC and MIDI messages, where it can also take care of sending them in the right order and probably with the cc a little ahead of time.

This still means you can't have two notes happening at the same time with different CC values but that's a limitation of MIDI that I think it's not possible to get around.. On the plus side you can pattern effects separately from notes which isn't yet possible in superdough (#665).

stack(note("c e f g"),
  lpf(cosine.mul(3000)).distort(sine).segment(100)
).midi(...)

[felixroos]

we could use lists to send multiple cc pairs. i wouldnt scrap all weve discussed so far :x but yeah it makes defining a generic remapping more difficult.. id rather implement midi mapping in a non generic but useful way than trying coming up with an all encompassing format. we can refactor it later when we have collected some experience with it..

[felixroos]

on the issue of sending ahead of time: the clock works in a way that any callback is around 100ms early so i dont think this is a problem (unless proven otherwise)

[yaxu]

we could use lists to send multiple cc pairs. i wouldnt scrap all weve discussed so far :x but yeah it makes defining a generic remapping more difficult..

Yes I thought about that but it'd need to be patterns of lists of lists. I think that would need a bit of refactoring / breaking changes to support, and I'd generally worry about infecting strudel with midi's constraints. I think it could be better to leave core patterns based on simple key/value objects and translate to noteon/noteoff/cc at the time of scheduling.

Now I remember why I own some nice midi synths that I never use.. But a 'best case' way of translating to midi ccs+notes would be nice to make them a bit more useable!

[nkymut]

Following the MIDI mapping discussion and while I understand the urgency for the 1.2 release and @Bubobubobubobubo's upcoming performance needs.

This would be a proposal fo a future update.

The current midi.mjs handles MIDI messages in three distinct pipelines.

1. Individual MIDI messages (notes, CC, program changes) from hap.value Add MIDI Program Change, SysEx, NRPN, PitchBend and AfterTouch Output  #1244
2. System messages (clock, start, stop) via midicmd add midi clock support #710
3. Device-specific parameter mappings External mappings for MIDI instruments #1212

Perhaps we could consider splitting this into two packages:

@strudel/midi for low-level MIDI communication (1 and 2)
@strudel/midimap or @strudel/mididevice for device configurations and mappings (with JSON support for loading configurations, similar to sample files)

I see a lot of potential usecases for midimap to do sth like.

midimaps('github:mixedfeelingson/behringer')

note("c a f e").lpf(cosine.mul(3000)).midimap('behringer_td-3').midi(0) // TB-1
note("c a f e").lpf(perlin.mul(3000)).midimap('behringer_td-3').midi(1)  // TB-2

which relates to
#126

[nkymut]

I accidentally included some experimental code to test a GM module in the PR #1244,
but here's an interesting use case to consider:
https://github.com/tidalcycles/strudel/blob/c5d7f95441f067a35890ee4e674ddbc8bec8340d/packages/midi/gm.mjs

For example, with a GM (General MIDI) sound module configuration which maps program changes with instrument names,
you could do something like:

note("c a f e").sound("gm_piano") // Composing with a soundfont on the go

note("c a f e").sound("gm_piano").midimap("RolandSC88").midi(0) // Performing with a vintage hardware

[felixroos]

I think it could be better to leave core patterns based on simple key/value objects and translate to noteon/noteoff/cc at the time of scheduling.

ok then let's do it like this:

midimaps({
  'arturia-dx7': {
    lpf: {cc: 80, range: [0,20000], exp: 2},
    lpq: {cc: 81: .... },
  }
})
// or midimaps('github:felixroos/midimaps')

note("C E G C").lpf(400).midimap('arturia-dx7').midi('MIDI Bus 1')

• midimaps fills a registry with control to cc mappings
• .midimap is an ordinatry control
• .midi checks for each hap if the midimap control is set, and if it is, it'll transform all mapped controls to cc messages and send them slightly before note messages

agree?

Perhaps we could consider splitting this into two packages

sounds a bit overkill at first, but let's discuss this separately then

[yaxu]

Sounds good!

[Bubobubobubobubo]

Sounds good indeed. Could go further with more advanced PRs after 1.2 but this is a good milestone already.

[felixroos]

ok here we go: #1274 gonna close this one. thanks @Bubobubobubobubo for kicking things off!