Configuration language

[llogiq]

Since the current way of writing synth schemas is...well, pragmatic from a code simplicity standpoint, I set out to improve this. So after looking around for suitable languages I happened upon koto which ticked all the boxes (well, except one that is being worked on, we use my fork for now). Why I made that choice is for another discussion. Here I want to show what should be working really soon, and where I intend to go with this:

Let's say we have a database table:

DATABASE_TYPE: postgres
SCHEMA: app
TABLE: users
id: primary key, Number
name:  VARCHAR
email: VARCHAR

The corresponding synth schema could look roughly like this:

{
  "type": "array",
  "length": {
     "type": "number",
     "subtype": "u64",
     "range": {
       "low": 1,
       "high": 4,
       "step": 1
     },
  },
  "content": {
    "type": "object",
    "id": {
      "type": "number",
      "subtype": "u64",
      "id": {}
    },
    "name": {
      "type": "string",
      "faker": {
        "generator": "name",
      },
    },
    "enail": {
      "type": "string",
      "faker": {
        "generator": "email",
      },
    }
  }
}

As you can see, this isn't exactly easy to get right (I've written this by heart right now and I'm sure I must've made some errors). Nor is it easy to read. With the current koto bindings, we should get (roughly, this is still untested):

synth.Namespace({
    synth.Name("users"): synth.Content.Array (
	{
	        synth.Name("id"): synth.Content.Number(NumberContent.Id(schema.Id)),
	        synth.Name("name"): synth.Content.String(StringContent.Faker("firstname", ["EN"])),
	        synth.Name("email"): synth.Content.String(StringContent.Faker("email", ["EN"])),
	},
	synth.Content.Number(NumberContent.Constant(10))
    })
})

I'm trying to add a trick to coerce values from koto to Rust that would give us:

synth.Namespace({
    "users": synth.Content.Array (
	{
	        "id": synth.Content.Number(NumberContent.Id(schema.Id)),
	        "name": synth.Content.String(StringContent.Faker("firstname", ["EN"])),
	        "email": synth.Content.String(StringContent.Faker("email", ["EN"])),
	},
	synth.Content.Number(NumberContent.Constant(10))
    })
})

Also adding the koto runtime will enable us to extend synth with scripting capabilities that I want to explore cautiously later on. We may also opt to add some syntactic sugar to make it even nicer to read. What do you folks think?

[christos-h]

Really excited about this. It should make for a much nicer user experience.

I'm going to be populating this thread with some of our internal notes over the next week to inform longer-term design.

[fretz12]

I'm really digging the expressiveness of this... and somehow it feels right that a scripting language can help us tackle the harder problem of- how do I custom parse something like a binary file.

Couple of comments/questions:

• Would this be an optional support in addition to the current JSON configs? While devs would dig this format, non-devs might prefer JSON
• Since this is basically a DSL, some of the nice text editor or IDE syntax helpers would be lost. Any ideas/plans there?
• I'm sure you've got solid reasons for choosing koto over other rust scripting languages... curious why rhai, which seems more mature, didn't fit the bill?
• if config verbosity is a concern, wondering if something like jsonnet can help us here? Nonetheless, still thinking the power of scripting can help us with the problem of custom parsing