Graph V2: Simpler, more flexible configuration by moving beyond modules

[thsig]

Summary

For several months now, we've been working on a big refresh on how Garden projects are configured. We've been calling this project "Graph V2" internally.

These changes will be part of Garden 0.13, which we're planning to release by the end of the year.

The new configuration format no longer nests builds, services, tests and tasks under modules—instead, these are configured independently as actions (see the [before & after](https://www.notion.so/Graph-V2-GitHub-ticket-draft-0d8a779c430d42d28e34e789fa48e9e5) example below for a preview of this).

Important note: Module configs will still work in 0.13—Garden converts them to actions internally. We're planning to deprecate them in 0.14, but this will give users plenty of time to slowly make the transition to writing action configs.

This will make Garden configuration simpler to understand, and fundamentally more flexible. It will also open the door to some exciting new features, and serve as a much simpler foundation than the old module system for our planned plugin SDK (since it will enable users to simply contribute action definitions, instead of always having to provide full-on module types).

We’ve been queuing several other planned features, improvements and lower-priority bugfixes for 0.13 to help keep things simple, since merging the current main branch into 0.13 is often nontrivial (given the extent of the changes involved). Once 0.13 is out, expect a much more rapid pace of new features and improvements in Garden!

We're creating this ticket to open the high-level design up for discussion.

Please let us know what you think, and don't hesitate to ask for more details where needed!

Problem statement

The way Garden projects are configured has, at least conceptually, remained relatively static since the first stable releases came out 2+ years ago. Since then, we’ve learned quite a lot and in many ways we’ve outgrown the current semantic structures. We’ve also learned where people tend to get confused, and there is a strong desire to make Garden’s semantics easier to grasp.

In short there is room for improvement, and doing so involves some non-trivial changes.

Here are some of the issues we’re addressing in 0.13:

1. The concept module is confusing, inconsistent, and doesn’t map neatly to how the Stack Graph is visualized and processed.
2. The general term “module” is very easily misleading since a “module” in some other context tends to mean something quite different.
3. The graph and its execution doesn’t actually feature the module concept much. We just talk about building, deploying, testing etc.
4. A module type actually envelops a build (but not always...), zero or more services, zero or more tests and zero or more tasks.
5. This is generally unintuitive and often it is difficult to decipher how this mapping happens for each module type.
   1. For example, a container module may or may not need to be built, and can contain any number of services, tasks and/or tests. A kubernetes module never needs to be built, always maps to exactly one service, and optionally one or more tasks and/or tests.
6. This also makes inheritance of types quite difficult. Say, if one provider would like to create a specific way to build an image, but not affect runtime execution, but another provider wants to create a specific service type and not affect the build. This would presently fit poorly into the model.
7. A single module can become quite massive and clumsy.
8. Separating the Garden-computed version for each entity in a module is presently done in a pretty hacky way (works in practice but the code is error-prone).
9. Having to carry around modules around our codebase, that serve a wide variety of different use cases and have this awkward nesting of entities, has become messy and difficult to understand.
10. We really don’t want to burden people with more terms and hierarchies to understand.
11. We want to make plugins conceptually simpler to implement.
12. The semantics of the graph are overly complex, beyond just the module concept. Between all the verbs and nouns and the various implications of certain dependencies over others, it’s often quite tricky to understand and even explain.
13. Sometimes the nouns we use don’t map naturally to what’s actually happening (is a Terraform stack a service?) or conflate with other surrounding semantics (e.g. a Garden service vs. a Kubernetes Service).

How we're addressing the above

In 0.13, we'll do the following:

1. Do away with the module concept entirely. No more confusing umbrella term.
2. Reduce the number of baked-in terms further, narrowing the graph-native concepts to the following verbs, collectively known as Actions:

• Build
• Deploy
• Run
• Test

We leave it undefined what to name the output from each of these. Garden doesn’t mind what you’re building (an artifact, image, binary etc.) or deploying (services, infrastructure etc.).

1. Allow anything to depend on anything, doing away with some prior constraints that have tripped people up in the past. A dependency is then specified as <kind>.<name>, e.g. build.my-image or deploy.my-service. There will still be some specific semantics for different types of dependencies, but quite a bit simpler than before (which was often difficult to reason about as well).

2. Introduce Groups. Any action can belong to a single Group, which is a config kind of its own.

3. A Group may define variables, which are then scoped and available to the nodes belonging to it. This accounts for one of the more useful aspects of the current module structure (module variables), but in a nicer and easier to understand manner.

4. Introduce a generalized garden do command, enabled by the simplified Garden semantics: garden do build.this deploy.that test.something 

   This represents a pure graph execution and other more specific commands (build, deploy, test etc.) become mostly short-hands for this underlying flow (but we absolutely still keep those!).

   1. We should also add optional action parameters, to control the behavior of each action. A crisp example would be dev mode for deployments: garden do deploy(dev=true).my-service.1. Existing modules are converted at runtime to the new, more atomic kinds, as a Group. We can also provide utilities for one-time conversions.1. We create a new GroupTemplate kind, which serves the same purpose as ModuleTemplates did prior.

5. We correspondingly create a new TemplatedGroup kind, which supplants the anyway-hacky templated module type, and does much the same thing except designed for the new config paradigm and referencing a GroupTemplate.

Concrete examples of action configs

A module like this:

kind: Module
name: frontend
description: Frontend service container
type: container
services:
  - name: frontend
    ports:
      - name: http
        containerPort: 8080
    healthCheck:
      httpGet:
        path: /hello-frontend
        port: http
    ingresses:
      - path: /hello-frontend
        port: http
      - path: /call-backend
        port: http
    dependencies:
      - backend
tests:
  - name: unit
    args: [npm, test]
  - name: integ
    args: [npm, run, integ]
    dependencies:
      - frontend

... now becomes a set of action configs:

kind: Build
type: docker-image
name: frontend
dockerfile: dev.Dockerfile

---

kind: Deploy
type: container
name: frontend
build: frontend
spec:
  ports:
    - name: http
      containerPort: 8080
  healthCheck:
    httpGet:
      path: /hello-frontend
      port: http
  ingresses:
    - path: /hello-frontend
      port: http
    - path: /call-backend
      port: http

---

kind: Test
type: container
name: frontend-unit
build: frontend
command: [npm, test]

---

kind: Test
type: container
name: frontend-integ
build: frontend
command: [npm, run, integ]
dependencies:
  - deploy:frontend

Here's an example of actions directly attached to a Group, using the group name as the shorthand for the action names:

kind: Group
name: api
variables: # These are group-specific overrides for variables, similar to old-style module variables.
  env:
    SOME_ENV_VAR: foo
actions:
- kind: Build
  type: docker-image
  name: api
  dockerfile: dev.Dockerfile

- kind: Deploy
  type: container
  name: api
  build: api
  env: ${var.env}

- kind: Test
  type: container
  name: api-unit
  build: api
  command: [npm, test]
  env: ${var.env}

- kind: Test
  type: container
  name: api-integ
  build: api
  command: [npm, run, integ]
  env: ${var.env}
  dependencies:
    - deploy:api