RFC: var steps + local var sources

[vito]

Rendered

Related to, but not dependent on #24.

[jgriff]

Love This

This single feature has enabled us to streamline so many of our pipelines. 🙏
And in some cases overcome roadblocks for resources that didn't (easily) support file-based vars.

Improvement....Maybe?

I think one thing that could potentially make this better, at the risk of overcomplicating a simple file read, is support for jq-like extraction expressions (and others?) when loading the value.

For example, in our (GitLab) merge request pipelines we extract some meta info about the merge request (gitlab-merge-request-resource), which comes to us as a single merge-request.json buried in the .git folder of the merge request's branch that was fetched.

What I want to do is load the merge request number (.iid attribute) into a var.

Current Approach

To do that, I have to first jq extract the value to a file:

      - task: extract-merge-request-info
        config:
          platform: linux
          image_resource:
            type: registry-image
            source:
              # any image with 'jq' will do, our mr resource happens to have it and will have already been pulled to the job container so just use it (for job speed)
              repository: samcontesse/gitlab-merge-request-resource
          inputs:
            - name: merge-request
          outputs:
            - name: merge-request-info
          run:
            path: sh
            args:
              - -exc
              - |
                # extract the merge request number ---> merge-request-info/number
                jq <merge-request/.git/merge-request.json '.iid' >> merge-request-info/number
                
                # ...other attributes we may want (individually)

now I can use load_var to get what I want:

      - load_var: merge-request-number
        file:     merge-request-info/number

I could of course clean this up a little and write it as an external task, but the indirection step is still there and is still more cruft than I'd like.

Potential Improvement

It would be smoother if I could give load_var some instruction via a filter on what to extract from the file, instead of loading the entire content.

      - load_var: merge-request-number
        file:     merge-request/.git/merge-request.json
        filter:   {jq: '.iid'}

Other filters could be given/supported, perhaps a grep regex (but only one acceptable).

Increasing Complexity

This would of course make load_var more complex than it is currently. However I can see this as a pretty powerful feature for load_var to support scenarios where you are loading variables from non-simple files.

[vito]

@jgriff Cheers! Glad it's paying off already. :)

Hmm, can't you already load the JSON content and access the .iid field off the resulting var?:

- load_var: merge-request-number
  file:     merge-request/.git/merge-request.json
- put: foo
  params: {id: ((.:merge-request-number.iid))}

[jgriff]

@vito ah I see, yes! Did not realize the var was navigable like that, very nice 👍

[sahil87]

This is really great.

This allowed me to remove a custom task whose job was just to create a json file that could be loaded as docker build_args (containing json like { "BUILD_VERSION": "b-7a2abea" }). Now, this is replaced by a BUILD_VERSION var. I need this because my docker images depend on the build versions (b-<git-ref>) to create CDN paths containing built js files.

Earlier:

plan:
  - { get: repo_teaxrapi, trigger: true, params: { depth: 1 } } #git repo
  - task: task_get_build_properties #With the sole purpose of generating a json file
    ...
    outputs: [{name: build-props}]
  - put: drepo_teaxrapi #Using type: docker-image
    ...
    params:
      build_args_file: build-props/docker_build_args.json

Now:

plan:
  - { get: repo_docker-creds, trigger: true, params: { depth: 1 } }
  - { get: repo_teaxrapi, trigger: true, params: { depth: 1, short_ref_format: "b-%s" } }
  - { load_var: BUILD_VERSION, file: repo_teaxrapi/.git/short_ref }
  - task: build #Using type registry-image to push, oci-build-task to build
    ...
    params:
      BUILD_ARG_BUILD_VERSION: ((.:BUILD_VERSION))
      DOCKER_CONFIG: repo_docker-creds

All this resulted in builds being 30% faster now. The only side effect of this change was having to inject docker credentials for multi-stage builds as they are now handled by oci-build-task.

[xeivieni]

Hi guys,

Great job on this feature, it really simplified our pipelines.

A small improvement I can see would be to add a parameter to allow displaying the content of the file (thus, the value of the var) in the UI. It could be necessary in some cases.

👏

[Bluesboy]

Please do not refuse this feature, extremely helpful.

[holgerstolzenberg]

I also have to agree that this RFC is very useful and helped us to achieve some things that would have otherwise be overly complicated. Please keep it.

[tjhiggins]

Seconded. Dramatically cleaned up our pipeline.

[lukasmrtvy]

Guys? What about set_var ? Would not be better to interduce some K/V resource based on https://github.com/dgraph-io/badger ?

[vito]

@lukasmrtvy That makes sense to me! Just need someone to write up an RFC showing how it'd work. I didn't have any immediate use cases in mind so I didn't include it, but it seems intuitive.

Thanks everyone for the feedback! :)